That is common misperception. For example, in the most recent MQ I published (Magic Quadrant for Employee Performance Management Software) one of the leaders is not a client. Being a client does not impact placement on the MQ. Also, regardless of whether or not the vendor is a client, each rated vendor has the opportunity in the fact checking part of the process to understand why we rated them the way we did.

However, most vendor clients (I know there are certainly some that have the same misperception) become clients so that they can get access to more insights about the market, what is on the minds of end users, and how their solutions stack up against the competition (among other things).

Do we need an “independent” entity that doesn’t rely on revenues from vendors to rate this vendors? We are all familiar with the mess created by the financial rating agencies (Moody’s, etc) which relied on fees from financial institutions to rate their debt products.

There are certainly perceptions about “goodness” and “badness” about dots on a MQ. It would be disingenuous to say that it does not matter to some customers. However, most clients that I talk to via inquiry with want to find the best solution for their needs. They do not really care if it is in the upper right hand quadrant. I know that is not what most vendors think. That is one of the reasons that I did the post.

We are more transparent on criteria than we used to be (we now show the criteria and relative weight), but, personally (and this is my opinion) I would like to see us expose more detail of the evaluation. In some of the research, like Critical Capabilities, the evaluation criteria and weightings are spelled out very specifically. It is my personal hope that we show more in the future.

Gartner’s long-standing lack of transparency on MQs makes matters worse. Even if you do have a reader with the time and sophistication to dig into the details of an MQ, the relatively importance to THEM, not Gartner, of various criteria has no voice. By not exposing the specific criteria, the ratings and the weightings the reader has no way to weigh in. Some users have waved goodbye to MQs and switched to other benchmarks because the criteria is laid out, and the user can input their own weightings.

Maybe Gartner considers the “secret sauce,” the mystery, to be key to the panache, appeal and power of the MQ. I understand that from a marketing perspective certainly. But as an ex-analyst, an ex-IT buyer/product manager, and now in AR my viewpoint on MQ is we deal with it because we have little choice, it is to IT benchmark analysis what Microsoft Windows is to PC client operating systems.

Just as many of us hope that Windows 7 will truly offer an enhanced computing experience (for those of us fated for one reason or another to not be Mac users), there is similarly latent hope that someday Gartner will completely dump the secret sauce for full transparency, and even better let the user manipulate what is important to them in the model. Gartner has heard this for years (decades?) and made cursory strides. Your post peels off a layer of the onion, it would be great if Gartner showed all the layers some day.

Readers that would like to read the original blog posts that prompted Jim’s note may click on my name above. My post is linked, and I link to those of others that also weighed in.

But why not engage in a direct conversation with Frank Scavo, Michael Doane, me and others on blogs we have recently written particularly about the recent mid-market ERP MQ In the new world of web transparency that would be so much better.

I see vendors SAP (in particular) respond directly on blogs. Be nice if analysts did the same. Cheers