Blog post

How not to use a Magic Quadrant

By Lydia Leong | June 25, 2009 | 2 Comments

Industry

The Web hosting Magic Quadrant is currently in editing, the culmination of a six-month process (despite my strenuous efforts to keep it to four months). Many, many client conversations, reference calls, and vendor discussions later, we arrive at the demonstration of a constant challenge: the user tendency to misinterpret the Magic Quadrant, and the correlating vendor tendency to become obsessive about which quadrant they’re placed in.

Even though Gartner has an extensive explanation of the Magic Quadrant methodology on our website, vendors and users alike tend to oversimplify what it means. So a complex methodology ends up translating down to something like this:

But the MQ isn’t intended to be used this way. Just because a vendor isn’t listed as a Leader doesn’t mean that they suck. It doesn’t mean that they don’t have enterprise clients, that those clients don’t like them, that their product sucks, that they don’t routinely beat out Leaders for business, or, most importantly, that we wouldn’t recommend them or that you shouldn’t use them.

The MQ reflects the overall position of a vendor within an entire market. An MQ leader tends to do well at a broad selection of products/services within that market, but is not necessarily the best at any particular product/service within that market. And even the vendor who is typically best at something might not be the right vendor for you, especially if your profile or use case deviates significantly from the “typical”.

I recognize, of course, that one of the reasons that people look at visual tools like the MQ is that they want to rapidly cull down the number of vendors in the market, in order to make a short-list. I’m not naive about the fact that users will say things like, “We will only use Leaders” or “We won’t use a Niche Player”. However, this is explicitly what the MQ is not designed to do. It’s incredibly important to match your needs to what a vendor is good at, and you have to read the text of the MQ in order to understand that. Also, there may be vendors who are too small or too need-specific to have qualified to be on the MQ, who shouldn’t be overlooked.

Also, an MQ reflects only a tiny percentage of what an analyst actually knows about the vendor. Its beauty is that it reduces a ton of quantified specific ratings (nearly 5 dozen, in the case of my upcoming MQ) to a point on a graph, and a pile of qualitative data to somewhere between six and ten one-or-two-sentence bullet points about a vendor. It’s convenient reference material that’s produced by an exhaustive (and exhausting) process, but it’s not necessarily the best medium for expressing an analyst’s nuanced opinions about a vendor.

I say this in advance of the Web hosting MQ’s release: In general, the greater the breadth of your needs, or the more mainstream they are, the more likely it is that an MQ’s ratings are going to reflect your evaluation of the vendors. Vendors who specialize in just a single use case, like most of the emerging cloud vendors, have market placements that reflect that specialization, although they may serve that specific use case better than vendors who have broader product portfolios.

Comments are closed

2 Comments

  • Lydia, thanks for the honesty. Gartner is good at distilling information into a bullet and a blurb. What I wish Gartner would do is create an actual capabilities matrix based on the MQ criteria. You probably have the beginnings of such a list, but becuase you don’t deep dive into technology as far as I have seen you do not capture key details that would help clients select vendors based on capablities not size.

  • Lydia Leong says:

    I think, looking at the MQ, that it doesn’t just reflect size — it also reflects the breadth of product portfolio and overall market presence. For instance, Rackspace and Terremark have been consistently highly placed for 5 years, even when both were small vendors.

    But you’re right, the MQ is not designed to show feature breakdowns. That’s why I’m building a critical capabilities note for cloud system infrastructure services — because featureset is much more of a driver in cloud computing right now, whereas classically enterprise Web hosting hasn’t had much genuine feature differentation, (the main differences in enterprise hosters have been service, sometimes driven by back-end technology but also sometimes driven by just raw human attention).

    I feel pretty strongly that Gartner clients doing vendor selection should do an analyst inquiry call rather than just relying on the written research. There’s a ton of detail and color that simply doesn’t fit well into the bullet-point format of the MQ, not to mention the usefulness of custom RFP / pricing / contract review.