Jim Holincheck

I know I have not posted for a while.  Work commitments and posting more on Twitter (at jimholincheck) have taken an inordinate amount of time (that and being on vacation and then ill after vacation).  However, two things happened that pushed me to do this post.  First, Bill Kutik did a nice article on what an industry analyst does.  Second, I saw a tweet questioning the value of Gartner Magic Quadrants (MQs).  It is certainly not the first time nor will it be the last time that there is criticism about MQs, but I think there is a lot of misunderstanding out there about what they are and how they are used.  Since Bill did such a good job on the industry analyst bit, I thought this would be a good follow on.

What is a Magic Quadrant?

Let me begin by saying these are not the official Gartner definitions with all of the nuances.  I am trying to keep it simple.  With the disclaimer set, a Magic Quadrant is a relative comparison of vendors in a specific market on their ability to execute and completeness of vision which is plotted in a 2×2 matrix (four quadrants – each with a different name: Leader, Challenger, Visionary, and Niche).  Let me parse this further:

• Relative Comparison – The placement on the MQ chart is based on relative scores against the criteria.  An important implication is that a MQ is a point-in-time view of a market.  To update a MQ, all of the vendors considered need to be re-evaluated at the same time (or else you cannot do a relative rating)
• Specific Market – An analyst evaluates vendors in a market in a MQ.  It is not just a product evaluation (though product/service is one of the criteria).  A common complaint I hear from vendors is "how can you rate vendor X better than vendor Y when vendor Y clearly has a better product".  The answer is that product/service is just one part of the overall rating of the vendor, but it is not the only thing evaluated.
• Ability to Execute – This axis of the 2×2 matrix is a composite of a number of evaluation criteria including:
  • Product/Service Capability
  • Overall Viability
  • Sales Execution/Pricing
  • Market Responsiveness/Track Record
  • Marketing Execution
  • Customer Experience
  • Operations
• Completeness of Vision – This axis is a composite of the following evaluation criteria:
  • Market Understanding
  • Marketing Strategy
  • Sales Strategy
  • Offering (Product) Strategy
  • Business Model
  • Vertical/Industry Strategy
  • Innovation
  • Geographic Strategy

The analyst(s) writing a MQ chooses weights for each of these criteria.  This is pretty straightforward.  However, it is surprising how many times vendors will make comments like "how can vendor X be rated more highly than vendor Y when we beat them all of the time in deals".  Well, if you look at the evaluation criteria, you can see how that can happen.  Vendor Y may beat Vendor X, but they are in far fewer deals because they have much lower market awareness, for example.

Sometimes you will also hear criticism that Ability to Execute and Completeness of Vision do not matter as decision criteria for customers.  I think the evaluation criteria within Ability to Execute and Completeness of Vision give a well-rounded view of a vendor and its place in a particular market and that is really the point of a Magic Quadrant.  A 2×2 matrix is a really convenient way of showing the relative comparison of vendors.  However, there are actually a number of different ways we could choose to present the results based on how the scoring is actually done. 

How Clients Use Magic Quadrants

My previous comments have included mostly vendor reactions to (ok complaints about) Magic Quadrants.  There is a perception out there that clients basically will look at a MQ and put the vendors in the Leaders quadrant on their short list.  There is no doubt that some clients do that.  They shouldn’t, but they do.  However, the majority of clients do not do that.   Most clients use MQs to get better educated about what is happening in a market.  In business applications, clients do not make these purchases very frequently so clients will use it to find out the current state of the market.  In addition, they often use it as a way to put together a long list of vendors to do more research about.  More importantly though, most clients will set up an inquiry (or a series of inquiries throughout the selection process) with an analyst to discuss their specific requirements. 

As I indicated above, a Magic Quadrant is a point-in-time view.  Things change, so clients want to see what has happened since publication.  In addition, customer requirements differ and priorities may be different from what we used in the MQ evaluation.  In many cases, non-Leader quadrant vendors are actually a better fit for a specific customer situation.  We make this point in the research and end user clients get it.  They want the best fit for their specific needs. 

The research we do into the markets and vendors and the interactions we have customers and Gartner clients gives us perspective to tailor our advice for customer-specific needs.  So, is the Magic Quadrant a good tool to use to make a vendor selection?  It can be, but using a Magic Quadrant along with analyst inquiry is a much better way to go (and the way most clients do go).

MarketScopes vs. Magic Quadrants

Magic Quadrants are among the most well-known Gartner research deliverables.  However, we do have other vendor comparison research we publish.  That brings us next to the MarketScope.  A MarketScope differs from a Magic Quadrant in the following ways:

• It is an absolute, not relative rating of vendors
• It is used primarily for early stage or late stage markets
• It uses a different graphic (each vendor is rated on a five point scale between Strong Negative and Strong Positive instead of plotting the vendors on a 2×2 matrix)
• It uses a subset of the Magic Quadrant evaluation criteria (up to seven criteria)

The process for creating a MarketScope is very similar to a Magic Quadrant.  However, it is easier to change as you do not need to update ratings for all vendors at the same time (because it is not a relative rating).  So, it can be changed more frequently for early stage markets or changed less frequently for later stage markets.

We publish a MarketScope for Large Enterprise HRMS (Gartner subscription required) because it is a later stage market (though you could argue that the emergence of SaaS alternatives may result in a new buying cycle – so we do keep an eye on things like that).  However, we now have a new note type that provides another vehicle for vendor comparisons called Critical Capabilities.  We will use this methodology for some of our forthcoming HRMS research.

What are Critical Capabilities?

This is a newer type of research note for Gartner and I am quite excited about it.  A Critical Capabilities note evaluates a group of vendors/products against key product capabilities for different use case scenarios.  Critical Capabilities (CCs) differ from MQs and MarketScope in the following ways:

• CCs evaluate product and product capabilities not vendors in markets.  So, the product capabilities could span more than one market.
• MQs or MarketScopes have one set of weightings for the evaluation criteria.  Each use case in a CC has a different weighting. 

A CC note can be very complimentary to a MQ or MarketScope because it allows a deeper drill down into product capabilities than we can get into in the other vendor comparison research.  I will put in a shameless plug for one of the analysts on my team, Debbie Wilson, who just published one of these notes called Critical Capabilities for Best of Breed E-Procurement Vendors (Gartner subscription required), In addition, it allows for vendor comparisons before a market is really established around a set of capabilities.  The Critical Capabilities for Talent Management Suites note I am working on with Thomas Otter and Carol Rozwell is a good example of this usage.

Hopefully, I have cleared up some of the misunderstanding out there.  Questions and comments are certainly welcome.