Gartner Blog Network


The Value of Vendor Reference Checking

by Janelle Hill  |  May 4, 2010  |  3 Comments

Jim Sinur (https://blogs.gartner.com/jim_sinur/) and I are hard at work updating our BPMS Magic Quadrant. Part of the research process is reference checking. I’ve been doing alot of them lately. (Some even with simultaneous translation as yours truly is the ‘stupid American’ who only speaks English! Ugh!) Having completed somewhere near 15, I am astounded by my findings.

So far, most of the references I’ve checked are incredibly weak! The disappointments I’ve experienced include erroneous contact information, individuals that decline our request for a confidential conversation, references who are technology partners testing the tool or implementing it on behalf of an end user rather than the end user speaking first hand about their experience with the product, references that aren’t even using the product yet, references that arent experiencing the value from their BPMS that we would expect or who are just using it as ‘yet another development tool’ rather than a platform to empower business roles to better manage their processes, and references who receive tons of free products and professional services that wouldnt normally be available to any buyer.

Having worked in Marketing myself at both large and small software technology providers, I have a good idea of how much work it takes to maintain quality references. And it’s alot! Plus it requires constant attention! I have assumed that this effort is far better managed at larger software companies that have large marketing departments with plenty of resources to support their sales efforts. From my experience this year, I’ve concluded that this assumption is astoundingly wrong! There is no correlation between the size of the vendor and the quality of the references!

We always recommend that buyers check vendor references before finalizing their technology product and vendor selections. I sure hope you have better results than me! I wonder…..Do the providers give prospects real, valid references? Are they just being sloppy and casual about our request for references because I’m just an industry analyst rather than a prospective buyer?

We ask the vendors for multiple references with a committment that we will call at least two as part of the MQ process. We also tell the providers that showing us more references is always better than just a few. (It’s more impressive if a provider can quickly give me a list of 20 references, including links to their own write ups!) Is it possible that they think we wont actually call their references??  

What is the value of reference checking? Why do we do it and what are we looking for? Upon providing feedback to a vendor about a particularly poor reference, I discovered that the provider didnt really understand what I look for in a reference.

The whole point of reference checking is to find another customer, ideally in your own industry, who is using the product you are considering in a fashion very similar to your own use case. You want to learn about their experience in order to increase your confidence that you too can be successful in your project with this product and vendor.  I look for an experience that is likely to be replicated in other buyer situations. So if the provider provides the software for free, provides free professional assistance, installs and configures the product themselves for their customer, and delivers other examples of value that are not part of the “normal” offer, then this is not a good reference. With so much assistance, of course the buyer is likely to have a more positive experience than they might have otherwise. If the provider is doing everything, then the buyer likely has little feedback about weak areas of the product, challenges they’ve had with the product or even their project, and overall little to offer in the way of  lessons learned. Buyers want to learn from each other’s mistakes and successes. A reference that says, “the product is missing X feature” or “it seems like it should be easier to do X,” is a far better reference than one who says, “I dont know” to every question I ask.

Having been a buyer myself, even if I spoke to a reference who told me about their challenges, I was still likely to buy the product. I just went into the deal more educated. By the time most buyers actually conduct reference checking, they are pretty darn sure they want to buy vendor X’s product. Only an absolutely terrible reference would have deterred me from my product selection.

Tell me about your experiences. Please. Restore my faith in this worthwhile effort!

Category: 

Janelle B. Hill
Research VP
8 years at Gartner
25 years IT industry

Janelle Hill is a vice president in Gartner Research. Her current research area is business process management disciplines (such as modeling, analysis, measurement, process ownership/governance, etc.) and new, BPM-enabling technologies. Read Full Bio


Thoughts on The Value of Vendor Reference Checking


  1. […] This post was mentioned on Twitter by Erica. Erica said: The Value of Vendor Reference Checking: You want to learn about their experience in order to increase your confide… http://bit.ly/aXog1V […]

  2. […] Quotes of the week On Vendor Reference Checking – Janelle Hill From my experience this year, I’ve concluded that this assumption is […]

  3. Janelle,
    I understand your frustration on experiencing your end of the reference interview process. I’d also like you to understand the frustration of one or the other reference client as well. Their main job is not to wait for your or any other analyst firm’s call in the first place. They’re very unhappy with the opaqueness of your calling process, instead calling back to their vendor who originally urged them to participate, asking for the status of the firm’s inquiry. Unfortunately the vendor has no insight into the analyst firm’s process either, so everybody is unhappy about this “don’t know” state. This leads many great referenceable clients to declining participation in any analyst reference calls. What was in it for them in the first place?
    Harald



Comments are closed

Comments or opinions expressed on this blog are those of the individual contributors only, and do not necessarily represent the views of Gartner, Inc. or its management. Readers may copy and redistribute blog postings on other blogs, or otherwise for private, non-commercial or journalistic purposes, with attribution to Gartner. This content may not be used for any other purposes in any other formats or media. The content on this blog is provided on an "as-is" basis. Gartner shall not be liable for any damages whatsoever arising out of the content or use of this blog.