The Value of Vendor Reference Checking

Jim Sinur (https://blogs.gartner.com/jim_sinur/) and I are hard at work updating our BPMS Magic Quadrant. Part of the research process is reference checking. I’ve been doing alot of them lately. (Some even with simultaneous translation as yours truly is the ‘stupid American’ who only speaks English! Ugh!) Having completed somewhere near 15, I am astounded by my findings.

So far, most of the references I’ve checked are incredibly weak! The disappointments I’ve experienced include erroneous contact information, individuals that decline our request for a confidential conversation, references who are technology partners testing the tool or implementing it on behalf of an end user rather than the end user speaking first hand about their experience with the product, references that aren’t even using the product yet, references that arent experiencing the value from their BPMS that we would expect or who are just using it as ‘yet another development tool’ rather than a platform to empower business roles to better manage their processes, and references who receive tons of free products and professional services that wouldnt normally be available to any buyer.

Having worked in Marketing myself at both large and small software technology providers, I have a good idea of how much work it takes to maintain quality references. And it’s alot! Plus it requires constant attention! I have assumed that this effort is far better managed at larger software companies that have large marketing departments with plenty of resources to support their sales efforts. From my experience this year, I’ve concluded that this assumption is astoundingly wrong! There is no correlation between the size of the vendor and the quality of the references!

We always recommend that buyers check vendor references before finalizing their technology product and vendor selections. I sure hope you have better results than me! I wonder…..Do the providers give prospects real, valid references? Are they just being sloppy and casual about our request for references because I’m just an industry analyst rather than a prospective buyer?

We ask the vendors for multiple references with a committment that we will call at least two as part of the MQ process. We also tell the providers that showing us more references is always better than just a few. (It’s more impressive if a provider can quickly give me a list of 20 references, including links to their own write ups!) Is it possible that they think we wont actually call their references??  

What is the value of reference checking? Why do we do it and what are we looking for? Upon providing feedback to a vendor about a particularly poor reference, I discovered that the provider didnt really understand what I look for in a reference.

The whole point of reference checking is to find another customer, ideally in your own industry, who is using the product you are considering in a fashion very similar to your own use case. You want to learn about their experience in order to increase your confidence that you too can be successful in your project with this product and vendor.  I look for an experience that is likely to be replicated in other buyer situations. So if the provider provides the software for free, provides free professional assistance, installs and configures the product themselves for their customer, and delivers other examples of value that are not part of the “normal” offer, then this is not a good reference. With so much assistance, of course the buyer is likely to have a more positive experience than they might have otherwise. If the provider is doing everything, then the buyer likely has little feedback about weak areas of the product, challenges they’ve had with the product or even their project, and overall little to offer in the way of  lessons learned. Buyers want to learn from each other’s mistakes and successes. A reference that says, “the product is missing X feature” or “it seems like it should be easier to do X,” is a far better reference than one who says, “I dont know” to every question I ask.

Having been a buyer myself, even if I spoke to a reference who told me about their challenges, I was still likely to buy the product. I just went into the deal more educated. By the time most buyers actually conduct reference checking, they are pretty darn sure they want to buy vendor X’s product. Only an absolutely terrible reference would have deterred me from my product selection.

Tell me about your experiences. Please. Restore my faith in this worthwhile effort!