A vendor asked if we had a formal procedure that analysts follow to check user references provided by vendors and was specifically interested in whether analysts are obligated to contact the companies that the vendors provide as references.
Analysts are not obligated to ask for references when creating any type of document, but they often do because they believe that feedback will enrich their research findings and advice. When references are requested, Gartner has an established policy around checking them. Here are a few key highlights from the policy that were shared with the vendor:
If an analyst requests references from a vendor, that analyst is required to contact those references or advise the vendor which of the provided references will not be contacted. References are considered “contacted” once the analyst attempts to reach them (usually via phone, web survey or email), whether they respond or not. Analysts are encouraged to make more than one attempt to contact a reference if they don’t hear back, because anyone can forget to respond to a call or email.
If one or more references for a vendor does not respond, this would reflect negatively on the evaluation of the vendor and there is sufficient time, analysts generally give the vendor the opportunity to provide additional references. Note: The analysts are not required to reveal which references did not respond.
Here’s another relevant question we sometimes get: Why doesn’t Gartner contact references once a year, and in a more general way, so the answers can be shared between research teams? While that would certainly be easier for both parties, here’s the hard part: analysts strive to ask references questions that will give strong insight into how a product or service “really” works – or not. You can’t get at this information by asking generic questions, so the more specific the questions are, the more relevant the answers are to the research.
So what? Providing a list of high-quality references is hard work for vendors; it takes effort, skill and finesse. Knowing this, Gartner Research crafted its reference policies so these efforts are not wasted (and maintains them to ensure this, as well). It isn’t perfect, of course. How can it be improved? Let us know what you think. We’ll be sure to share your ideas with Research, who manages these policies.