Gartner Blog Network

Who Has the Best BI Tool? (aka Interpreting Critical Capabilities Report)

by Cindi Howson  |  June 1, 2015  |  5 Comments

You asked for it, you got it: a simple answer to who has the best BI tool. Last week we published the Critical Capabilities for BI & Analytic Platforms. The report ranks vendors based on scores of over 470 detailed criteria (and I thought BI Scorecard was excessive at 300+ criteria!) and four use cases.

Is There a Single Best?

Truth be told, I do not like the ranking charts in this report. The most meaningful parts of this report are the scores, the use cases, and the discussion of strengths and weaknesses.

Remember when high schools ranked students? Perhaps it’s clear who should be class Valedictorian, but beyond that, student rankings can be highly contentious and

Best Graduates

Best Graduates

of questionable value. How does a school account for differences in students who take basket weaving versus AP Physics? The same can be said of the BI tools. It’s hard to compare requirements such as row-level security (critical in healthcare verticals) versus mobile (critical for BI for field sellers), for example. I also see little value in ranking a vendor who receives an overall score of 3.61 and a vendor who receives a 3.58. Both are considered good to excellent, so while the charts in the report may appear as a strong rank order, I would not interpret them this way.

I have long maintained that there is not a single best BI tool, but rather, a best fit, and the interactive capabilities of the Critical Capabilities let you assess that.

Set Your Own Weightings

The Critical Capabilities report suggests four different uses cases for BI: whether deployed centrally or not, embedded or stand-alone. For each use case, you can set a weighting for 15 categories of the detailed criteria. For example, if you want governed data discovery, we have applied a weighting for cloud capabilities that you can increase or decrease.

Customize Your RequirementsChanging the weightings can drastically affect a vendor’s overall score and thus their ranking. For example, one customer initially disagreed with our ranking, that, for one use case put TIBCO ahead of SAP, and in another behind. First, the customer forgot that TIBCO acquired Jaspersoft last year, so assumed we were only scoring Spotfire. Secondly, decreasing the weighting of analytic dashboards and increasing the weighting of traditional BI and authored reports significantly changes the ranking score. This is just one example.  If you see any surprises in the rankings, play with the weightings based on your requirements and see how the ranking scores change. Study the scores for major criteria on page 63.

The Halo Effect

In addition to the broad categories of capabilities, detailed scores are based on a combination of analyst opinion and customer feedback. We have noticed what my colleague Rita has dubbed a “halo effect”:  customers who love a vendor or product are enthusiastic in their scores, whereas customers who are angry at the vendor or disappointed in a product seem overly negative in their product assessment. So while we analysts may think a certain capability is good, many irate customer responses may temper analyst scores. It’s impossible to tell just how much this happens, only that it does, and we’ve tweaked the evaluation model to give you the most balanced view of capabilities.

Read the Fine Print

While the charts are eye catching and front-page news, I want you to read the fine print, starting with the strengths and areas for improvement for each vendor. Because no matter which products you are currently using or are considering deploying, these are the sweet spots you want to leverage to and the limitations that you need to work around. With any deployment, it’s about managing user expectations and developing a strategy for a product’s shortcomings. Every product has them, and we’ve highlighted those that are the most important.

Not Exhaustive List of Vendors

The Critical Capabilities report is a companion report to the Magic Quadrant. It focuses on product capabilities and does not include other important strategic buying considerations such as vendor viability and vision, factors considered in the MQ. We also have focused on only those vendors that were positioned as Leaders or Challengers in 2014 and 2015. So it’s a deep dive on 12 vendors of the 26 on the MQ. Keeping in mind that every vendor has multiple products, it’s an assessment of more than 50 products.

Let Us Know What You Think

If the results still puzzle you, set up an inquiry call with any of the report authors. That’s what we are here for!  Let us know what you think – about the findings and the presentation of them. This is s a new report type, and my goal continues to be to provide you with the best BI insights, in a way that most readily helps you manage your BI portfolio.


Cindi Howson

Additional Resources

View Free, Relevant Gartner Research

Gartner's research helps you cut through the complexity and deliver the knowledge you need to make the right decisions quickly, and with confidence.

Read Free Gartner Research

Category: business-analytics  business-intelligence  

Cindi Howson
Research VP
1 years at Gartner
25 years IT Industry

Cindi Howson is a Research Vice President at Gartner, where she focuses on business intelligence (BI) and analytics. Her work includes writing about market trends, vendors and best practices and advising organizations on these subjects. Read Full Bio

Thoughts on Who Has the Best BI Tool? (aka Interpreting Critical Capabilities Report)

  1. William Frank says:

    Thanks for this guidance. In the end the best BI tool is likely the one that has provided the most value for a particular use case. While admittedly this does not measure technical capabilities, TCO and other areas, it sometimes is in “the way that you use it” vs. “what it’s capable of”

  2. […] for the halo effect. As I wrote in this blog, customers who are happy with their vendor and product seem to rate product capabilities higher, […]

  3. […] I fear it boils a very complex set of requirements into an overly simplistic ranking as I wrote in last year’s blog. So you tell […]

Leave a Reply

Your email address will not be published. Required fields are marked *

Comments or opinions expressed on this blog are those of the individual contributors only, and do not necessarily represent the views of Gartner, Inc. or its management. Readers may copy and redistribute blog postings on other blogs, or otherwise for private, non-commercial or journalistic purposes, with attribution to Gartner. This content may not be used for any other purposes in any other formats or media. The content on this blog is provided on an "as-is" basis. Gartner shall not be liable for any damages whatsoever arising out of the content or use of this blog.