You asked for it, you got it: a simple answer to who has the best BI tool. Last week we published the Critical Capabilities for BI & Analytic Platforms. The report ranks vendors based on scores of over 470 detailed criteria (and I thought BI Scorecard was excessive at 300+ criteria!) and four use cases.
Is There a Single Best?
Truth be told, I do not like the ranking charts in this report. The most meaningful parts of this report are the scores, the use cases, and the discussion of strengths and weaknesses.
Remember when high schools ranked students? Perhaps it’s clear who should be class Valedictorian, but beyond that, student rankings can be highly contentious and
of questionable value. How does a school account for differences in students who take basket weaving versus AP Physics? The same can be said of the BI tools. It’s hard to compare requirements such as row-level security (critical in healthcare verticals) versus mobile (critical for BI for field sellers), for example. I also see little value in ranking a vendor who receives an overall score of 3.61 and a vendor who receives a 3.58. Both are considered good to excellent, so while the charts in the report may appear as a strong rank order, I would not interpret them this way.
I have long maintained that there is not a single best BI tool, but rather, a best fit, and the interactive capabilities of the Critical Capabilities let you assess that.
Set Your Own Weightings
The Critical Capabilities report suggests four different uses cases for BI: whether deployed centrally or not, embedded or stand-alone. For each use case, you can set a weighting for 15 categories of the detailed criteria. For example, if you want governed data discovery, we have applied a weighting for cloud capabilities that you can increase or decrease.
Changing the weightings can drastically affect a vendor’s overall score and thus their ranking. For example, one customer initially disagreed with our ranking, that, for one use case put TIBCO ahead of SAP, and in another behind. First, the customer forgot that TIBCO acquired Jaspersoft last year, so assumed we were only scoring Spotfire. Secondly, decreasing the weighting of analytic dashboards and increasing the weighting of traditional BI and authored reports significantly changes the ranking score. This is just one example. If you see any surprises in the rankings, play with the weightings based on your requirements and see how the ranking scores change. Study the scores for major criteria on page 63.
The Halo Effect
In addition to the broad categories of capabilities, detailed scores are based on a combination of analyst opinion and customer feedback. We have noticed what my colleague Rita has dubbed a “halo effect”: customers who love a vendor or product are enthusiastic in their scores, whereas customers who are angry at the vendor or disappointed in a product seem overly negative in their product assessment. So while we analysts may think a certain capability is good, many irate customer responses may temper analyst scores. It’s impossible to tell just how much this happens, only that it does, and we’ve tweaked the evaluation model to give you the most balanced view of capabilities.
Read the Fine Print
While the charts are eye catching and front-page news, I want you to read the fine print, starting with the strengths and areas for improvement for each vendor. Because no matter which products you are currently using or are considering deploying, these are the sweet spots you want to leverage to and the limitations that you need to work around. With any deployment, it’s about managing user expectations and developing a strategy for a product’s shortcomings. Every product has them, and we’ve highlighted those that are the most important.
Not Exhaustive List of Vendors
The Critical Capabilities report is a companion report to the Magic Quadrant. It focuses on product capabilities and does not include other important strategic buying considerations such as vendor viability and vision, factors considered in the MQ. We also have focused on only those vendors that were positioned as Leaders or Challengers in 2014 and 2015. So it’s a deep dive on 12 vendors of the 26 on the MQ. Keeping in mind that every vendor has multiple products, it’s an assessment of more than 50 products.
Let Us Know What You Think
If the results still puzzle you, set up an inquiry call with any of the report authors. That’s what we are here for! Let us know what you think – about the findings and the presentation of them. This is s a new report type, and my goal continues to be to provide you with the best BI insights, in a way that most readily helps you manage your BI portfolio.
View Free, Relevant Gartner Research
Gartner's research helps you cut through the complexity and deliver the knowledge you need to make the right decisions quickly, and with confidence.Read Free Gartner Research
Comments or opinions expressed on this blog are those of the individual contributors only, and do not necessarily represent the views of Gartner, Inc. or its management. Readers may copy and redistribute blog postings on other blogs, or otherwise for private, non-commercial or journalistic purposes, with attribution to Gartner. This content may not be used for any other purposes in any other formats or media. The content on this blog is provided on an "as-is" basis. Gartner shall not be liable for any damages whatsoever arising out of the content or use of this blog.