Blog post

Perspectives on Hadoop Part Two: Pausing Plans

By Merv Adrian | June 04, 2015 | 5 Comments

RDBMSOSSOracleopen sourceMicrosoftMapRIndustry trendsIBMHortonworksGartnerDBMSData IntegrationClouderaApache MapReduceApache HadoopApacheData and Analytics StrategiesTechnology and Emerging Trends

By Merv Adrian and Nick Heudecker

In the first post in this series , I looked at the size of revenue streams for RDBMS software and maintenance/support and noted that they amount to $33B, pointing out that pure play Hadoop vendors had a high hill to climb. (I didn’t say so specifically, but in 2014, Gartner estimates that the three leading vendors generated less than $150M.)

In this post, Nick and I turn from Procurement to Plans and examine the buying intentions uncovered in Gartner surveys. We recently described some of our findings in the Gartner research note Survey Analysis: Hadoop Adoption Drivers and Challenges, and since its publication we’ve been inundated by questions about why “Gartner is so negative about Hadoop.” Questions include:

How has Gartner come to this position when other analyst firms are saying the opposite?

The answer is simple: we looked at the data. We received survey responses from 284 participants in the Gartner Research Circle, a sizable research panel of enterprises from less than $500M to global companies in the multibillions, across a representative set of verticals and geographies, and including both Gartner clients and non-clients. 54% of them said they have no plans to invest in Hadoop in the next two years.

We commissioned the survey from our primary research team to supplement what we were hearing from clients during inquiry. (An inquiry is a 30-60 minute phone call between an analyst and a client. It’s essentially how we contextualize our written research.) Combined, the two of us take about 1000 inquiries per year and more than half of them involve Hadoop. Based on these conversations, we simply aren’t seeing the breakneck adoption the overall hype indicates. Clusters over 20 nodes are rare, and production deployments are even more so. This mountain of anecdotal data gave us a sense that Hadoop adoption wasn’t living up to the hype, but turning several hundred conversations into published research is challenging without incredibly dedicated effort – and we still have inquiries to take and research to write. Hence the survey.

Are there positive notes in this report, especially for vendors in the Big Data/Hadoop space? Aren’t companies like Cloudera, Hortonworks and MapR posting YoY growth?

Two observations here:

  • The total number of enterprises measures in the hundreds of thousands. To say that our data suggests that 46% of them will be investing in Hadoop or are already hardly describes a tiny market. It’s not a “negative” statement. Only an accurate one – so far.
  • The growth of the Hadoop distributors is from a small base – they are now each claiming hundreds of customers. Our data tells us that many companies are using more than one, so some double counting occurs here. Also, there are differences in what constitutes a “customer.” That in no way diminishes their prospects; as we note in the report, “revenue growth for providers would appear to be in moving to larger deployments within their existing customer base,” and there is ample evidence of that. As with the installed bases of DBMSs, new name business is only a part of the opportunity, even at this very early stage of market development.

Do you expect to see the slowdown continue?

No. Part of the story here is in the maturity of the market as measured by last year’s Gartner Hype Cycle shown below. We’ve noted that Hadoop distributions are in the Trough of Disillusionment, a stage all emerging technologies must go through before leveling off into broad deployment, and that is a normal market development. Intent to purchase tracks this cycle and it’s due to accelerate as Hadoop moves to later stages. To get there, objections such as the ones we describe in the document and in other research will need to be overcome – mostly the issue of “I don’t see the value.”

Screen Shot 2015-06-03 at 6.23.55 PM

Getting Hadoop past the Trough requires two things: skills and applications. Despite investment in training programs, broad availability of Hadoop administration skills (value enablers) and analytics skills (value creators) is 2-3 years out. Don’t underestimate this uphill climb. Next, applications taking advantage of the Hadoop ecosystem need to emerge. To this point, the work done on Hadoop has effectively been artisanal data management – everything is built by hand. That’s fine if you’ve got the skills, but most enterprises don’t build software – they buy it. Hadoop can be a great place for those applications to reside.

Where do you see the greatest opportunities for emerging vendors in this space?

Both for distribution vendors and for players in adjacent spaces, the opportunities lie in communicating the business value of their technology. As we move to mainstream buyers (and that is another way to read the Hype Cycle – broad adoption happens after the early adopters are in – and they are now), the buyers are not after technology, but business value. They think differently, and they buy differently.

Moreover, these later stage prospects are looking for “enterprise class” features: fit and finish, ease of deployment, use and management, security, governance…. Opportunities abound, and Hadoop continues with the economic refactoring of the data asset. It’s no accident the big players are all in the game. In fact, their presence will have a big impact on the plans of prospects. The job of Sales begins after the first No.

Comments are closed


  • Sachin Ghai says:

    (1) I am hoping we don’t discount the partner ecosystem of Hadoop distros in this revenue analysis. Hadoop is as much a platform as much it has an overlap with database functions. It would be interesting to see revenue derived by partners from Hadoop and big data use cases versus traditional database use cases. Collective revenue of Hadoop distros plus partners on Hadoop use cases would complete the picture.
    E.g. Hadoop + DI; Hadoop + services to develop some tool(not just apps)

    (2) As Hadoop has moved from just hdfs and MR to ecosystem of tools and SQL becoming first class citizen, warehouse have converged towards the Hadoop based DW/data hub/data lake. It would be fascinating to see how many new warehouse or modernization plans do not have Hadoop in it or sold without Hadoop integration story.

    3) The next revenue frontier for Hadoop will be crossed when we see OLTP workloads and web apps integrating with Hadoop . With Hive crossing the transaction chasm and updates part of its query language, it is just a 2 year wait now for transactional systems to evolve on Hadoop and streaming workflows and not just RDBMS.

    Combine all these and 33b $ market by 2016 end may suddenly start looking different from an elephant ride view…

  • Merv Adrian says:

    Thanks for the comments.
    1) Related revenues get harder to measure as they move away form the primary vendor. Larger vendors do offer some vague discussion of related revenues, but it’s too early in the Hadoop ecosystem formic of that yet. Still, I agree that there will be considerable related revenue as the market matures.

    20 Messages have certainly advanced, and early release projects are in distributions (or not) for these things, I agree. Adoption so far is still very early.

    3) Will we see some transactional uses in 2 years? Sure; we’re seeing early event processing now.

    This is where you lose me. 33B is a big number when you start at $150M. 2 years? Not in anybody’s wildest dreams.

  • Tom Deutsch says:

    Merv – I’d suggest that getting past through this requires three, not two, things. I’d add methodology to this to help:

    1) Pick the right use cases
    2) Run and projects the right way (they are often different sort of projects) and often fall into Bimodal IT

    Great note and series, very helpful and good insight.

  • Since I am not a paying customer, I do not have access to “Survey Analysis: Hadoop Adoption Drivers and Challenges” to be able to analyze its methodology. Also, I should mention that I am a Hadoop Consultant and work for a company that is a partner of both Hortonworks and Cloudera, both companies with Hadoop distributions. All of my consulting experience has been with Fortune 100 clients, which seem to have the problem set sufficiently large to warrant the Hadoop investment. Every business needs a database, regardless of size. Not every business needs Hadoop. So should you really be using the DBMS marketplace as a yardstick? And how does your Gartner Research Circle compare in distribution by problem size to that of the potential Hadoop market? Can you even say what the potential Hadoop market is? Is the potential Hadoop market more like the Enterprise Data Warehouse market? Should you be comparing the Hadoop revenue to that of Oracle’s Data Warehouse products, instead of their entire suite of products, including relational databases, appliances, data warehouse, middleware, etc. Hortonworks and Cloudera make one product, their respective Hadoop distributions. How many products does Oracle market?

    • Merv Adrian says:

      Thanks for all the comments and questions, Robert. The RC is a highly representative sample. F100 companies would be classic early adopter candidates in most technology markets, so I’m not surprised your experience is not the same as that of the broader market. All businesses do indeed need to manage data; Hadoop vendors’ express positioning is that it will become a lower cost alternative to everything RDBMSs do, although when pressed they graciously acknowledge relational databases are better for some so far. The blog post comparison was not about functionality, though; it was simply a way to illustrate maturity and relative economic impact at a time when the hype is overblown.
      The rest of your questions, argumentative as they are, miss the point of the piece – you seem to be responding to assertions that are not made. Gartner does not believe there is a “Hadoop market;” it is one alternative platform in the information management space, and likely not the last one. You are correct to point out that Oracle (and its competitors) make many other software products apart from RDBMSs. We track them as well, but the numbers here were only for DBMS revenue and maintenance/support. Adding all those other revenue streams in would only make the disparity greater, but that was not our intent. Hadoop is not only pointing at DWs, and in fact early vendor rhetoric about outright replacement has given way to some more realistic co-existence and data lake positioning.
      So, summing up, I’m a little unsure what point you were trying to make. Mine was, I think, pretty simple – Hadoop is an interesting, growing market that today is a tiny fraction of the size of the closest analogous one, which it is positioned as competing with. Perspective is useful, and that was my objective: to provide some.