Gartner Blog Network


Big Data is Falling into the Trough of Disillusionment

by Svetlana Sicular  |  January 22, 2013  |  36 Comments

My presentation on big data for the upcoming BI Summit in Barcelona is obsolete. In this presentation, I use the Gartner Hype Cycle curve to show that big data is at the peak of inflated expectations.  And, as it happens with quickly developing technologies,  I am already behind and big data goes ahead.

Last several weeks show that big data is falling into the trough of disillusionment. I realized it earlier today, when I was describing a recent Elephant Riders meetup to my colleagues at Gartner. MapR, HortonWorks and Cloudera were debating the state of Hadoop.  And I heard from the very core of the Hadoop movement that MapReduce has always been Hadoop’s bottleneck or that Hadoop is “primitive and old-fashioned.”  This is the video of the event.  If you watch it, you can notice more points, which signal the beginning of disillusionment (and get a lot of useful information too).  Congratulations, big data technology is maturing fast!

Gartner Hype Cycle: Where is Big Data Now?

Gartner Hype Cycle: Where is Big Data Now?

Meanwhile, my most advanced with Hadoop clients are also getting disillusioned.  They do not realize that they are ahead of others and think that someone else is successful while they are struggling. These organizations have fascinating ideas, but they are disappointed with a difficulty of figuring out reliable solutions.  Their disappointment applies to more advanced cases of sentiment analysis, which go beyond traditional vendor offerings.  Difficulties are also abundant when organizations work on new ideas, which depend on factors that have been traditionally outside of their industry competence, e.g. linking a variety of unstructured data sources.  Several days ago, a financial industry client told me that framing a right question to express a game-changing idea is extremely challenging: first, selecting a question from multiple candidates; second, breaking it down to many sub-questions; and, third, answering even one of them reliably.  It is hard.

Formulating a right question is always hard, but with big data, it is an order of magnitude harder, because you are blazing the trail (not grazing on the green field). At the upcoming BI Summit in Barcelona, I will facilitate a user round table exactly about this — From “Satisficing” to Satisfying Business Requirements.  Validating answers is also a tough job  big data analytics deals with uncertainty: you do not deduct the number and say that the meaning of life is 42 — you get a proof of your hypothesis with a certain degree of confidence.  And it is up to you to decide what level of confidence is satisfying and what is “satisficing.” (A “satisficing” solution is the first solution that appears good enough.)

Back to the trough of disillusionment.  Or, rather, forward to the trough.  To minimize the depth of the fall, companies must be at a high enough (satisficing) level of analytical and enterprise information management maturity combined with organizational support of innovation.  Oops, I promised myself to be a reporter, not an analyst in my blogs.

The only consistent success, reported by my clients, is with log analysis using Splunk. Why? Because Splunk is a (nice) tool.  And plateau of productivity will be reached when tools and product suites saturate the market. Meanwhile, according to the Gartner Hype Cycle, the next stop for big data is negative press.  Does this blog post count as such?

 

Follow Svetlana on Twitter @Sve_Sic

Category: data-scientist  analytics  big-data-2  big-data-market  crossing-the-chasm  data-paprazzi  eim  events  hadoop-2  information-everywhere  innovation  local-news  

Tags: bi-summit  big-data  big-data-adoption  data-paprazzi  data-scientist-2  data-spy  end-users  hadoop  hadoop-distribution  information-everywhere  innovation  silicon-valley  vendors  

Svetlana Sicular
Research Director
1 year at Gartner
19 years IT industry

Svetlana Sicular has a uniquely combined experience of Fortune 500 IT and business leadership, product management at world-class software vendors, and Big Four consulting. She primarily handles inquiries in the areas of data management strategy, ...Read Full Bio


Thoughts on Big Data is Falling into the Trough of Disillusionment


  1. Greg says:

    Hi Svetlana
    Good article.
    We have a lot of success using Logscape for log analysis. It hits the spot the same way splunk does. It’s a nice tool that’s easy to use.
    http://www.logscape.com
    Greg

  2. […] yes, it does sound like a bunch of marketing people playing Dungeons and Dragons. Sicular posits in her blog that big-data is sliding past the top of that peak and into a period where people will start […]

  3. […] then 15 days later over at Gartner one of their analysts, Svetlana Sicular, has said that Big data is over the hype curve.  Cause and effect?  Probably not as if you know big data you know that correlation != […]

  4. […] is falling into the trough of disillusionment,” Gartner analyst Svetlana Sicular wrote in a blog post this week, referring to one of the stages on the “Hype Cycle” scale Gartner […]

  5. […] is falling into the trough of disillusionment,” Gartner analyst Svetlana Sicular wrote in a blog post this week, referring to one of the stages on the “Hype Cycle” scale Gartner […]

  6. […] was entering the part of the Gartner Hype Cycle known as the Trough of Disillusionment. Sicular posts in her blog that big data is sliding past the top of the peak and into a period where people will start […]

  7. […] reaches her conclusions about Big Data based on a series of conversations with IT professionals over the past few weeks, in […]

  8. […] reaches her conclusions about Big Data based on a series of conversations with IT professionals over the past few weeks, in […]

  9. […] to Gartner’s Svetlana Sicular, technologies typically follow a similar pattern, in which they are first discovered and then […]

  10. […] to Gartner’s Svetlana Sicular, technologies typically follow a similar pattern, in which they are first discovered and then […]

  11. […] thinks that the Big Data hype is going to die down a little for the lack of progress… (see here) Companies without web-scale, big, data are finding it hard to do anything commercially […]

  12. […] Gartner研究总监Svetlana Sicular近日在博客中发表一篇题为“大数据光环的幻灭”的文章,指出大数据正处于新技术厂商炒作周期和用户预期的顶峰(如下图),但是Hadoop等大数据技术的缺陷和用户失败案例的不断增多意味着大数据泡沫即将破裂。 […]

  13. […] analyst Svetlana Sicular argues in a blog post that Big Data may have reached that point. Her evidence is “hearing from people in the center […]

  14. […] analyst Svetlana Sicular argues in a blog post that Big Data may have reached that point. Her evidence is “hearing from people in the center […]

  15. […] analyst Svetlana Sicular argues in a blog post that Big Data may have reached that point. She has been “hearing from people in the center of […]

  16. […] analyst Svetlana Sicular argues in a blog post that Big Data may have reached that point. She has been “hearing from people in the center of the […]

  17. […] analyst Svetlana Sicular argues in a blog post that Big Data may have reached that point. She has been “hearing from people in the center of the […]

  18. […]    市场调查公司Gartner最新调查报告显示,“大数据”炒作已经接近预期最高值,“大数据泡沫”即将破灭。事实是否入此呢?分析师Sicular与多名IT专家进行了交谈,并与采用Hadoop分布式数据库框架(非常适合大数据应用)的大数据企业进行了圆桌会议,他们都同意大数据即将进入泡沫破灭阶段,市场即将开始淘汰落后企业。 […]

  19. […] Gartner have spotted the hype and note that Big Data has entered the Trough of Disillusionment. […]

  20. […] Big Data: According to Gartner the current hype will soon lead to disillusionment. This is caused not only by suppliers (”Big Data will solve all your problems and make you rich”). It is also caused by a common misunderstanding of the results of those projects. In the end you will get a framework to conduct statistical analyses. A company can only benefit of this if they don’t expect “yes” vs. “no” answers. But they need to be able to use probabilities, confidence intervals, and the testing of hypotheses. http://blogs.gartner.com/svetlana-sicular/big-data-is-falling-into-the-trough-of-disillusionment/ […]

  21. Rajesh Naik says:

    I would call this “commoditisation of Big Data Analytics” than the “trough of disillusionment”. A number of my customers, some of them belonging to the SME sector are adopting Big Data Analytics at a much faster rate than their bigger counterparts. And Big Data is certainly much more than Hadoop and MapR. Speaking to Cloudera a few weeks back, I was rightly told that Hadoop is what Oracle was in the 80s but it is maturing at a lightening speed. I couldn’t agree better.

  22. […] Big Data is Falling into the Trough of Disillusionment / Gartner […]

  23. […]    市场调查公司Gartner最新调查报告显示,“大数据”炒作已经接近预期最高值,“大数据泡沫”即将破灭。事实是否入此呢?分析师Sicular与多名IT专家进行了交谈,并与采用Hadoop分布式数据库框架(非常适合大数据应用)的大数据企业进行了圆桌会议,他们都同意大数据即将进入泡沫破灭阶段,市场即将开始淘汰落后企业。 […]

  24. […] as Hadoop are integrated into mainstream analytic applications. Gartner analyst Svetlana Sicular wrote that CIOs are finding it difficult to get value from Big Data software such as Hadoop, largely […]

  25. Phil Simon says:

    Big Data is a big deal. Yes, many CIOs and thought leaders need time to get their arms around it.

    But, really, how is Big Data different in this regard than social media, cloud computing, ERP/CRM, etc? Companies that get on board early are more likely to see big rewards.

  26. […] a recent blog, Sicular argued that big-data has reached the peak of "Inflated Expectations" and is heading toward […]

  27. John Haddad says:

    Svetlana, thanks for pointing out some of the difficulties companies face with big data projects. After reading your blog I responded with my thoughts on this topic on the Informatica Perspectives blog, “How to Avoid the Big Data Trough of Disillusionment” (http://bit.ly/117GfZO). I gave my perspective as to why some big data projects fail to deliver on the promise we hear about in all the big data discourse and hype. I then describe what organizations can do to pass over the trough like a Lamborghini at 200mph.

  28. […] 'UA-22979035-1', data_ga_social: true }; According to Gartner, big data has now reached the peak of inflated expectations in its hype-cycle model, and is falling into the ‘trough of disillusionment’—a […]

  29. Love the observation Big Data has certainly “Nuked the fridge” a point I made on the Syncsort blog back in April 2012 http://blog.syncsort.com/2012/04/has-big-data-nuked-the-fridge/
    I have to say it’s really great to see this from an analyst especially given the hype cycle – your spot on in pointing out that IT organisations are not satisfied with Hadoop progress and it is definitely being hyped but I would argue not over-hyped. Hadoop is an adolescent, disruptive technology that’s growing fast, but as a result experiencing growing pains – and is certainly not fully mature. The good news is that vendors and customers alike are recognising the potential of Hadoop and working to deliver tools to help ease the transition to maturity. My peers and I at Syncsort will continue to blog regularly in the coming weeks and months about what IT organizations are doing with Big Data and Hadoop and who is where in the maturity model of addressing Big Data. In fact, my colleague Jorge Lopez recently posted a blog that talks about 2013 being the year that Big Data gets traction and Hadoop ramps-up.

  30. […] several weeks show that big data is falling into the trough of disillusionment,” wrote Gartner research director Svetlana Sicular in a January 22 blog […]

  31. […] credit: Wikipedia)Last week, a Gartner investigate executive admitted that large information is entering a tray of disillusionment. How tiresome!What a hype cycle describes is a wearied tech news media. Big information was final […]

  32. […] BD’s outlook is presently situated, and reveals that many among her NoSQL clientele have exited their honeymoon periods with the promising technology, and are coming up against the real-world […]

  33. […] few weeks, in addition to a roundtable with Hadoop vendors Cloudera, Hortonworks, and MapR. She writes: I heard from the very core of the Hadoop movement that MapReduce has always been Hadoop’s […]



Comments are closed

Comments or opinions expressed on this blog are those of the individual contributors only, and do not necessarily represent the views of Gartner, Inc. or its management. Readers may copy and redistribute blog postings on other blogs, or otherwise for private, non-commercial or journalistic purposes, with attribution to Gartner. This content may not be used for any other purposes in any other formats or media. The content on this blog is provided on an "as-is" basis. Gartner shall not be liable for any damages whatsoever arising out of the content or use of this blog.