Blog post

Big Data is Falling into the Trough of Disillusionment

By Svetlana Sicular | January 22, 2013 | 5 Comments

Local NewsinnovationInformation EverywherehadoopeventsEIMdata paprazziData and Analytics StrategiesCrossing the Chasmbig data marketanalytics"Data Scientist"

My presentation on big data for the upcoming BI Summit in Barcelona is obsolete. In this presentation, I use the Gartner Hype Cycle curve to show that big data is at the peak of inflated expectations.  And, as it happens with quickly developing technologies,  I am already behind and big data goes ahead.

Last several weeks show that big data is falling into the trough of disillusionment. I realized it earlier today, when I was describing a recent Elephant Riders meetup to my colleagues at Gartner. MapR, HortonWorks and Cloudera were debating the state of Hadoop.  And I heard from the very core of the Hadoop movement that MapReduce has always been Hadoop’s bottleneck or that Hadoop is “primitive and old-fashioned.”  This is the video of the event.  If you watch it, you can notice more points, which signal the beginning of disillusionment (and get a lot of useful information too).  Congratulations, big data technology is maturing fast!

Gartner Hype Cycle: Where is Big Data Now?
Gartner Hype Cycle: Where is Big Data Now?

Meanwhile, my most advanced with Hadoop clients are also getting disillusioned.  They do not realize that they are ahead of others and think that someone else is successful while they are struggling. These organizations have fascinating ideas, but they are disappointed with a difficulty of figuring out reliable solutions.  Their disappointment applies to more advanced cases of sentiment analysis, which go beyond traditional vendor offerings.  Difficulties are also abundant when organizations work on new ideas, which depend on factors that have been traditionally outside of their industry competence, e.g. linking a variety of unstructured data sources.  Several days ago, a financial industry client told me that framing a right question to express a game-changing idea is extremely challenging: first, selecting a question from multiple candidates; second, breaking it down to many sub-questions; and, third, answering even one of them reliably.  It is hard.

Formulating a right question is always hard, but with big data, it is an order of magnitude harder, because you are blazing the trail (not grazing on the green field). At the upcoming BI Summit in Barcelona, I will facilitate a user round table exactly about this — From “Satisficing” to Satisfying Business Requirements.  Validating answers is also a tough job  big data analytics deals with uncertainty: you do not deduct the number and say that the meaning of life is 42 — you get a proof of your hypothesis with a certain degree of confidence.  And it is up to you to decide what level of confidence is satisfying and what is “satisficing.” (A “satisficing” solution is the first solution that appears good enough.)

Back to the trough of disillusionment.  Or, rather, forward to the trough.  To minimize the depth of the fall, companies must be at a high enough (satisficing) level of analytical and enterprise information management maturity combined with organizational support of innovation.  Oops, I promised myself to be a reporter, not an analyst in my blogs.

The only consistent success, reported by my clients, is with log analysis using Splunk. Why? Because Splunk is a (nice) tool.  And plateau of productivity will be reached when tools and product suites saturate the market. Meanwhile, according to the Gartner Hype Cycle, the next stop for big data is negative press.  Does this blog post count as such?

 

Follow Svetlana on Twitter @Sve_Sic

Comments are closed

5 Comments

  • Greg says:

    Hi Svetlana
    Good article.
    We have a lot of success using Logscape for log analysis. It hits the spot the same way splunk does. It’s a nice tool that’s easy to use.
    http://www.logscape.com
    Greg

  • Rajesh Naik says:

    I would call this “commoditisation of Big Data Analytics” than the “trough of disillusionment”. A number of my customers, some of them belonging to the SME sector are adopting Big Data Analytics at a much faster rate than their bigger counterparts. And Big Data is certainly much more than Hadoop and MapR. Speaking to Cloudera a few weeks back, I was rightly told that Hadoop is what Oracle was in the 80s but it is maturing at a lightening speed. I couldn’t agree better.

  • Phil Simon says:

    Big Data is a big deal. Yes, many CIOs and thought leaders need time to get their arms around it.

    But, really, how is Big Data different in this regard than social media, cloud computing, ERP/CRM, etc? Companies that get on board early are more likely to see big rewards.

  • John Haddad says:

    Svetlana, thanks for pointing out some of the difficulties companies face with big data projects. After reading your blog I responded with my thoughts on this topic on the Informatica Perspectives blog, “How to Avoid the Big Data Trough of Disillusionment” (http://bit.ly/117GfZO). I gave my perspective as to why some big data projects fail to deliver on the promise we hear about in all the big data discourse and hype. I then describe what organizations can do to pass over the trough like a Lamborghini at 200mph.

  • Love the observation Big Data has certainly “Nuked the fridge” a point I made on the Syncsort blog back in April 2012 http://blog.syncsort.com/2012/04/has-big-data-nuked-the-fridge/
    I have to say it’s really great to see this from an analyst especially given the hype cycle – your spot on in pointing out that IT organisations are not satisfied with Hadoop progress and it is definitely being hyped but I would argue not over-hyped. Hadoop is an adolescent, disruptive technology that’s growing fast, but as a result experiencing growing pains – and is certainly not fully mature. The good news is that vendors and customers alike are recognising the potential of Hadoop and working to deliver tools to help ease the transition to maturity. My peers and I at Syncsort will continue to blog regularly in the coming weeks and months about what IT organizations are doing with Big Data and Hadoop and who is where in the maturity model of addressing Big Data. In fact, my colleague Jorge Lopez recently posted a blog that talks about 2013 being the year that Big Data gets traction and Hadoop ramps-up.