Guest Post: Big Data is Falling into the Trough of Disillusionment

From my esteemed colleague Mark Beyer, the co-lead for Gartner big data research.

The intent of Gartner’s advice is to restore order while we provide guidance to a panicking throng scrambling to buy limited edition big data toys. Gartner has advised caution to the market. But, panic in the streets and chaos often drowns out reasonable advice. Gartner believes big data is headed for the Trough of Disillusionment. This has given rise to cries of “foul” and “wrong”. There is an amusement factor here. Overall, it is proof that some vendors simply do not understand exactly what the Trough means. Even more fun, is that Gartner’s presumptive market pundit competitors don’t understand market dynamics as well as the Hype Cycle portrays. Perhaps some remedial education is in order. Gartner’s hype cycle keeps proving itself year after year (Yes, it is Gartner’s trough and Our Hype Cycle. Effectively, all those nay-sayers don’t get votes anyway).

But, first things first. To our end-user and professional organizations, we say simply “Implementers take heed!” Both System Integrators and employee internal developers alike should know how hype and Our Hype Cycle work. The Trough means that market dynamics have changed. There are false claims of simplicity and promises beyond reason which should be carefully vetted and even ignored in favor of maturing solutions. It means that among the honest vendors, a few charlatans selling “cheap knock-off merchandise” from the back of their shady parking lot truck have arrived. Legitimate vendors and legitimate solutions will be confounded by these itinerant peddlers who offer lower prices for shiny trinkets that glitter then break under the first strain. As the Trough deepens, everyone will start to think the practices and solutions are failures because of this unwarranted dilution in the market offerings. And then the very real and highly valuable progress big data contributes to IT and computer science could be lost. It is the job of the hype cycle to warn implementers to be wary of poor solutions — but to NOT give up hope.

Gartner is not saying big data is dead, or gone. To the contrary — we say it becomes the new normal and does so somewhere between 2015 and 2017.

For those vendors that proffer legitimate solutions, we call them to join Gartner to help big data mature into normal IT practices. Experienced market vendors and implementers KNOW what it takes for a solution to mature and reach enterprise capacity. When the market starts to reach 15-20% adoption, then big data will have reached the Plateau, that’s the end of “hype” and the beginning productivity. So, for something to move into the Trough is a maturation process. Implementers and organizations will begin to choose the winning solution architectures and technologies that support them. The chaff will be reduced away from the wheat. By the way, the definition of hype is over-promising without a basis of market experience and proof. The Trough is what does that. So, were I vendor offering a solution, I would be GLAD to see the Trough arrive. My confidence in my own engineering and practices will prove out — in the Trough. Then it will rise along the Slope of Enlightenment while others drop by the way side. You see, if big data were a red-cross dot on the hype cycle, that would mean it is doomed to never reach maturity. But big data is not a red-cross dot. Hmmm? So, when you hear the salesman on the street offering some new, cheap big data trick — give us a call. We’re already hearing where organizations are happy with their solutions and from those who are getting incomplete solutions. Of course, Gartner is not the only place to get answers — but we talk to literally thousands of end-users and implementers every quarter.

And, our customers pay us to tell them the good, the bad and the ugly. “Psst! You wanna buy a big data watch?”