Blog post

What’s Next for Big Data?

By Svetlana Sicular | December 23, 2014 | 4 Comments

Inquire WithininnovationInformation Everywherehadoopdata paprazziData and Analytics Strategiesdataanalytics"Data Scientist"

Shortly after joining Gartner, I noticed subtle magic in the air. Not Magic Quadrants – they were too obvious. It was I told you so repeated by many analysts on many occasions. It seemed almost mystical – how could they know? At some point, I caught myself saying I told you so too — it became natural after seeing and researching so much. For example, I talked about personal analytics, multidisciplinary teams instead of a single data scientist, and about Hadoop being a live archive — these came to fruition and became common place. I told you so. (Mom, I hate when you say it.)


In the big data field research back in 2012, we saw that there was a big data maturity gap. It needed a couple of years to close. I told you so. Glad to report, 2014 was the first year when enterprises became serious about big data: They started asking questions beyond “how do I begin my big data initiative?” or “how do I select a Hadoop distribution?” Hortonworks even went public, the first Hadoop vendor to do so. My colleague Merv Adrian went into great depth on the Hortonworks IPO in his blog post Hortonworks IPO – Why Now?

The question is — what’s next for big data? First of all, big data will become the new normal sometime between 2016 and 2018. My colleagues Donald Feinberg and Mark Beyer will say (with well-earned pride and a flair of mysticism), I told you so.

Organizations are finally ready for big data in the cloud. In the second half of 2014, my clients started asking about a data warehouse and Hadoop in the cloud. I am an analyst in Gartner for Technical Professionals — 90% of our clients are practitioners who are doing things right now. Therefore, I anticipate many interesting developments around big data in the cloud soon.

In 2015, I expect a plethora of big data applications on top of data platforms (remember, these platforms already demonstrate acceptable maturity). Big data applications will be mostly analytical, and they will be small, in the “app store” style, with few customizations — that makes support and maintenance relatively easy. People would be able to download big data apps they need and use them like Lego blocks to make their own customizations. Big data apps will put a process or a workflow into the spotlight.

If big data apps proliferate, they will need… data. This means a focus on data governance, data preparation and an ability to painlessly load data into big data stores. This also means self-service, or rather SELF-SERVICE. It would be a bigger and bigger subject from the data management and analytics perspectives. And from the process perspective too, I already told you so.

People are impatient. Those who want self-service are demanding increasingly real-time data access, response and gratification. This will lead to in-memory and streaming advances, but I don’t think it will be next: for practitioners, it will be next after next.

The Internet of Things (IoT) is at the peak of inflated expectations of the hype cycle. Organizations collect more data than they can process: for them, it’s still not just about finding the needle, but getting the hay in the stacks. The Internet of Data sustains the IoT. Companies are collecting new data, asking about new external data sources and searching for dark data within. Many new data sources are personal data – privacy and ethics accompany them. This year, I wrote Maverick* Research: Put Your Data in the Bank, Get Dividends, where I foresee intermediaries — personal data banks — representing individuals. A personal data bank will keep deposited data and multiply its wealth through commercialized digital markets. The Internet of Data and personal data banks are not reality yet, but I am sure soon I would be able to say about them I told you so.


The Internet of Data sustains all other Internets
The Internet of Data sustains all other Internets


Follow Svetlana on Twitter @Sve_Sic

Comments are closed


  • As a veteran of the BI space, I have mined the ebb and flow of impatience relative to data access. Twenty five years ago the data was kept in departments and reported on by the business. This meant there was no streaming or machine generated data available as it wasn’t accessible to the business. It doesn’t mean it didn’t exist. It did, and no one was doing anything with it except the odd report if something failed or needed attention.

    Today there is a vibrancy about the need for real time data, but the preponderance of the data consumption is still periodic and analyzed off-line. As the two segments catch up to each other – data and people – they will find that it is not about the data scientists, but about the everyday people who know their core focus area and are not ready to analyze the IoT and IoD. They are focused on the IoP and productivity in real time; they are making micro-decisions that improve business processes. They don’t read graphs. They don’t perform calculations. They know thresholds and target KPIs. They don’t need BI, they need a performance monitoring system that enables them to make decisions – based on visual cues. Enabling a worker to visualization a process and visual function cues to help them use all the relevant data to improve performance before it becomes a problem.

    VisualCue is the first product to market allowing businesses to include a class of data consumer in the decision-making process who was previously completely overlooked. Bringing the capacity to visualize more than 5 metrics simultaneously to an hourly worker who can react in real time is a performance breakthrough. This heralds a new level of data consumption in the enterprise and will be the inflection point of data understanding at all levels for context-sensitive decision-making. Finally, an “I told you so” about who the data consumer really is – everybody.

  • This is so true and predicted in our book; read inside the book!!

  • Context is the essence of the Internet of Analytics. This will be the glue that binds People, Ideas, Data, and Things into the fabric of communities and will be at the core of individuation.