Two Main Questions about Big Data

More and more people describe the speed of big data technologies in dog years. In the lightning-fast cadence (“fail fast”), considerations for actual technical professionals who are deploying big data solutions fall into cracks. Vendors look at abstract “customers.” Big data ecosystem talks about “data scientists.” A new term “data engineer” is bubbling to the surfaces of data lakes. And real people — information architects, software engineers, managers of analytics teams — are coping with uncertainty. They don’t know about each others’ uncertainty; they think that they are dealing alone with rapid changes, brittle solutions and permanent immaturity brought by a yet one more promising technology. And they all ask the same two questions:

• How do we bridge our information silos?
• How do we get to the data faster?

In essence, two main questions are about data availability. In some cases, big data technologies are the answer, and in some cases, they are not. It depends… on governance. Big data technologies are capable of solving both questions when they are implemented under governance that supports new solutions.

Gartner actively promotes the idea of bimodal IT. Mode 1 (“marathon”) embodies stability and long-term planning; Mode 2 (“sprint”) embodies agility and quick results. Bimodality is necessary to survive in the digital economy. Often, albeit not always, the traditional and big data technologies are divided along the lines of marathon and sprint. The “sprint” mode can address both key questions — bridging information silos and getting to the data faster — when companies implementing new technologies consciously plan bimodality.

Let’s take an example of the EDW and Hadoop.

Sometimes, the results are achieved in Hadoop because of pure technology, but sometimes because Hadoop is not as strictly governed as the EDW. And it is okay to let people loose in Hadoop (or any other technology for that matter) as long as you are conscious of the differences in governance, and moreover, as long as you know how to switch information governance from sprint to marathon when your new solutions become mission-critical.

I wrote a new research note Upgrade the Enterprise Data Warehouse Architecture With Hadoop for actual technical professionals who are asking these two questions:

• How do we bridge our information silos?
• How do we get to the data faster?

In the last question, “faster” often means “sooner.”

Follow Svetlana on Twitter @Sve_Sic