Blog post

Big Data – Are we there yet…..?

By Ian Bertram | August 25, 2013 | 2 Comments

Hearing  a question with the term “Big Data” in it is, for me is like traveling with my kids on a long road  trip and hearing ” are we there yet, are we there yet, are we there yet….!!!” . AAARRRRGGGGGGGHHHHH… makes you want to just pull over to the side of the road and say get out…!

 Now thankfully many execs I speak to are getting over the term as well, so when the term does come up – the eye rolling starts to happen not just from me. So this begs the question – when will we be ‘there’ when it comes to Big data..?

Governments have traditionally  been seen as laggards when it comes to adoption of new leading edge environments. Statistically when it comes to “Big Data”, a recent Gartner survey conducted by us with over 700 respondents globally shows that very well with only 16% of Governemnt respondent saying they have invested in “Big Data”, which as you can see by the Chart below is the furthest right column next to Utilities at 17%. Leading the pack, no surprise was Media and Communications followed by Banking.


The announcement by the Australian Government Information Management office(AGIMO) of their “Big Data Strategy” I believe indicates that we’ve arrived at a point where “Big Data” could be considered just the norm.  So thank you AGIMO. ‘Hey Kids – look it’s Wally World.’ Or should I say “Big Data World”.

 However , and don’t you hate howevers’, we’re not quite riding the “Big Data” roller coasters just yet, and I don’t think we’re even in the car park, rather we’re heading down the main road with “Big Data world” in the distance. What a strategy document like the one from AGIMO does is really helps provide the navigation for the last few streets to get us into the car park.

The AGIMO strategy papers outlines 6 principles: (Or Navigational elements that organizations public or private could learn from)

1.       data is a national asset (turn that into just Data is Asset)

2.       privacy by design

3.       data integrity and the transparency of processes

4.       skills, resources and capabilities will be shared

5.       collaboration with industry and academia

6.       enhancing open data


In no way does it provide us the answers, despite being an excellent navigational document. What it does do is opens the door on the debate on some really sticky issues such as privacy and ethics.


There are obvious and subtle privacy issues, but there are also large security benefits. Let’s say we can monitor all employee message traffic (e.g.: internal email and Facebook activity when logged on during office hours) and physical movements – we might be able to identify a potential security problem in the making. This sort of analysis has been pursued for years to detect people that might become violent in the workplace and law enforcement is taking similar approaches to perform triage for making the most effective resource deployment. Some would say hooray I’m being protected in the work place and I know I’m safe.

But, in that pursuit, we also learn about all sorts of irrelevant, but intensely private issues. Your religion, gender preference, the address of your therapist, the brand of shoes you prefer, the colour lipstick you wear, the hotel you frequent with a co-worker, etc. How comfortable are you with your employer knowing all this?

 So we have the sign posts to get us to the “Big Data” car park. Yet when we get there, we still have to find parking, we then have the long queues to get a ticket and then once you’re inside, some one will immediately want to go to the bathroom which will take you off track and delay you further from getting on that roller coaster.

 The question I would like pose is- why call it “Big Data” at all, what makes it big? Rather why not call it just “data” or “Information” as aren’t we just talking about different sources and extracting value from the combination of these sources? Aren’t we trying to find patterns to build models, identify risk, understand intent and sentiment and develop networks?

 When we conducted a “Big Data” survey in 2012, and asked which characteristic is the biggest issue for your organization – it wasn’t the volume of data that was the issue (which is really the ‘big’ part), it was the variety, such as video and audio, that dominated. In fact 50% of the respondents thought that was a bigger issue over Volume and/or Velocity of data so 2:1 thought variety more challenging. And, variety of data doesn’t need to be big.  Or is it that “Big Data” is just a catchy phrase that we’re never going to get rid of?

 So imagine you’ve bought your ticket, you’ve been to the bathroom and you’re now walking down main street of “Big Data World” and you look up to one of the CCT cameras’ and just then you get a text on your phone that says, “Please be advised, due to your recent heart condition, you will not be allowed access to Big Data Mountain ride. We do however have your favorite brand of herbal tea available at Big Data Café which is 200 meters to your right.”

The Gartner Blog Network provides an opportunity for Gartner analysts to test ideas and move research forward. Because the content posted by Gartner analysts on this site does not undergo our standard editorial review, all comments or opinions expressed hereunder are those of the individual contributors and do not represent the views of Gartner, Inc. or its management.

Comments are closed


  • Rob says:

    It is only recently that we started getting effective in geometric pattern recognition. This, with voice recognition of natural language, is opening up new opportunities in a host of scenarios.

    “big data” is just the necessary word to explain complexity to golf-carted CEOs 🙂

  • Data 101 says:

    Ian, nice article on Big Data. With the explosion of big data, companies are faced with data challenges in three different areas. First, you know the type of results you want from your data but it’s computationally difficult to obtain. Second, you know the questions to ask but struggle with the answers and need to do data mining to help find those answers. And third is in the area of data exploration where you need to reveal the unknowns and look through the data for patterns and hidden relationships. The open source HPCC Systems big data processing platform can help companies with these challenges by deriving insights from massive data sets quick and simple. Designed by data scientists, it is a complete integrated solution from data ingestion and data processing to data delivery. Their built-in Machine Learning Library and Matrix processing algorithms can assist with business intelligence and predictive analytics. More at