Big Data – Are we there yet…..?

Hearing  a question with the term “Big Data” in it is, for me is like traveling with my kids on a long road  trip and hearing ” are we there yet, are we there yet, are we there yet….!!!” . AAARRRRGGGGGGGHHHHH…..it makes you want to just pull over to the side of the road and say get out…!

 Now thankfully many execs I speak to are getting over the term as well, so when the term does come up – the eye rolling starts to happen not just from me. So this begs the question – when will we be ‘there’ when it comes to Big data..?

Governments have traditionally  been seen as laggards when it comes to adoption of new leading edge environments. Statistically when it comes to “Big Data”, a recent Gartner survey conducted by us with over 700 respondents globally shows that very well with only 16% of Governemnt respondent saying they have invested in “Big Data”, which as you can see by the Chart below is the furthest right column next to Utilities at 17%. Leading the pack, no surprise was Media and Communications followed by Banking.

The announcement by the Australian Government Information Management office(AGIMO) of their “Big Data Strategy” I believe indicates that we’ve arrived at a point where “Big Data” could be considered just the norm.  So thank you AGIMO. ‘Hey Kids – look it’s Wally World.’ Or should I say “Big Data World”.

 However , and don’t you hate howevers’, we’re not quite riding the “Big Data” roller coasters just yet, and I don’t think we’re even in the car park, rather we’re heading down the main road with “Big Data world” in the distance. What a strategy document like the one from AGIMO does is really helps provide the navigation for the last few streets to get us into the car park.

The AGIMO strategy papers outlines 6 principles: (Or Navigational elements that organizations public or private could learn from)

1.       data is a national asset (turn that into just Data is Asset)

2.       privacy by design

3.       data integrity and the transparency of processes

4.       skills, resources and capabilities will be shared

5.       collaboration with industry and academia

6.       enhancing open data

In no way does it provide us the answers, despite being an excellent navigational document. What it does do is opens the door on the debate on some really sticky issues such as privacy and ethics.

There are obvious and subtle privacy issues, but there are also large security benefits. Let’s say we can monitor all employee message traffic (e.g.: internal email and Facebook activity when logged on during office hours) and physical movements – we might be able to identify a potential security problem in the making. This sort of analysis has been pursued for years to detect people that might become violent in the workplace and law enforcement is taking similar approaches to perform triage for making the most effective resource deployment. Some would say hooray I’m being protected in the work place and I know I’m safe.

But, in that pursuit, we also learn about all sorts of irrelevant, but intensely private issues. Your religion, gender preference, the address of your therapist, the brand of shoes you prefer, the colour lipstick you wear, the hotel you frequent with a co-worker, etc. How comfortable are you with your employer knowing all this?

 So we have the sign posts to get us to the “Big Data” car park. Yet when we get there, we still have to find parking, we then have the long queues to get a ticket and then once you’re inside, some one will immediately want to go to the bathroom which will take you off track and delay you further from getting on that roller coaster.

 The question I would like pose is- why call it “Big Data” at all, what makes it big? Rather why not call it just “data” or “Information” as aren’t we just talking about different sources and extracting value from the combination of these sources? Aren’t we trying to find patterns to build models, identify risk, understand intent and sentiment and develop networks?

 When we conducted a “Big Data” survey in 2012, and asked which characteristic is the biggest issue for your organization – it wasn’t the volume of data that was the issue (which is really the ‘big’ part), it was the variety, such as video and audio, that dominated. In fact 50% of the respondents thought that was a bigger issue over Volume and/or Velocity of data so 2:1 thought variety more challenging. And, variety of data doesn’t need to be big.  Or is it that “Big Data” is just a catchy phrase that we’re never going to get rid of?

 So imagine you’ve bought your ticket, you’ve been to the bathroom and you’re now walking down main street of “Big Data World” and you look up to one of the CCT cameras’ and just then you get a text on your phone that says, “Please be advised, due to your recent heart condition, you will not be allowed access to Big Data Mountain ride. We do however have your favorite brand of herbal tea available at Big Data Café which is 200 meters to your right.”