I love it when a “big data” hype fanatics say, “there is more and growing ‘dark data’ and it’s the biggest source of competitive advantage and business transformation!”. The implication is that you need to run hot foot to your nearest BI vendor and buy even more BI stuff. Really?
Dark data is the cute name given to all that data an organization gathers that is not part of their day to day operations. It is old stuff, stuff that turned up in the mail that you kept, ‘just in case’. It is data that you didn’t erase, because “it might come in handy some time”. There is, in most cases, more dark data than there is light data (I guess the data your organizations does use every day). Dark data is a storage vendors’ dream. All those archiving specialists and traditional Information Lifecycle vendors who, despite their name, do NOT look after the birth and life of data, but only at the death (and after life) of data, are jumping for joy. And the BI vendors are just behind it since you will need their tools to help mine that dark vault.
But there is a flaw in this hyped argument about dark data. Unless you, the business user, have an idea of what you want to ask of this dark data, there is no point worrying about it. Good ideas don’t just come out of the woodwork, or spring forth from a data mart. Business people have to have an idea, a question, an argument to test, a theory to explore, a posit to push against. Without this, all that dark data reminds of that extra furniture we all have in that dark cupboard at the top of the stairs. You know what I mean. We all keep that stuff, just in case. We only ever add to the stuff. We only ever remove (and trash) it when the cupboard becomes full and we can no longer afford to move house, just to collect more dark data…er…furniture and stuff.
So please, can we become a little more considerate for business value might be created before we turn on the light on all that big, dark, data? It might even be useful to think about this before we spend too much hard earned cash on managing that stuff that no one bothers to look at. Can we ask of ourselves, what kind of business related question or opportunity might possibly be impacted, and in what way, before opening up the cupboard door?