Four Causes of Information Governance Failures

So, before you start reading the main part of this, I need to say:  although it may not seem like it at first, this post is about information governance.  Stay with me, I’ll get there in the end.

 Ready?  The is the answer to ‘why is information governance so hard’?

 In his book Collapse, Jared Diamond chronicles the breakdown and eventual extinction of colonies, societies and whole civilizations.   Diamond does case studies of a number of these, already extinct, but the book begins with a discussion of a set of contemporary issues which center on the Bitterroot Valley in Montana.  Long story short, everyone he describes are acting from a set of motives that mix the good of community with self interest.  What is at stake is the economic viability of the area, which is often, well, almost always, a short term consideration.  People need jobs now.  People need money now. 

The opposing set of interests and motives center around the long term ecological fate of the area.  Many of the issues that face the Bitterroot Valley have to do with water, which is a scare and precious resource.  What usually happens is that short term considerations win out over longer term ones and that it is very difficult to get people (companies or even governments) to sign up to long term plans that sacrifice short term goals. That’s not quite true:  they will sign up.  They just won’t execute when the time comes. 

Diamond’s book was published in 2005, so while writing this, I Googled (how did we LIVE before Google?) the Bitterroot Valley and ended up looking a story from January 14, 2010: (http://www.missoulian.com/news/local/article_7e7158dc-019e-11df-92ad-001cc4c03286.html) which demonstrates that the same set of forces are still in play.

I’m not picking on Montana, the Bitterroot Valley or any of the good people there.  I’m simply noting that in the course of human affairs, today’s concerns almost always take precedence over the concerns of next week, next month or next year, never mind years into the future.  Although we worry about the future for our children, many of us have not yet met our grandchildren and certainly only very few of us will live to see great grand kids born.  (To be honest, I wouldn’t want to:  ‘Great Grandma?’  Arrrrggghhh)  Anyway.

 Diamond talks about a number of reasons why we behave this way, which he argues were the cause of the ultimate collapse of these now extinct groups, societies and civilizations.  We are not facing anything as serious as ‘Collapse’, but we have had some spectacular failures in managing and acting on information in the recent past, and this will only get worse, unless we choose to do something about it.

While reading these causes as detailed in Collapse – or rather rereading them – its struck me that these are the very same reasons as to why we (in corporations, government and even personally) are really not all that interested in information governance or if we are, have a difficult time in making it work.  These failures are social, rather than technological. 

The first reason that many of us have an information governance (or information overload) problem is that we failed to see it coming due to lack of  prior experience.  The revolution in information technology that we have experienced over the last 25 to 30 years is unprecedented and has changed the way we live and work.  One of the questions that I always get asked is ‘how much…?’  Well, if you must know, according to a report released late last year:

 In 2008, Americans consumed information for about 1.3 trillion hours, an average of almost 12 hours per day. Consumption totaled 3.6 zettabytes and 10,845 trillion words, corresponding to 100,500 words and 34 gigabytes for an average person on an average day. A zettabyte is 10 to the 21st power bytes, a million million gigabytes. These estimates are from an analysis of more than 20 different sources of information, from very old (newspapers and books) to very new (portable computer games, satellite radio, and Internet video). Information at work is not included.The full report is available online at http://hmi.ucsd.edu/howmuchinfo.php.

To be honest, I think this kind of analysis is meaningless in real terms and not just because it doesn’t include what we do at work.  Its meaningless because no one has any idea (really) of the true nature of a zettabyte (a word that my spell checker doesn’t even seem to know).  In order to make it real, we need to look at it in terms of things we understand, like how much work we can get done in a day with what tools against what goals.  There is only so much that one person can do in a day or even in a lifetime.  A zettabyte sounds to me like more than one lifetime’s worth of work, in fact.  Masses of information tend to work against us, especially if they are unorganized.  The answer is not more.  Its less.

 Robert Jarvis makes this point in an article in the Boston Globe, discussing the recent failure of the U.S. intelligence service to pick up data points and then connect the dots regarding the attempted terrorist act  on Flight 253.  He says that in intelligence work as in life, the problem isn’t too little information, it is too much and even though we have too much, its not perfect and can be interpreted in various ways.  His recommendation is to craft policies, institutions and analytical habits that compensate for our universal human biases.  His recommendation is NOT to gather  more information and better technology to analyze it.

 In IT circles, we talk about  Moore’s Law and Kryder’s Law (about how storage technology keeps getting denser and cheaper) and relied on them to keep us out of trouble.  IT and business people have always assumed that computers would get faster and faster, and that storage would get cheaper and cheaper and that’s how we’d keep up. But that’s not keeping up.  Keeping up implies that the information that we can generate and store can be used – and reused – that it generates some value.  That is what is difficult and that is what we are failing to do, because of its sheer volume and because of our lack of attention to its governance.  Our old assumptions, that technology would solve the problems that technology created, have now stopped working.

The second reason we have not solved the problem is that we fail to perceive it at all. Data and information accumulated, day by day, on our hard drives and in our systems and on a daily basis it didn’t look like all that much.   Then one day, seemingly suddenly, there were terabytes of the stuff, storage was costing a fortune and the regulatory climate had grown increasingly aggressive and sometimes hostile.  It is a problem that only looks obvious in hindsight.  We don’t notice gradual change, because gradual change is normal.  Its like watching your own kids grow up, versus seeing someone else’s that you hve not seen for a while.  In the latter case you remember a four year old and a fourteen year old stands in front of you, seemingly suddenly.  That change was gradual and invisible, while your own children grew gradually but visibly.  So we’re used to information overload and paying lots for storage.  It seems normal and necessary, simply part of the way things are.  Then someone happens – we fail an audit, there’s a lawsuit or an economic crisis forces a rethink of all budgets in the company – and we are forced to look at the problem in a whole new light.  The question of ‘How did THIS happen’? is the one most frequently asked in this situation.  This is even truer of high level managers, who simply stated, do not even  have information management problems.  They have personal assistants who do things for them and get us to do things for them.  IT leaders understand the problem  – it impacts their budget – and users understand the problem – they never see to be able to find what they want – but managers often don’t.  Their failure to perceive is a huge handicap when it comes to getting high level support (and money) to solve the problem.

Third, and this reason is probably the most important one for our purposes, we fail to act on a problem once we have noticed it.  Different groups and individuals have different views of information management.  Users do not want to create more process discipline around the way they create and manage information, nor are they willing to look at what has accumulated and decide whether or not it has value.  That’s taking time away from the things that they need to do to get their jobs done, their objectives met and their promotions secured. There is a clash of interest between corporate interests at the highest level, business users interests and the interests of IT.  Group interests, even within work group and project teams, clash with individual interests.  There is a need for collective action on information governance to take place:  high level executive support, business user buy in and IT advice and implementation.  Information governance is the effective regulation over the use of common resources in a way that satisfies, but not completely, the competing interests of different groups.  People are engaging in perfectly rational but ‘bad’ behavior, because it allows them to put their own self interests first.  Incentives for good behavior need to be implemented, otherwise, nothing will change.

 Diamond’s final cause is that we simply do not have the means available to solve whatever problem it is that we are looking at.  I don’t think this is true in the case of information governance, but I do think that many organizations look in the wrong place for that solution.

So what’s the right one?