To paraphrase (ok, misquote completely) Ronald Coarse, “If you aggressively interrogate the data for long enough, it will confess everything and anything”.
And yet, the inherent dangers of overly rigorous analysis are often completely ignored. Certainly, the phenomena of analysis paralysis is debated frequently. But the potential dangers associated with premature conclusion are rarely considered. Far better perhaps to consider the minimalist viewpoint of Ludwig Mies van der Rohe…
Less is more – Particularly when it comes to presumptions and preconceptions…
If you are searching for a “smoking gun” that you already “know” exists, then don’t be surprised when you find it. By scouring the horizon with your high powered analytical telescope, you have unwittingly blinkered yourself. Your peripheral vision is impaired. Sure, you will eventually find a “smoking gun”, but is it THE smoking gun? Indeed, was there just ONE smoking gun in the first place? Perhaps a multitude of “lone” gunmen were involved? But perhaps no gunmen (“lone” or otherwise) played any part in the proceedings at all? Perhaps your particular lone gunman shot someone or something else and you just haven’t found the body yet? Maybe it was a superficial flesh wound and your systems are limping along despite the gunman’s actions? Perhaps the fatal wound was inflicted by another weapon classification all together? Perhaps a combination of seemingly unrelated factors conspired to cause the issue? Conspiracy theorists can’t all be nuts can they? Who knows? One thing is for sure, if you enter into the analytical process with too many preconceptions and hypotheses then you may arrive at a conclusion that fits the evidence and expectations you have rather than at a solution that fits for all possible scenarios.
Less is more – Especially when considering the practicality of your n-dimensional matrix
Too much data can be as much of a hindrance as it can be a help. N-dimensional multi-variate analysis is a complex and resource intensive task at the best of times. Throwing too many variables into the mixing pot is no guarantee of developing a palatable recipe for prediction. You may get lucky and stumble across a fusion of flavours that works. Or you may never find the appropriate combination and proportions for culinary success. In analytics, as in cuisine, it is the quality of your ingredients that matters – not just the quantity. Understanding the provenance of your data, its freshness and its variability will help you to prepare it better which will ultimately help you to cook up a much better analytical dish. Ingredients matter. The recipe matters. The skills of the chef matter. But all of these things are for nothing if the chef fails to continuously validate their actions. The best chefs sample their food frequently during the cooking process, you should do the same. If it tastes bad or funny, it probably is… Don’t waste your time trying to make a bad dish taste better. At best, you will only mask the flavors and the inherent sourness will always come through. Go back to the recipe and begin refining it once more.
Less is more – Providing accuracy versus timeliness trade offs maximize benefits
Many providers are overly concerned about accuracy. They believe that they must be able to predict every eventuality and corner case for their models to be useful. This is absolute nonsense. Being able to predict the most common forms of system failure with a reasonable degree of certainty is far far more useful than being able to predict every failure with questionable accuracy. Pragmatism is a vital component to all analytical endeavours. All too often the statisticians, data miners and analytical modellers forget this. When you do hit upon a model that works reasonably well for some of the issues your customers face – use it! By all means refine it over time but please please please do not leave it sitting on the shelf gathering dust until you manage to solve the ultimate problem. You may never manage it. That’s not because you aren’t bright enough or don’t have enough of the “right” data. Although both of these may hinder your progress. It could be, and most probably is, because there is no tangible link between every failure mode and every conceivable influencing factor. There is no “uber” solution. Get over it. Move past that. Begin embracing the practical solutions that can and do answer the questions that need to be answered.
Less is more – Time spent at the periphery will pay dividends in many ways
Don’t over think it. Keep your eyes and ears open. But more importantly, always try to keep your mind open to possibilities and solutions that you haven’t yet conceived. This is not easy. Far from it. But a good starting point is to regularly stop focusing on the issue in hand and to actively stop looking and try to begin seeing. See what there is to be seen. Take time to step back and enjoy the view. Breathe in the surroundings and allow the scenery to work its way into your subconscious. It could be the one thing that helps you to crack even the toughest of analytical nuts. Maybe not today, but perhaps one day the knowledge you have gained from this intellectual diversion will fall into place and unlock untold secrets with unexpectedly positive results. Then again, maybe it won’t. But that’s kind of the point. If you never step back and consider all possibilities and eventualities then you will definitely never find the unexpected serendipitous solution.
Less obsession on data volumes and accuracy is more likely to delivery tangible business orientated results…
Sounds simple when I say it like that doesn’t it? Perhaps the hype inflated big data balloon will burst on the back of this post? Nah. That’s just wishful thinking on my part. I know that this blog entry will have minimal impact on the industry as a whole. But that doesn’t mean it isn’t worth saying. And in saying it, perhaps I am redressing the balance in favor of common sense and reason a little? Here’s hoping!
Minimalism on it’s own is not enough. Pragmatism is its essential companion for success. As Agent K in the Men In Black series says to Agent J when explaining the need to temporarily disconnect from the matter in hand to gain a broader perspective… “You have to trust the pie.”