The failure to accurately predict the outcome of the elections has caused some backlash against big data and algorithms. This is misguided. The real issue is failure to build unbiased models that will identify trends that do not fit neatly into our present understanding. This is one of the most urgent challenges for big data, advanced analytics and algorithms. When speaking with retailers on this subject I focus on two important considerations. The first is that convergence of what we believe to be true and what is actually true is getting smaller.
This is because people, consumers, have more personal control than ever before. They source opinions from the web, social media, groups and associations that in the past where not available to them. For retailers this is critical because the historical view that the merchandising or marketing group holds about consumers is likely growing increasingly out of date. Yet well meaning business people performing these tasks continue to disregard indicators and repeat the same actions. Before consumers had so many options this was not a huge problem since change happened more slowly. Today if you fail to catch a trend there are tens or hundreds of other companies out there ready to capitalize on the opportunity. While it is difficult to accept, business people must learn a new skill, leveraging analytics to improve their instincts.
The second is closely related to the first but with an important distinction; go where the data leads. I describe this as the KISS that connects big data to decisions.
The KISS is about extracting knowledge, testing innovations, developing strategies, and doing all this at high speed. The KISS is what allows the organization to safely travel down the path of discovery – going where the data leads – without falling down a rabbit hole.
Getting back to the election prognosticators, there were a few that did identify the trend. They were repeatedly laughed at and disregarded. This is the foundation of the problem, organizations must foster environments where new ideas are embraced and safely explored. This is how we will grow the convergence of things we know.
Read Complimentary Relevant Research
Organizing for Big Data Through Better Process and Governance
With big data past the Peak of Inflated Expectations on the Hype Cycle, organizations are addressing next-level challenges and asking,...
View Relevant Webinars
Hadoop and Spark: Understanding Open Source Opportunities and Risks
As companies build foundational data and analytics infrastructure with Spark and Hadoop, the market continues to shift and evolve in...
Comments or opinions expressed on this blog are those of the individual contributors only, and do not necessarily represent the views of Gartner, Inc. or its management. Readers may copy and redistribute blog postings on other blogs, or otherwise for private, non-commercial or journalistic purposes, with attribution to Gartner. This content may not be used for any other purposes in any other formats or media. The content on this blog is provided on an "as-is" basis. Gartner shall not be liable for any damages whatsoever arising out of the content or use of this blog.