Gartner Blog Network

Lessons From The U.S. Election On Big Data And Algorithms

by Robert Hetu  |  November 10, 2016  |  2 Comments

The failure to accurately predict the outcome of the elections has caused some backlash against big data and algorithms. This is misguided. The real issue is failure to build unbiased models that will identify trends that do not fit neatly into our present understanding. This is one of the most urgent challenges for big data, advanced analytics and algorithms.  When speaking with retailers on this subject I focus on two important considerations.  The first is that convergence of what we believe to be true and what is actually true is getting smaller.

Gartner 2016

Gartner 2016

This is because people, consumers, have more personal control than ever before.  They source opinions from the web, social media, groups and associations that in the past where not available to them.  For retailers this is critical because the historical view that the merchandising or marketing group holds about consumers is likely growing increasingly out of date.  Yet well meaning business people performing these tasks continue to disregard indicators and repeat the same actions.  Before consumers had so many options this was not a huge problem since change happened more slowly.  Today if you fail to catch a trend there are tens or hundreds of other companies out there ready to capitalize on the opportunity.  While it is difficult to accept, business people must learn a new skill, leveraging analytics to improve their instincts.

The second is closely related to the first but with an important distinction; go where the data leads.  I describe this as the KISS that connects big data to decisions.

Gartner 2016

Gartner 2016

The KISS is about extracting knowledge, testing innovations, developing strategies, and doing all this at high speed.  The KISS is what allows the organization to safely travel down the path of discovery – going where the data leads – without falling down a rabbit hole.

Getting back to the election prognosticators, there were a few that did identify the trend.  They were repeatedly laughed at and disregarded. This is the foundation of the problem, organizations must foster environments where new ideas are embraced and safely explored.  This is how we will grow the convergence of things we know.

Additional Resources

100 Data and Analytics Predictions Through 2024

Gartner’s annual predictions disclose the varied importance of data and analytics across an ever-widening range of business and IT initiatives. Data and analytics leaders must consider these strategic planning assumptions for enhancing their vision and plans.

Read Free Gartner Research

Category: data-and-analytics-strategies  

Tags: advanced-analytics  algorithms  big-data  innovation  retail  

Robert Hetu
VP, Analyst Retail
7 years at Gartner
29 years IT Industry

Bob Hetu is a Research Director with the Gartner Retail Industry Services team. His responsibilities involve tracking the technology markets and trends impacting the broad-based retail merchandising and planning areas. Mr. Hetu is an expert in the areas of brand, vendor and assortment management, merchandise planning, allocation, and replenishment. Read Full Bio

Thoughts on Lessons From The U.S. Election On Big Data And Algorithms

  1. Sing Koo says:

    I fully agree with what is said here. When one derived an outcome from analytics that is not consistent with the prevailing view, it remain buried. Our company developed an unstructured text analytics to process the debate exchanges between the president candidates. We used a proprietary technique of sentiment analysis and communication alignment analysis resulting in a prediction of winning edge for Trump over Hillary. Since it is not the prevailing conclusion back in Oct., our finding was never made it to the main stream media news.

    Now that the election is over, our analytics is pointing to the analysis of the equity market. The preliminary result indicated that the market is negatively cautious while DOW for the last two days went up a few hundred points. We shall see if we get this right again. We have been using this technology to predict the outlook of major public companies.

  2. Cameron Koo says:

    Astute observations, Bob.

    Don’t forget about the completeness of data. You can’t analyze what you can’t see. The polls & models were predominantly focused on quantitative analysis. Case-in-point: HRC’s Ada algo. How many models considered demographic sentiment by platform/issues? Organizations are starting to think about it, but aren’t sure how to move forward. Have you seen this piece by CBSNews from May?

    Combining structured and unstructured analysis is critical to truly understanding your audience. Understanding textual data is not easy. Many idioms and expressions in natural language require various NLP tech to parse for meaning let alone predictive insight. For years, grads & post-grads have spent years to figure it out.

    The inability to process signals at scale have forced media analysts & organizations alike to reduce the spectrum they’re receiving. Telescope this to commercial sector and you’ll see what I mean. Open-ended opinions are eschewed for NPS-esque rankings. You gotta get the data to understand it.

    New unstructured analytics technologies (like our CIF platform) are emerging to help organizations expand the spectrum of signal they capture and transform them into actionable vectors.

    For example, we published an analysis of presidential debate transactions back in October. We triangulated key messages that resonated with an objective simulation of the American population, distilling hashtags (trends). It didn’t come as a surprise that the polls were off given CIF’s analysis.

    If folks had gone to where the data was – as your KISS framework proscribes – I’m sure they wouldn’t have been either.

Leave a Reply

Your email address will not be published. Required fields are marked *

Comments or opinions expressed on this blog are those of the individual contributors only, and do not necessarily represent the views of Gartner, Inc. or its management. Readers may copy and redistribute blog postings on other blogs, or otherwise for private, non-commercial or journalistic purposes, with attribution to Gartner. This content may not be used for any other purposes in any other formats or media. The content on this blog is provided on an "as-is" basis. Gartner shall not be liable for any damages whatsoever arising out of the content or use of this blog.