We held our first VIRTUAL Data Science and Machine Learning Bake-off on September 15th as part of our BuySmart webinar series in conjunction with the BI Bake-Off conducted by Rita Sallam on September 8th. The bake-offs are fast-paced, demonstrative sessions that let you see vendors side-by-side using scripted demos and common data sets in a controlled setting.
Some may recall that last year was our very first DSML Bake-off. The event was hosted with the BI Bake-Off at our Data & Analytics Summit in Dallas. Last year, these sessions proved to be some of our most popular. Consistent with many of our activities THIS year, however, COVID has caused us to rethink, repurpose and revamp our approach. This content is very important and informative, so Rita and I felt strongly that we needed to find a way to offer Bake-Offs even though the physical events were a no-go…
Enter the virtual Bake-Offs!
We are grateful to our DSML Bake-off participants this year: Dataiku, IBM and RapidMiner. They will surely attest to the fact that vendor participation in these sessions is not for the faint of heart. The session requires careful, thoughtful planning, preparation AND delivery. Imagine being tasked with standing side-by-side with your toughest competitors, demonstrating not only your ability to put your product through its paces, but also providing what is unique and differentiating in your offering – all within a scripted framework and carefully, measured timeframe. Talk about pressure! But these folks delivered in a big way – and were interesting and entertaining to boot.
From left to right, top to bottom: Jordan Tidaback, our Poll Master (Gartner), Carlie Idoine (Gartner), Dr. Martin Schmitz (RapidMiner), Christina Hsiao (Dataiku), Dirk deRoos (IBM)
If you are a Gartner client (excluding all vendors), you can access the replay of this session through your Gartner account here.
Historically, we have used the Bake-Offs as a platform for Data for Good. Data for good is a movement in which people and organizations transcend organizational boundaries to use data to improve society. Consistent with the spirit of data for good, we choose population health data for this year’s events. We are grateful to our partners at the World Health Organization for granting us permission to use their data and also to the World Bank. These data sets are made available through the Humanitarian Data Exchange or HDX. The Humanitarian Data Exchange (HDX) s an open data sharing platform managed by the United Nations Office for the Coordination of Humanitarian Affairs.
The data and script were originally chosen pre-pandemic. With the rescheduling in the virtual format post-pandemic, the vendors had the option to include COVID-related data and update their demos.
DSML Bake-off Findings and Demos
Check out the interesting findings from each vendor’s session and access the links to their videos below. As with any good model building exercise, we began with the questions we wanted to address. Our objective was to understand which variables have a positive impact on life expectancy and to identify efforts that can be made to impact those areas that have the most positive influence. The challenge was to build a predictive model demonstrating each step of the end-to-end analytic process.
The model built by Dataiku predicted an overall life expectancy of 78 years and 10 months as compared to the actual value from the World Health Organization data of 78 years and 7 months. This represented a slight decrease in life expectancy from 2014. Dataiku also examined historic and predicted values by sex. Even though the differences are quite small, less than 1% change, the rate that male lives are predicted to decrease is twice that of females. Dataiku noted that these types of interactions can be interesting to investigate and possibly adjust the project based on the findings. In this case, we might want to consider building independent models, for males and females, for more accurate predictions.
The most important predictors of life expectancy included prevalence of excessive weight in adults, and thinness in children and adolescents. If countries were able to reduce adult obesity and underweight youth, that would have the most positive influence on the predicted outcome. Beyond these variables, consumer price index, tobacco use, alcohol-related disease, and urbanization are the next most important indicators. Additional intriguing indicators the model relied upon included birth rate among teenaged girls, density of cellular subscribers in the population, income share held by the lowest 20%, total household expenditure on health, hospital beds available per 10,000 people and immunizations.
As a next step, Dataiku suggested collecting information on these most important variables at a state or zip code level, or stratified by other demographics such as race or income, in order to determine whether there are disparities among specific regions, socioeconomic subgroups, or health programs that the government should focus on first.
Below is a screenshot demonstrating a correlation matrix, one of Dataiku’s exploratory data analysis features.
As IBM started processing the data and evaluating indicators, there were several that trended quite closely with Life Expectancy at Birth. Some of them were surprising! For instance, the Body Mass Index variable has almost the exact slight slope upward. Intuitively, we know that on an individual basis, BMI and obesity are bad indicators for life expectancy.
IBM indicated that, practically speaking, given life expectancy continues to go up, there must be other factors acting as a powerful counterweight to what one would expect are opposing trends. IBM experimented with their AutoAI capability by feeding it multiple sets of features and settled on a model that behaved well with indicator data from a number of countries. For this model, health spending clearly has the strongest predictive value. In addition, educational attainment, drinking water quality, and some economic indicators had demonstrative predictive value.
The screenshot below shows IBM’s pipeline comparison view and leaderboard displaying the most common metrics used to assist in evaluating model performance.
RapidMiner’s end-to-end analysis and machine learning model discovered that some expected factors had a strong influence on life expectancy, including economic factors like GDP and unemployment, as well as personal lifestyle factors such as activity level and alcohol consumption.
But there were some surprising, unexpected factors that were uncovered as well. In the US, for example, maternal and infant mortality are a major factor in the country’s reduced life expectancy when compared to other developed countries. With the global COVID pandemic causing economic turbulence, lowering GDP, and worsening unemployment, it’s clear that the US needs to remain focused on, and continue to improve, the quality of care during and immediately after childbirth by ensuring that resources from the country’s high health care expenditures are properly routed, thereby creating a healthy future for American mothers and their babies.
Below is a screenshot of performance comparisons between the active model and challengers.
Show Floor Showdown Findings and Demos
Although we limited the Bake-Off to three vendors, we are highlighting videos below from vendors that were scheduled to participate in our live Show Floor Showdowns at Gartner’s Data and Analytics Summit in Dallas and London in March. These vendors were selected to participate based on random selection after an open application process. The selected vendors created their demos using the same demo script and data sets as used for the Bake-Offs.
DataRobot’s augmented approach to analysis and model building found that women live longer than men across all countries, which is consistent with many studies of developed countries. The analysis also found that many of the strongest predictors of longevity are actionable. The top ten predictors of longevity included several measures including immunization and the incidence of preventable diseases such as tuberculosis, measles, rubella, pertussis, and diphtheria. Childbirth was determined to be a risk factor with maternal mortality and neonatal mortality being important drivers. Education participation rates, especially for females, improve longevity and lifestyle choices such as access to fresh local produce, reduced pollution from fossil fuels, and switching to renewable energy, will help life expectancy outcomes. Interestingly, once a reasonable level has been achieved, extra expenditure on healthcare does not further improve longevity outcomes.
DataRobot also performed an analysis specific to the U.S. The US has the lowest incidence of tuberculosis, which is a top three ranked predictor of longevity. Diphtheria, tetanus, toxoid, and pertussis (DTP3) immunization coverage is mid-range compared to the other countries. Immunization rates for other diseases were also drivers of longevity. The US has the highest maternal mortality ratios of all countries in this analysis. Maternal mortality ratios consistently placed in the top three most important drivers of longevity. The analysis also found that the US did not submit data for several important drivers of longevity, including many of the education participation measures, pollution from fossil fuel, alternate energy and renewable energy usage, agriculture, forestry, and fishing value added, prevalence of obesity and domestic private health expenditure per capita.
The following screenshot demonstrates heuristics around the individual decisions automatically made when building the model:
SAS pointed out that the data selected for the model was reflective of real-life scenarios where the data to be used as input for the model is rarely in a format that is conducive to analysis and model building without initial data preparation. SAS also indicated that in a real-world scenario, the data set provided would be too small to produce statically sound results for predictions and was more useful as an explanatory model.
SAS found many macro-economic and social indicators were related to average life expectancy. The most relevant factors in the models were GDP per capita, per capita expenditure on health, total expenditure on health, mortality rate and infant mortality rate. Most of the relations were intuitive such as GDP per capita. For example, as GDP per capita increased there was a general increase in average life expectancy.
The screenshot below provides visualizations of drivers and characteristics driving life expectancy:
Tibco investigated each year’s measure of life expectancy and explored some of the additional hundreds of other variables (education levels, GDP, poverty rates, etc.) that might play into population health. They were able to show not just the predictions for each country, but also measures of each variable’s importance, demonstrating levers that governments could pull on to improve population health. Tibco also showed how these variables differed for Australia (or any other country) from the rest of the world.
The below screenshot demonstrates a comparative analysis, both temporally and geographically, on the variables under consideration:
Each year brings new challenges and opportunities for the bake-offs – but this year, the wrench in our plans was a doozy! Despite it all, we were pleased we were able to find a way to share the work, approaches and learnings from this event. We offer our sincere thanks to all that put in the extra time and effort to bring the bake-off sessions to fruition (pun intended)! Thanks also to the Show Floor Showdown vendors for providing their demos for this blog.
Disclaimer: Data in these analyses are for demonstration purposes only and may not be statistically accurate.