Gartner Blog Network

5 Techniques to Troubleshoot a Machine Learning Model

by Sumit Agarwal  |  August 5, 2020  |  Submit a Comment

Troubleshooting code in C++ or Java has been fairly easy. With a plethora of tools that help with a step-by-step execution of code to thread analysis at run time to de-compilation of code,  software developers probably take a debug or troubleshooting capability for granted. Data Scientists and other Machine Learning professionals still need to debug their model the hard way. They often face challenges such as :

  • Excellent model accuracy metrics during training, but very bad performance in production or with testing data
  • ML model accuracy is not production quality
  • Unable to explain the model features or results even if the model accuracy is good

There are several techniques that provide insights into the model quality and performance.  These techniques span across the model development process. The following techniques may be used once the problem is identified during model training or even post deployment:

  1. Visualize and profile data for patterns
  2. Use Explainability toolsets to verify feature importance
  3. Validate data for variety and fairness
  4. Verify the metrics for model underfit or overfit
  5. Improve the model performance by tuning hyper-parameters

The above techniques provide mechanism to validate if the model is using the right data – the right features – the right metrics and the right hyperparameters.  These techniques are not just troubleshooting techniques. They are also best practices that data scientists should include as a part of their machine learning model development process. This will help with a shift from a reactive approach to a more proactive approach.

I have defined these techniques in what may be my favorite and most fulfilling document –5 Techniques to Troubleshoot Your Machine Learning Model (requires a Gartner account). The document includes a downloadable Jupyter Notebook with code examples based on a home loans open data set. The document also includes a summary video providing a quick overview of the techniques. The techniques are essentially an extension to the machine learning training framework defined in Machine Learning Training Essentials and Best Practices

Additional Resources

View Free, Relevant Gartner Research

Gartner's research helps you cut through the complexity and deliver the knowledge you need to make the right decisions quickly, and with confidence.

Read Free Gartner Research

Category: artificial-intelligence  analytics-and-bi-solutions-for-technical-professionals  data-and-analytics-leaders  data-and-analytics-strategies  

Tags: algorithm-bias  artificial-intelligence-2  data-for-machine-learning  data-science  explainable-ai  fairness  hyperparameter-tuning  jupyter-notebook  machine-learning-2  overfit  visualization  

Sumit Agarwal
Sr Director Analyst
1 year at Gartner
24 years IT Industry

Sumit Agarwal provides guidance on Artificial Intelligence (AI), Machine Learning (ML), Data Science Architectures, Data Management and Data Integration architecture and strategies, based on upcoming ideas, current trends, and past project implementations. Read Full Bio

Leave a Reply

Your email address will not be published. Required fields are marked *

Comments or opinions expressed on this blog are those of the individual contributors only, and do not necessarily represent the views of Gartner, Inc. or its management. Readers may copy and redistribute blog postings on other blogs, or otherwise for private, non-commercial or journalistic purposes, with attribution to Gartner. This content may not be used for any other purposes in any other formats or media. The content on this blog is provided on an "as-is" basis. Gartner shall not be liable for any damages whatsoever arising out of the content or use of this blog.