Blog post

5 Techniques to Troubleshoot a Machine Learning Model

By Sumit Agarwal | August 05, 2020 | 0 Comments

Data and Analytics StrategiesData and Analytics LeadersAnalytics and BI Solutions for Technical Professionals#artificial-intelligence

Troubleshooting code in C++ or Java has been fairly easy. With a plethora of tools that help with a step-by-step execution of code to thread analysis at run time to de-compilation of code,  software developers probably take a debug or troubleshooting capability for granted. Data Scientists and other Machine Learning professionals still need to debug their model the hard way. They often face challenges such as :

  • Excellent model accuracy metrics during training, but very bad performance in production or with testing data
  • ML model accuracy is not production quality
  • Unable to explain the model features or results even if the model accuracy is good

There are several techniques that provide insights into the model quality and performance.  These techniques span across the model development process. The following techniques may be used once the problem is identified during model training or even post deployment:

  1. Visualize and profile data for patterns
  2. Use Explainability toolsets to verify feature importance
  3. Validate data for variety and fairness
  4. Verify the metrics for model underfit or overfit
  5. Improve the model performance by tuning hyper-parameters

The above techniques provide mechanism to validate if the model is using the right data – the right features – the right metrics and the right hyperparameters.  These techniques are not just troubleshooting techniques. They are also best practices that data scientists should include as a part of their machine learning model development process. This will help with a shift from a reactive approach to a more proactive approach.

I have defined these techniques in what may be my favorite and most fulfilling document –5 Techniques to Troubleshoot Your Machine Learning Model (requires a Gartner account). The document includes a downloadable Jupyter Notebook with code examples based on a home loans open data set. The document also includes a summary video providing a quick overview of the techniques. The techniques are essentially an extension to the machine learning training framework defined in Machine Learning Training Essentials and Best Practices

Leave a Comment