Troubleshooting code in C++ or Java has been fairly easy. With a plethora of tools that help with a step-by-step execution of code to thread analysis at run time to de-compilation of code, software developers probably take a debug or troubleshooting capability for granted. Data Scientists and other Machine Learning professionals still need to debug their model the hard way. They often face challenges such as :
- Excellent model accuracy metrics during training, but very bad performance in production or with testing data
- ML model accuracy is not production quality
- Unable to explain the model features or results even if the model accuracy is good
There are several techniques that provide insights into the model quality and performance. These techniques span across the model development process. The following techniques may be used once the problem is identified during model training or even post deployment:
- Visualize and profile data for patterns
- Use Explainability toolsets to verify feature importance
- Validate data for variety and fairness
- Verify the metrics for model underfit or overfit
- Improve the model performance by tuning hyper-parameters
The above techniques provide mechanism to validate if the model is using the right data – the right features – the right metrics and the right hyperparameters. These techniques are not just troubleshooting techniques. They are also best practices that data scientists should include as a part of their machine learning model development process. This will help with a shift from a reactive approach to a more proactive approach.
I have defined these techniques in what may be my favorite and most fulfilling document – “5 Techniques to Troubleshoot Your Machine Learning Model“ (requires a Gartner account). The document includes a downloadable Jupyter Notebook with code examples based on a home loans open data set. The document also includes a summary video providing a quick overview of the techniques. The techniques are essentially an extension to the machine learning training framework defined in “Machine Learning Training Essentials and Best Practices“
The Gartner Blog Network provides an opportunity for Gartner analysts to test ideas and move research forward. Because the content posted by Gartner analysts on this site does not undergo our standard editorial review, all comments or opinions expressed hereunder are those of the individual contributors and do not represent the views of Gartner, Inc. or its management.