Gartner Blog Network

Machine Learning – Shift from Modeling to Engineering

by Sumit Agarwal  |  August 21, 2020  |  Submit a Comment

In 2012, Harvard Business Review had published Data Scientist as the sexiest job of 21st century. In 2019, an Indeed survey had Computer Vision Engineer and Machine Learning Engineer as the highest paid tech jobs, ahead of the data scientist. Within seven years, the balance has shifted from modeling to implementation. From focused ML problems to more complex integrated solutions.

This could be due to several factors:

  • An explosion of data science focused MOOCs and graduate programs. Machine learning courses have been the most in-demand with classes filling up very quickly even in undergraduate programs, and not limited to computer science students. This should resolve some of the hiring availability challenges for entry level data scientists.
  • Tools such as AutoML and ML as a service (MLaaS) solutions from platform vendors have brought solutions for simple data science problems within the reach of non-data-scientists – such as business analysts, BI developers and application developers.
  • Increased maturity level of organizations where they are beginning to integrate machine learning solutions into the overall business process. They are not looking at a artificial intelligence (AI) based solution as an isolated application. But as a component within the overall business solution. This requires focus on the engineering aspect of the data science solution. The model optimization, code quality, continued performance and scalability have become required components.

The last factor is evident within our client interactions too, where a large number of organizations are trying to solve the architectures, tools and best practices for implementation of AI solutions. MLOps is probably the most popular AI buzz word currently. Open source initiatives such as Kubeflow and mlflow have lot of organizational support and excitement within the the developer community. Academic research, industry conference presentations, vendor platforms and even regulations are shifting towards advanced functionality such as explainable AI, model fairness, adversarial robustness, privacy, reproducibility and the overall governance of the model combining business and technical owners.

While these are topics of continued research, two of my recent documents – Machine Learning Training Essentials and Best Practices” and “5 Techniques to Troubleshoot Your Machine Learning Model”┬áprovide insights into few of these factors.

Additional Resources

View Free, Relevant Gartner Research

Gartner's research helps you cut through the complexity and deliver the knowledge you need to make the right decisions quickly, and with confidence.

Read Free Gartner Research

Category: ai  ai-in-the-enterprise  analytics-and-bi-solutions-for-technical-professionals  data-and-analytics-leaders  

Tags: ai-in-the-enterprise  data-scientist  data-science  machine-learning  machine-learning-engineer  

Sumit Agarwal
Sr Director Analyst
1 year at Gartner
24 years IT Industry

Sumit Agarwal provides guidance on Artificial Intelligence (AI), Machine Learning (ML), Data Science Architectures, Data Management and Data Integration architecture and strategies, based on upcoming ideas, current trends, and past project implementations. Read Full Bio

Leave a Reply

Your email address will not be published. Required fields are marked *

Comments or opinions expressed on this blog are those of the individual contributors only, and do not necessarily represent the views of Gartner, Inc. or its management. Readers may copy and redistribute blog postings on other blogs, or otherwise for private, non-commercial or journalistic purposes, with attribution to Gartner. This content may not be used for any other purposes in any other formats or media. The content on this blog is provided on an "as-is" basis. Gartner shall not be liable for any damages whatsoever arising out of the content or use of this blog.