Many pundits have been wondering if AI will create new jobs as it also destroys them. Well, for me the rise of AI has already created a new and complex task: interpreting predictions about how AI will impact jobs. Add think about all the effort that went into creating all those predictions and the work by the media in reporting on them. It’s possible that the biggest impact to jobs by AI so far is the work it creates trying to explain what it is and then figuring out predictions about it!
But seriously folks, outside our techno-AI bubble there are a lot of readers with good reasons to be interested. For them I’ve created a handy guide for media, vendors, businessfolk, and anyone else who enjoys consuming these predictions. I’m interested in those that cross three overlapping areas: artificial intelligence (AI), robots, and automation, which I abbreviate as AIRA.
With no further ado, here is my guide to evaluating these predictions:
What are the definitions?
“AI” can have a very broad set of definitions that greatly impacts estimates about it. And what we consider AI may change over time. “Robots” can refer to physical ones only (for manufacturing and warehousing) or software ones as well. And if you include “chatbots” it opens a whole different arena of customer service and support apps. “Automation” is pretty clear, but can include everything from well defined technology like RPA (robotic process automation) to the cotton gin. Even “jobs” and currency (USD? what inflation adjustments are assumed?) can use some clarification.
What are the parameters that were used to create the estimate? How many factors are in the model?
The best way to understand what kind of food you’re about to eat is to know the ingredients that went into it. There have been so many studies done on AIRA and job loss that the ones currently getting attention usually introduce a novel ingredient to the cake. For example, McKinsey’s novel parameter was to look at “tasks” vs the “jobs” that had been used since Frey and Osborne.
A model with fewer parameters is actually better than one with dozens. A good modeler will examine dozens of parameters, but quickly test and eliminate those that don’t increase its explanatory power, leaving the few that really make a difference.
What assumptions were made for those parameters?
Of course, using the wrong amounts of butter, sugar, eggs, flour, and baking powder can produce a terrible cake. Getting the amounts correct is clearly important. So where do the inputs come from?
What is the timeframe?
AI job loss predictions generally fall into 3 timeframes: mid-term (5-10 years), end point (20-50 years or just a vague “where this is all eventually heading”), and near term (today to 3 years). The mid-term ones often come from think tanks and academia and the end point ones from thought leaders. But there is a dearth of near term estimates, which I find troubling because future predictions have to be anchored from where we are today. Confidence ranges for estimates naturally get larger over time, so having a good and tightly bound estimate of current job market reactions to AIRA is essential.
How does the “net” divide into gain and loss?
Any good AIRA v Jobs prediction should consider jobs created as well as lost. It may estimate practically no gain, but it at least has to make the effort. In the worst cases, a prediction on “job loss” isn’t even clear about whether that is net of gains or just focusing on the loss (as some studies have done since that’s the worrying part). If I had to pick one, I’d estimate job gains since it’s the more interesting and challenging problem and there has already been so much work done on potential job losses.
Are emerging markets included?
It is a privileged class that generally reads and writes these predictions. Countries with a high degree of automation and robots, deeply involved in AI, and with high proportions of knowledge workers naturally have a certain profile for what jobs they have to lose and what technology can be introduced that is very different from that of emergent countries. Emerging markets that have pulled themselves out of poverty have done so with the help of low wage, routine work that allowed them to create jobs and slowly increase productivity and wages over time. AIRA could prove to be devastating competition to these countries, raising the first rung on the economic ladder, so demonstrating the very different impacts on them requires a more holistic view of the future of work.
Does the estimate look at tasks, jobs, or currency? Are there conversion assumptions made?
The unit of measure matters a lot, and these three aren’t easily interchangeable. Eliminating 20% of the tasks performed by 100 clerks doesn’t eliminate 20 clerks – there are assumptions that have to be made. And dollars may not directly translate to jobs over time as salary averages are impacted by AIRA.
Does the model include causality or feedback?
Another way to ask this is “which of the model’s parameters are assumed to remain fixed over time?” If AI really starts chomping away at whole swaths of the job market will wages remain fixed? If the average investment analyst makes $80k today and over the next 10 years half those jobs are lost to auto-investing, how will the wage rate change as they are lost? Whether the remaining humans will make a lot more or less is an assumption, but it isn’t going to stay constant. Some models have a more political and societal angle, and may include trigger points at which mass revolt or government intervention may interrupt the pretty curves in the model.
What is the range of confidence?
Sure, a model can pop out an estimate that 1,537,423.276 jobs will be lost in 20 years. But anyone publishing an estimate like that should get 9.8704 lashes. It just demonstrates the creator doesn’t know how to use the ROUND() function in Excel and hasn’t thought about (or doesn’t want to admit) how exact that figure is. My first preference is for ranges, second is for degree of error, third is for confidence interval, and last resort is rounding where the number of zeros indicates the level of confidence.
Are there other scenarios?
In domains with vast uncertainty, like this one, it is helpful to publish a set of scenarios and what the outcome would be in each. Ideally all scenarios would tie to the same model and show results of different assumptions of the parameters, although that’s not always the case. So is this the optimistic or pessimistic scenario? Sci-fi vs. plodding technological progress?
And finally, why do you care?
Are you a vendor or service provider trying to allocate R&D or create messaging? An end user doing budgeting? Or just a curious interested party? It’s hard to argue a prediction doesn’t meet your needs unless you acknowledge what your needs are. It seems many estimates are created without an exact audience in mind. Some models are good for strategic planning while others are more assertions for cocktail party conversations.
That’s just a brain dump of what’s been going through my head when I read AIRA v Jobs predictions. I hope you find it helpful if you’re looking at them as well. I don’t mean this to be a general primer on predictions and modeling. These are the ones that are important for AIRA v Jobs. If anyone can provide a link to good books or articles on judging models in general, please provide it in the comments section.
A final note: My intention is not to set a bar so high that no model is good enough. The impacts of AIRA on jobs is one of the most important economic issues we face and is best served by open discussion of what may happen and why. Whether finger-to-the-wind or detailed econometric models, we need them all to have the proper discussion. A simple, two factor model can bring up a good point that a complex, academic one overlooks. It’s just important that readers interpreting these estimates understand what they are looking at. And for those providing them to be open about how they have come about.
Read Complimentary Relevant Research
Predicts 2017: Artificial Intelligence
Artificial intelligence is changing the way in which organizations innovate and communicate their processes, products and services. Practical...
View Relevant Webinars
The IoT In Manufacturing Operations: Where Are We Now?
The Internet of Things (IoT) is a paradigm shift for manufacturing operations. Its fanfare creates uncertainty in state-of-the-art technology...
Comments or opinions expressed on this blog are those of the individual contributors only, and do not necessarily represent the views of Gartner, Inc. or its management. Readers may copy and redistribute blog postings on other blogs, or otherwise for private, non-commercial or journalistic purposes, with attribution to Gartner. This content may not be used for any other purposes in any other formats or media. The content on this blog is provided on an "as-is" basis. Gartner shall not be liable for any damages whatsoever arising out of the content or use of this blog.