Blog post

Worry Me, Natural Language Processing Doesn’t

By Adrian Lee | January 17, 2018 | 1 Comment

Digital CommerceConversational AI


The Newsweek headline screamed, “Robots Can Now Read Better Than Humans, Putting Millions of Jobs At Risk”. I nearly drowned laughing into my morning coffee. It was a slow news day?

The story described a reading accuracy test that pitched Alibaba’s natural language processing (NLP) AI versus a ‘rival human’.  Yes, the robot outdid the human by a score of 82.44 versus 82.305. Yes, that’s right, a difference of 0.135.  The article did not continue on to reveal which were the millions of jobs at risk. So, the alarmist stance that massive job cuts were on the horizon didn’t worry me. Yet. To put things into perspective, Gartner currently has over 900 research articles on the topic of conversational platforms, with more than 46 different analysts investigating the area of conversational AI.

Oh, Natural Language Processing (NLP) is the comprehension by computers of the structure and meaning of human languages, allowing users to interact with the computer using natural sentences. NLP features are dynamic and are evolving within conversational platforms. The one refrain I have for our clients when I advise on conversational platforms is this: we have so much more to learn.

The ‘processing’ of structured language has increased dramatically, especially in the advancement of entity recognition, machine translation, and text categorization driven by ever faster and low latency machine learning instances in the cloud.  But the ‘understanding’ of meaning – inference, sentiment, relations, and variations, are still work in progress. Seek not to process, but to understand. By which, I refer to Natural Language Understanding (NLU), and that’s important. for you to see the big picture. Below. I meant it, literally.

We still need to make sure the semantic aspects of the technology work. This is what drives the ‘understanding’ of human by machine.  As chatbots proliferate across customer service, financial services, retail, government and even healthcare;  the ability of technology vendors to provide contextually aware, multiturn-based natural-language conversations  presents an ongoing challenge to natural language understanding (NLU) handling. This is especially the case in the areas of semantic parsing and natural-language inferences.

Ambiguities in semantics arise when there are multiple grammatical interpretations possible. “Giant road bike” and “Giant road bike” on a digital commerce site can mean very different outcomes if the semantics engine cannot infer that “Giant” refers to a brand of bicycle and not the size of the bike in question. But you already knew that.

I shall stop here. If you want to read more about NLP or NLU and how it impacts digital commerce, please click here. Or we could get that same bot to do it.

Either way, you shouldn’t let it worry you.

2018-01-18 03_55_44-ConversationalAI_Presentation_AL_15Jan2018 - PowerPoint

The Gartner Blog Network provides an opportunity for Gartner analysts to test ideas and move research forward. Because the content posted by Gartner analysts on this site does not undergo our standard editorial review, all comments or opinions expressed hereunder are those of the individual contributors and do not represent the views of Gartner, Inc. or its management.

Leave a Comment

1 Comment

  • Aman Angrish says:

    Hi Adrian,
    We are seeing a distinct trend where more solution creators are embracing NLP for giving a better experience to users (eg. in better search, better transcription etc). Such user-experience enhancements won’t be replacing humans anyway.

    Millions of jobs are not at risk yet because of NLP, NLU for the simple reason that there are many areas of work which are not being performed by humans anyway (such as real-time text analysis, software ux improvements etc). There are some areas of work being performed by humans but most tech solutions don’t give you 100% human-free experience yet.
    Specifically, call-centers might be the ones that see a reduction in workload due to self-service enabling technologies.