AI can do many things and will evolve to do many more, but will it be able to detect sarcasm or recognize other human emotions? Probably not too well. Analysis of the use cases and the challenges.

Omdia view

Summary

AI can do many things and will evolve to do many more, but will it be able to detect sarcasm or recognize other human emotions? Probably not too well. Analysis of the use cases and the challenges.

Unless your name is Google, stop acting like you know everything

One of thousands of great sarcastic lines.  Do you get it?  Some people do, some do not.

In March, computer science researchers at the University of Central Florida (UCF) published a paper describing a sarcasm detector they created using text mining NLP and deep learning. The noble intent of the project is to help companies understand sentiment and customer feedback on social media platforms.

Sigh.

That sigh, if you could not tell, was a sarcastic comment by me about this announcement.

According to academic research, sarcasm attracts higher attention and creates deeper influence than other negative responses posted in social media, so there is certainly a need by brands who champion customer experience or care deeply about customer service to understand the sentiment of end users.

It is just that, how to put this… AI stinks at sarcasm.

I will note however that Alexa has some sarcastic chops. Consider my recent Alexa interaction:

Me: Alexa, are you sarcastic?

Alexa: I’m not very good at being sarcastic, or maybe I’m really, really, really good at it.

If at first, you don’t succeed

For several years now, computer scientists have spectacularly failed in their attempt to use AI to detect sarcasm, but bless their hearts (more sarcasm, bless their hearts is a colloquialism native to the Southern region of the U.S.), they keep trying.

This is not to say the UCF sarcasm detector will fail – I mean after all (note: heavy sarcasm will be used through the rest of this paragraph), it uses a “novel multi-head self-attention-based neural network architecture” and “the experiments…show significant improvement over the state-of-the-art models by all evaluation metrics.” The paper went on to note their results in universally glowing terms, in that the model “can indeed identify words in the input text that can provide clues for sarcasm.

Well, there you have it.

The UCF study leveraged datasets which help explain the futility of the sarcasm detector. Two Twitter datasets were used, one from 2013 of 175,000 tweets, 35,000 containing the hashtag #sarcasm and 140,000 random tweets; the other dataset from 2017 collected by a bot named @onlinesarcasm. Two other datasets used were the Sarcasm Corpus V2 Dialogues dataset from 2016 and Reddit’s SARC 2.0 dataset from 2018. In each of these datasets, there was a great deal of human-in-the-loop to assist the AI. Even then, the first Twitter study failed to achieve an accuracy of over 80% even with the hashtag #sarcasm to help.

Sarcasm is an inherently human thing, which depends not only on what is said, but the tone of how it is said and the body language/facial expression of the person conveying it. But we live in a world where our preference for spoken or in person communications is declining and written communications are increasing, and so we have people being sarcastic in written communication, which is harder for humans, much less machines, to pick up.

Linguist Robert Gibbs said sarcasm includes “words used to express something other than and especially opposite of the literal meaning of the sentence.” Text mining alone won’t detect written sarcasm consistently.

Multiple modal = higher confidence

Some companies are exploring ways to use AI to interpret speech intonation and body language cues in the broader field of sentiment analysis and emotion recognition. There are companies that have explored interpreting vocal biomarkers (analyzing acoustic features) such as Vocalis Health, and others who have tinkered with facial recognition in vehicles, such as Smart Eye and Eyeris or video analysis for product marketing research, such as RealEyes.

In theory, a sarcasm detection solution could be cobbled together using multiple modal inputs. A company called Visual-AI proposes that logo detection and image recognition combined with sentiment analysis in images and text could be the answer, but it sounds like more of an idea than an actual solution. Cost in relation to the use case and business ROI would seem to be an issue in any multi-modal approach.

Sarcasm use cases

We have established that detecting sarcasm in customer feedback is beneficial to companies focused on customer experience. The other potentially lucrative use case for sarcasm detection involves virtual digital assistants (VDA) for customer service.  As customer service becomes increasingly automated, how will a VDA understand when an interaction becomes sarcastic?

Omdia spoke with two different VDA vendors about this, and there are work arounds. For now, best practice is to put escalation control in the hands of customers. “Very difficult for VDAs to figure out when someone’s being sarcastic,” said one VDA vendor executive, “so we have a handover button there all the time for the customer to escalate their conversation to a human agent. It puts the onus back on the human customer.”

Sentiment analysis and emotion recognition

In the end, the quest to automate the detection of sarcasm will likely be addressed more properly in the larger effort to automate the detection and understanding of human emotion.

Studies have shown that digital communications are eroding our ability to empathize with others. As digital communications become even more prevalent, expressed empathy becomes more important within those communications.

Companies are increasingly focused on making customer experience their competitive differentiator. More and more, companies are finding that the customer experience is not just about customer satisfaction, but more about the customer’s emotional connection.

There is no scientific consensus on a definition of emotion, but many experts agree that emotion influences thinking, decision-making, actions, social relationships and well-being. A better understanding of emotion will help AI technology create more empathetic customer and healthcare experiences, drive our cars, enhance teaching methods, and figure out ways to build better products that meet our needs.

In a final note to balance the continued effort to automate understanding humans, consider this quote from a lecture given by Dr. Tom Chatfield in 2016 for the Oxford Research Center in the Humanities:

“Human nature is a baggy, capacious concept, and one that technology has altered and extended throughout history. Digital technologies challenge us once again to ask what place we occupy in the universe: what it means to be creatures of language, self-awareness and rationality.

Our machines aren’t minds yet, but they are taking on more and more of the attributes we used to think of as uniquely human: reason, action, reaction, language, logic, adaptation, learning. Rightly, fearfully, falteringly, we are beginning to ask what transforming consequences this latest extension and usurpation will bring.”

Appendix

Further reading

Emotion Recognition and Sentiment Analysis (March 2018)

Ramya Akula and Ivan Garibay, “Interpretable Multi-Head Self-Attention Architecture for Sarcasm Detection in Social Media,” MDPI (March 2021)

Wei Peng, Achini Adikari, Damminda Alahakoon, and John Gero, “Discovering the influence of sarcasm in social media responses,” Wiley Online Library (July 2019)

American Technion Society, “System Detects & Translates Sarcasm on Social Media,” Newswise (June 2017)

David Bamman and Noah A. Smith, “Contextualized Sarcasm Detection on Twitter,” Carnegie Mellon University (April 2015)

Author

Mark Beccue, Principal Analyst, AI and NLP

askananalyst@omdia.com