What is emotional AI and should we be concerned?

Emotional AI, or affective computing, refers to the technology that can sense and respond to human emotions by analysing data from facial expressions, voice tone, body language, and even physiological signals. While the idea of machines recognising emotions might seem futuristic, it’s already being implemented in industries like advertising, healthcare, and customer service. But with this rapid adoption comes critical questions about privacy, ethics, and the true impact of emotional AI. “Brands will always want that information to strengthen their connection with the audiences. Why wouldn’t they want to know their customers? The tricky part is where they draw the line. Should customers know their feelings are being recorded, captured, and used?” Tanuj Khanna, content lead, Wondrlab Network, said. Speaking as a creative person, knowledge allows us to create sharp communication. But, maybe the right way to get that knowledge is the old-fashioned way, with face-to-face interaction, he added.

How does it work?

Emotional AI uses algorithms trained to detect and interpret emotional cues through technologies like facial recognition, which tracks expressions, speech analysis which monitors tone, pitch, and pace, and biometric sensors which measure physiological responses such as heart rate or skin temperature. “Emotional AI often relies on various data sources, such as facial recognition, voice analysis, and sentiment analysis from social media. These methods can gather extensive personal data without explicit consent, breaching individual privacy,” Sindhu Biswal, founder and CEO, Buzzlab, added. For instance, a virtual assistant might detect frustration and adjust its tone to ease tension. However, the question remains can these systems truly grasp the complexity of human emotions, or are they oversimplifying the intricate nature of how emotions work?

Is it a tool for manipulation?

Brands have embraced emotional AI for its promise to drive more personalised interactions. 51% of marketers are using AI, according to a survey of over 1000 marketers conducted by Salesforce this year. But is this level of personalisation ethical? With real-time emotional analysis, marketers can tailor ads based on how someone feels—potentially leading to manipulative practices. For example, could brands exploit a consumer’s emotional vulnerability to push products when they’re most susceptible? However, there might be another way to look at it too. “When brands can read their consumer’s emotions they can sell more of their products. I don’t think it crosses ethical boundaries unless we are talking about kids and vulnerable adults,” Pawan Prabhat, co-founder, Shorthills AI, opined.

The stakes are high. In an era when advertising is already under scrutiny for privacy violations, the question is whether emotional AI will take consumer tracking too far. Can a line be drawn between personalisation and manipulation?

In healthcare, emotional AI holds potential, especially in mental health monitoring. Devices equipped with affective computing can track emotional states and provide insights to therapists about a patient’s well-being. “Emotional AI is widely used across industries like call centers in banks, insurance companies, and healthcare services for sentiment analysis, helping understand customer emotions. In marketing research and digital advertising, it’s applied in public spaces, and theme parks use it with computer vision to monitor crowd moods and line lengths,” Glenn Gore, chief executive officer, Affinidi, said. For example, if a long line leads to frustration, the park may send over characters to entertain, enhancing the experience without intruding on privacy. The focus is on improving collective experiences by assessing general emotions, not on identifying or tracking individuals, making it a valuable tool for enhancing customer satisfaction, he added.

Consider the implications of constant emotional surveillance. Could this lead to a new level of intrusion into consumer privacy, where emotional data is misinterpreted or misused? If AI detects someone’s stress level rising, should this automatically trigger alerts, even in cases where it may not be clinically relevant? While the benefits of proactive mental health treatment are clear, the potential for overreach raises red flags.

Customer service: Personalisation or privacy violation?

Emotional AI is also being integrated into customer service systems. AI-powered chatbots can detect frustration in a customer’s voice and escalate the issue to a human agent. 80% of customer service and support organisations will be applying generative AI technology by 2025, according to Gartner. While emotional AI promises more personalised interactions, it may also push boundaries that consumers are uncomfortable with. 52% of Americans are more concerned than excited about AI in daily life, according to Pew Research study. If a machine can detect when you’re upset or stressed, what other data is it collecting, and how is it being used? There is also the risk that companies may store and analyse emotional data in ways that consumers didn’t consent to. Could this lead to a future where emotional surveillance becomes the norm?

Privacy and ethics: Where do we draw the line?

The rise of emotional AI also brings significant ethical challenges. The collection and use of emotional data could lead to even greater concerns about privacy. For instance, if emotional insights are harvested by tech companies, what guarantees are there that this data won’t be exploited for manipulative advertising or, worse, sold to third parties?

Europe’s GDPR provides some protection, but emotional data falls into murky legal waters. Should new laws specifically govern how emotional data is used? Some experts call for more transparency and consent mechanisms, ensuring that consumers know when their emotions are being tracked and how that data is being used. But how many people are aware of the extent to which emotional AI is already part of their digital interactions?

“Data protection regulation should adopt a risk-based approach for emotion recognition systems, which process biometric, physiological, and behavioural data. Unlike GDPR, the DPDP Act, 2023 doesn’t distinguish between personal data categories. Regulations should differentiate positive use cases that enhance consumer welfare from those that pose risks in areas like housing, education, and healthcare. The DPDP Act should emphasise transparency through notice and consent, and businesses should develop standards for high-risk applications. Regulations must also address social exclusion due to system inaccuracies, with Data Protection Boards collaborating on grievance redressal mechanisms in sensitive sectors,” Sidharth Deb, public policy manager, The Quantum Hub, said.