Doctoral Research: New Technology Allows AI Agents to Read and Respond to People's Facial Expressions
Humans constantly read each other's facial expressions during conversation, adjusting their responses based on what they see. Abdallah Hussein Sham, a recent doctor at Tallinn University, developed a new technology that allows AI agents and metahumans to do the same.
Technology now allows the creation of increasingly realistic AI agents. Human-like AI agents, such as the digital characters that appear as virtual assistants, game characters, and the increasingly lifelike "metahumans" used in film and interactive media, can look remarkably realistic, but their facial behavior during conversation remains stiff and generic. A doctoral thesis defended at Tallinn University addressed this gap by building a system that enables an AI agent to both interpret and generate facial expressions.
How does it work?
Abdallah Hussein Sham's thesis developed the Enactive Facial Expression Pipeline (EFEP), a modular camera-based system built on the idea that meaningful interaction emerges from the ongoing exchange between a person and an AI agent, rather than from reading a single snapshot of a face.
Using only a standard webcam, the system reads the person's expression, estimates the reaction it would naturally elicit, and synthesizes the agent's facial response accordingly.
The research combined existing facial expression datasets with a new dataset collected from 60 participants across five social scenarios. A key finding was that working with individual facial muscle movements, rather than broad labels such as "happy" or "sad," gave the agent finer, more stable control over its responses.
The research also placed strong focus on fairness, finding a significant accuracy gap when models were trained on data from only one demographic group; a gap that disappeared with diverse training data.
Real-world impact
This matters because AI agents are increasingly used in low-risk settings, such as interactive media, creative applications, and user experience research, where the quality of the interaction depends on nonverbal communication. A human-like agent that responds with an appropriate expression, rather than a scripted animation, could meaningfully improve how people experience these technologies.
As a next step the technology could be expanded to include voice and body language, making interactions between humans and AI agents feel even more natural.
Ethics are important
In line with the EU AI Act, the system was intentionally limited to low-risk areas like interactive media and user experience research.
The thesis defense
Abdallah Hussein Sham defended his doctoral thesis at the Tallinn University School of Digital Technologies on May 15, 2026. The topic of his doctoral thesis is "Enactive facial expression pipeline for dyadic interaction between humans and human-like agents".
The thesis supervisors are Pia Tikka, Associate Professor and David Jose Ribeiro Lamas, Professor at Tallinn University, and Gholamreza Anbarjafari, Professor from Estonian Business School.
The opponents are Giulio Jacucci, Professor at the University of Helsinki and Fernando Loizides, Associate Professor at Cardiff University.
The doctoral thesis is available in the ETERA digital environment of the Tallinn University Academic Library.
