Imagine movies could “read” your emotions
We are used to the privilege of observing movie characters involved in the most dramatic and intimate situations. It does not matter if we get emotionally touched by the character’s sad fate in similar manner if we watch films from our living room tv-screens, or from our mobile screens when flying across the Baltic Sea.
We are used to trust that the transparent “wall” separating us from the movie world. Despite of our thoughts and feelings towards the character we cannot help the character in her most difficult situation. Not yet, but someday this may radically change.
The research professor at Tallinn University MEDIT Pia Tikka is combining art and science to study what she calls enactive co-presence between the viewer and screen character. In the simplest, this means an emotionally loaded non-verbal connection created and managed in real-time by movie character’s direct eye-contact with the viewer. It all started back in 2000 when the fuzz over interactive cinema spread around the world.
The enthusiasts of the time hyped that soon one did not have to be a professional filmmaker to be able to edit one’s own film. Even more, the movie-making world would soon need no professional film directors. The raw material for the films would be automatically searched from the exhaustive media data-bases that would include all audiovisual material ever produced. Based on one’s own preset preferences, the data-base cinematic systems would edit films onto your personal screen. Yet, in this techno hype many forgot the main advantage of the movies.
Trusting one’s experience to the hands of a team of skilful experts, their narratives would leave no-one cold and without a drop of tear in the corner of the eye. One can aske, could a film that you edit your self excite you, charge you emotionally, or surprise you as anew? Most probably not, Pia replies. To move away from this hype of interactive cinema, Pia found herself wondering if another kind of interactivity would be possible. What about a movie tracking the viewer’s unconscious emotional experiences? So, instead of making conscious choices, say, on how you would like the film to end, or which two characters you would like to fall in love, the computer system would automatically figure out your feelings about a character or a situation using biosensor tracking. It would then use that information to edit the story without your conscious decisions.
This idea became realized in Pia’s concept of “enactive cinema”, a movie that allows your emotional interaction with the story without disturbing your experience of the story.
Get ready to meet your soulmate
The competitive 5-year research grant that brought Pia to Tallinn University allows her and her collaborators to study the unconscious interaction using human-like screen characters, which she calls “enactive avatars”. Imagine that in a dramatic situation of the story the character suddenly turns to you. She looks at you and your body-brain system reacts to her attention. Your psychophysiological reactions are measured by biosensors and this encounter changes the course of the story. The avatar is in sync with your state of mind, you may start feeling attached to her, understand her dramatic situation and the problems she is facing in the story. This may sound freaky, but after all your desires may find their expressions in the virtual environment of your storytelling mind. Consider for instance how the male character in Spike Jonze’s film „Her“ (2013) fell in love with the voice-based artificial personal assistant. We do develop a kind of artificial intelligence, comparative to the humanlike robots developed in the world during last years. The overall purpose for this line of research is to produce artificial characters that can understand human emotions and respond empathically, with positive facial expressions. However, our research that emerges from filmmaking and narratives emphasizes the contextual dependence of the viewer experience. In our view when you separate an enactive avatar or an enactive robot from context, it becomes an empty shell. Without understanding the contextual situatedness of the character it becomes difficult to project any feelings to it even though it would show sad expressions. This is why context is our third study object in addition to the viewer and the avatar.
Combining arts and sciences
In the research project Pia Tikka and team in Tallinn University will focus on the viewer’s experience of co-presence when facing a real-time adaptive movie character. The team will conduct scientific experiments where they collect data from several viewers. While people will be interacting with the avatar the research team will measure changes in one’s brain signals, heart-rate, skin conductance, eye movements, etc. in order to understand how we humans respond to the fictional characters on the screen. The work will provide new understanding also on how we respond to one another, in interaction with real people. Pia’s urge to understand what happens in the viewers’ minds when they get engaged with films had already led her to brain research. For previous seven years she collaborated with the leading brain researchers at the Aalto University brain research projects showing how surprisingly similarly viewers’ brains respond when watching movies. This happens when, for instance, witnessing unfairness towards someone on the screen, or any emotionally loaded scene. As in any scientific experiment that deals with human emotive-cognitive experiences, the interpretation of the data is always a tough task. This is especially so, when the data has been collected from socio-emotionally complex situations, comparative to situations in everyday life. Some fear that soon the brain imaging machines will be able to read anyone's mind. Pia Tikka understands this concern. However, based on her experience working with such data, it will take long before scientists could have access to your mind same way way you do have access to your own mind, she smiles comfortingly.
-- The story was first published in the Nordica Aviation Group in-flight magazine Time Flies, Winter/Spring 2018 edition. Prof. Pia Tikka is a Research Professor at the Tallinn University Baltic School of Film, Media, Arts and Communication. She has joined Tallinn University Centre of Excellence in Media Innovation and Digital Culture (MEDIT) after winning Estonian Research Council’s EU Mobilitas Top Researcher Grant.
