Chenyi Wang

The concept of this work is to create an individualized experience based on facial expression detection technology, which was inspired by the emotion is an abstract thing around us and we constantly need to interpret it from others, especially through facial expressions, to make appropriate reactions and decisions during socialization.

This demo is an interactive story about a rock that has multiple storylines and endings, and the audience will have different experiences based on their emotion during the time of watching, which will be the facial expression detected in this implementation.

In our conception, the rock, which represents stillness and grounding, embodies numbness and is void of any emotion. However, it transforms when it encounters different events triggered by the facial expressions of the participants. The essence of the storyline of the rock's evolving nature is to inspire hope and curiosity in people's lives. Regardless of one's current emotions, it reminds people that there is always a positive future ahead.

We utilized face-api and p5.js to implement our demo. The face-api is used to detect the audience’s facial expression throughout the time when the scene is playing. During the time of a scene is playing, the facial expression of the audience will be constantly detected and recorded. At the end of the scene, the most appeared facial expression will be computed by finding the corresponding expression of highest value of the detected confidence. Thus, when each scene ended, the computed result facial expression of the audience of that scene will be used to determine the next scene being played. At the moment of scene switch, a message will also be printed to the console with the information of expression detected to inform the audience of their control of the storyline.

Live Demo

Code available on GitHub.

Tools: p5.js, face-api

In collaboration with: Riva Wang, Seba Alabdullatif