In the Blink of an Eye: Event-based Emotion Recognition

Haiwei Zhang, Jiqing Zhang, Bo Dong, Pieter Peers, Wenwei Wu, Xiaopeng Wei, Felix Heide, Xin Yang

Research output: Chapter in Book/Report/Conference proceedingConference contribution

7 Scopus citations

Abstract

We introduce a wearable single-eye emotion recognition device and a real-time approach to recognizing emotions from partial observations of an emotion that is robust to changes in lighting conditions. At the heart of our method is a bio-inspired event-based camera setup and a newly designed lightweight Spiking Eye Emotion Network (SEEN). Compared to conventional cameras, event-based cameras offer a higher dynamic range (up to 140 dB vs. 80 dB) and a higher temporal resolution (in the order of μ s vs. 10s of ms). Thus, the captured events can encode rich temporal cues under challenging lighting conditions. However, these events lack texture information, posing problems in decoding temporal information effectively. SEEN tackles this issue from two different perspectives. First, we adopt convolutional spiking layers to take advantage of the spiking neural network's ability to decode pertinent temporal information. Second, SEEN learns to extract essential spatial cues from corresponding intensity frames and leverages a novel weight-copy scheme to convey spatial attention to the convolutional spiking layers during training and inference. We extensively validate and demonstrate the effectiveness of our approach on a specially collected Single-eye Event-based Emotion (SEE) dataset. To the best of our knowledge, our method is the first eye-based emotion recognition method that leverages event-based cameras and spiking neural networks.

Original languageEnglish (US)
Title of host publicationProceedings - SIGGRAPH 2023 Conference Papers
EditorsStephen N. Spencer
PublisherAssociation for Computing Machinery, Inc
ISBN (Electronic)9798400701597
DOIs
StatePublished - Jul 23 2023
Event2023 Special Interest Group on Computer Graphics and Interactive Techniques Conference, SIGGRAPH 2023 - Los Angeles, United States
Duration: Aug 6 2023Aug 10 2023

Publication series

NameProceedings - SIGGRAPH 2023 Conference Papers

Conference

Conference2023 Special Interest Group on Computer Graphics and Interactive Techniques Conference, SIGGRAPH 2023
Country/TerritoryUnited States
CityLos Angeles
Period8/6/238/10/23

All Science Journal Classification (ASJC) codes

  • Software
  • Computer Vision and Pattern Recognition
  • Computer Graphics and Computer-Aided Design

Keywords

  • Event-based cameras
  • eye-based emotion recognition

Fingerprint

Dive into the research topics of 'In the Blink of an Eye: Event-based Emotion Recognition'. Together they form a unique fingerprint.

Cite this