TY - GEN
T1 - In the Blink of an Eye
T2 - 2023 Special Interest Group on Computer Graphics and Interactive Techniques Conference, SIGGRAPH 2023
AU - Zhang, Haiwei
AU - Zhang, Jiqing
AU - Dong, Bo
AU - Peers, Pieter
AU - Wu, Wenwei
AU - Wei, Xiaopeng
AU - Heide, Felix
AU - Yang, Xin
N1 - Publisher Copyright:
© 2023 ACM.
PY - 2023/7/23
Y1 - 2023/7/23
N2 - We introduce a wearable single-eye emotion recognition device and a real-time approach to recognizing emotions from partial observations of an emotion that is robust to changes in lighting conditions. At the heart of our method is a bio-inspired event-based camera setup and a newly designed lightweight Spiking Eye Emotion Network (SEEN). Compared to conventional cameras, event-based cameras offer a higher dynamic range (up to 140 dB vs. 80 dB) and a higher temporal resolution (in the order of μ s vs. 10s of ms). Thus, the captured events can encode rich temporal cues under challenging lighting conditions. However, these events lack texture information, posing problems in decoding temporal information effectively. SEEN tackles this issue from two different perspectives. First, we adopt convolutional spiking layers to take advantage of the spiking neural network's ability to decode pertinent temporal information. Second, SEEN learns to extract essential spatial cues from corresponding intensity frames and leverages a novel weight-copy scheme to convey spatial attention to the convolutional spiking layers during training and inference. We extensively validate and demonstrate the effectiveness of our approach on a specially collected Single-eye Event-based Emotion (SEE) dataset. To the best of our knowledge, our method is the first eye-based emotion recognition method that leverages event-based cameras and spiking neural networks.
AB - We introduce a wearable single-eye emotion recognition device and a real-time approach to recognizing emotions from partial observations of an emotion that is robust to changes in lighting conditions. At the heart of our method is a bio-inspired event-based camera setup and a newly designed lightweight Spiking Eye Emotion Network (SEEN). Compared to conventional cameras, event-based cameras offer a higher dynamic range (up to 140 dB vs. 80 dB) and a higher temporal resolution (in the order of μ s vs. 10s of ms). Thus, the captured events can encode rich temporal cues under challenging lighting conditions. However, these events lack texture information, posing problems in decoding temporal information effectively. SEEN tackles this issue from two different perspectives. First, we adopt convolutional spiking layers to take advantage of the spiking neural network's ability to decode pertinent temporal information. Second, SEEN learns to extract essential spatial cues from corresponding intensity frames and leverages a novel weight-copy scheme to convey spatial attention to the convolutional spiking layers during training and inference. We extensively validate and demonstrate the effectiveness of our approach on a specially collected Single-eye Event-based Emotion (SEE) dataset. To the best of our knowledge, our method is the first eye-based emotion recognition method that leverages event-based cameras and spiking neural networks.
KW - Event-based cameras
KW - eye-based emotion recognition
UR - http://www.scopus.com/inward/record.url?scp=85162809793&partnerID=8YFLogxK
UR - http://www.scopus.com/inward/citedby.url?scp=85162809793&partnerID=8YFLogxK
U2 - 10.1145/3588432.3591511
DO - 10.1145/3588432.3591511
M3 - Conference contribution
AN - SCOPUS:85162809793
T3 - Proceedings - SIGGRAPH 2023 Conference Papers
BT - Proceedings - SIGGRAPH 2023 Conference Papers
A2 - Spencer, Stephen N.
PB - Association for Computing Machinery, Inc
Y2 - 6 August 2023 through 10 August 2023
ER -