Mapping between fMRI responses to movies and their natural language annotations

Kiran Vodrahalli, Po Hsuan Chen, Yingyu Liang, Christopher Baldassano, Janice Chen, Esther Yong, Christopher Honey, Uri Hasson, Peter Jeffrey Ramadge, Kenneth Andrew Norman, Sanjeev Arora

Research output: Contribution to journalArticle

14 Scopus citations

Abstract

Several research groups have shown how to map fMRI responses to the meanings of presented stimuli. This paper presents new methods for doing so when only a natural language annotation is available as the description of the stimulus. We study fMRI data gathered from subjects watching an episode of BBCs Sherlock (Chen et al., 2017), and learn bidirectional mappings between fMRI responses and natural language representations. By leveraging data from multiple subjects watching the same movie, we were able to perform scene classification with 72% accuracy (random guessing would give 4%) and scene ranking with average rank in the top 4% (random guessing would give 50%). The key ingredients underlying this high level of performance are (a) the use of the Shared Response Model (SRM) and its variant SRM-ICA (Chen et al., 2015; Zhang et al., 2016) to aggregate fMRI data from multiple subjects, both of which are shown to be superior to standard PCA in producing low-dimensional representations for the tasks in this paper; (b) a sentence embedding technique adapted from the natural language processing (NLP) literature (Arora et al., 2017) that produces semantic vector representation of the annotations; (c) using previous timestep information in the featurization of the predictor data. These optimizations in how we featurize the fMRI data and text annotations provide a substantial improvement in classification performance, relative to standard approaches.

Original languageEnglish (US)
Pages (from-to)223-231
Number of pages9
JournalNeuroimage
Volume180
DOIs
StatePublished - Oct 15 2018

All Science Journal Classification (ASJC) codes

  • Neurology
  • Cognitive Neuroscience

Keywords

  • FMRI
  • Multi-modal model
  • Natural language processing
  • Natural movie stimulus
  • Shared response model
  • Text annotations

Fingerprint Dive into the research topics of 'Mapping between fMRI responses to movies and their natural language annotations'. Together they form a unique fingerprint.

  • Cite this