TY - JOUR
T1 - iCatcher+
T2 - Robust and Automated Annotation of Infants’ and Young Children’s Gaze Behavior From Videos Collected in Laboratory, Field, and Online Studies
AU - Erel, Yotam
AU - Shannon, Katherine Adams
AU - Chu, Junyi
AU - Scott, Kim
AU - Kline Struhl, Melissa
AU - Cao, Peng
AU - Tan, Xincheng
AU - Hart, Peter
AU - Raz, Gal
AU - Piccolo, Sabrina
AU - Mei, Catherine
AU - Potter, Christine
AU - Jaffe-Dax, Sagi
AU - Lew-Williams, Casey
AU - Tenenbaum, Joshua
AU - Fairchild, Katherine
AU - Bermano, Amit
AU - Liu, Shari
N1 - Publisher Copyright:
© The Author(s) 2023.
PY - 2023/4/1
Y1 - 2023/4/1
N2 - Technological advances in psychological research have enabled large-scale studies of human behavior and streamlined pipelines for automatic processing of data. However, studies of infants and children have not fully reaped these benefits because the behaviors of interest, such as gaze duration and direction, still have to be extracted from video through a laborious process of manual annotation, even when these data are collected online. Recent advances in computer vision raise the possibility of automated annotation of these video data. In this article, we built on a system for automatic gaze annotation in young children, iCatcher, by engineering improvements and then training and testing the system (referred to hereafter as iCatcher+) on three data sets with substantial video and participant variability (214 videos collected in U.S. lab and field sites, 143 videos collected in Senegal field sites, and 265 videos collected via webcams in homes; participant age range = 4 months–3.5 years). When trained on each of these data sets, iCatcher+ performed with near human-level accuracy on held-out videos on distinguishing “LEFT” versus “RIGHT” and “ON” versus “OFF” looking behavior across all data sets. This high performance was achieved at the level of individual frames, experimental trials, and study videos; held across participant demographics (e.g., age, race/ethnicity), participant behavior (e.g., movement, head position), and video characteristics (e.g., luminance); and generalized to a fourth, entirely held-out online data set. We close by discussing next steps required to fully automate the life cycle of online infant and child behavioral studies, representing a key step toward enabling robust and high-throughput developmental research.
AB - Technological advances in psychological research have enabled large-scale studies of human behavior and streamlined pipelines for automatic processing of data. However, studies of infants and children have not fully reaped these benefits because the behaviors of interest, such as gaze duration and direction, still have to be extracted from video through a laborious process of manual annotation, even when these data are collected online. Recent advances in computer vision raise the possibility of automated annotation of these video data. In this article, we built on a system for automatic gaze annotation in young children, iCatcher, by engineering improvements and then training and testing the system (referred to hereafter as iCatcher+) on three data sets with substantial video and participant variability (214 videos collected in U.S. lab and field sites, 143 videos collected in Senegal field sites, and 265 videos collected via webcams in homes; participant age range = 4 months–3.5 years). When trained on each of these data sets, iCatcher+ performed with near human-level accuracy on held-out videos on distinguishing “LEFT” versus “RIGHT” and “ON” versus “OFF” looking behavior across all data sets. This high performance was achieved at the level of individual frames, experimental trials, and study videos; held across participant demographics (e.g., age, race/ethnicity), participant behavior (e.g., movement, head position), and video characteristics (e.g., luminance); and generalized to a fourth, entirely held-out online data set. We close by discussing next steps required to fully automate the life cycle of online infant and child behavioral studies, representing a key step toward enabling robust and high-throughput developmental research.
KW - cognitive development
KW - deep learning
KW - eye tracking
KW - online data collection
KW - open source
UR - http://www.scopus.com/inward/record.url?scp=85153610284&partnerID=8YFLogxK
UR - http://www.scopus.com/inward/citedby.url?scp=85153610284&partnerID=8YFLogxK
U2 - 10.1177/25152459221147250
DO - 10.1177/25152459221147250
M3 - Article
C2 - 37655047
AN - SCOPUS:85153610284
SN - 2515-2459
VL - 6
JO - Advances in Methods and Practices in Psychological Science
JF - Advances in Methods and Practices in Psychological Science
IS - 2
ER -