TY - JOUR
T1 - From convolutional neural networks to models of higher-level cognition (and back again)
AU - Battleday, Ruairidh M.
AU - Peterson, Joshua C.
AU - Griffiths, Thomas L.
N1 - Funding Information:
This work was conducted under Grant Number 1718550 from the National Science Foundation. R.M.B. drafted the manuscript. J.C.P. created the figures. T.L.G. conceived the paper design. All authors edited the manuscript. The authors would like to thank Thomas Palmeri and Jason Chow for their helpful comments in reviewing the manuscript.
Publisher Copyright:
© 2021 New York Academy of Sciences.
PY - 2021/12
Y1 - 2021/12
N2 - The remarkable successes of convolutional neural networks (CNNs) in modern computer vision are by now well known, and they are increasingly being explored as computational models of the human visual system. In this paper, we ask whether CNNs might also provide a basis for modeling higher-level cognition, focusing on the core phenomena of similarity and categorization. The most important advance comes from the ability of CNNs to learn high-dimensional representations of complex naturalistic images, substantially extending the scope of traditional cognitive models that were previously only evaluated with simple artificial stimuli. In all cases, the most successful combinations arise when CNN representations are used with cognitive models that have the capacity to transform them to better fit human behavior. One consequence of these insights is a toolkit for the integration of cognitively motivated constraints back into CNN training paradigms in computer vision and machine learning, and we review cases where this leads to improved performance. A second consequence is a roadmap for how CNNs and cognitive models can be more fully integrated in the future, allowing for flexible end-to-end algorithms that can learn representations from data while still retaining the structured behavior characteristic of human cognition.
AB - The remarkable successes of convolutional neural networks (CNNs) in modern computer vision are by now well known, and they are increasingly being explored as computational models of the human visual system. In this paper, we ask whether CNNs might also provide a basis for modeling higher-level cognition, focusing on the core phenomena of similarity and categorization. The most important advance comes from the ability of CNNs to learn high-dimensional representations of complex naturalistic images, substantially extending the scope of traditional cognitive models that were previously only evaluated with simple artificial stimuli. In all cases, the most successful combinations arise when CNN representations are used with cognitive models that have the capacity to transform them to better fit human behavior. One consequence of these insights is a toolkit for the integration of cognitively motivated constraints back into CNN training paradigms in computer vision and machine learning, and we review cases where this leads to improved performance. A second consequence is a roadmap for how CNNs and cognitive models can be more fully integrated in the future, allowing for flexible end-to-end algorithms that can learn representations from data while still retaining the structured behavior characteristic of human cognition.
UR - http://www.scopus.com/inward/record.url?scp=85116921760&partnerID=8YFLogxK
UR - http://www.scopus.com/inward/citedby.url?scp=85116921760&partnerID=8YFLogxK
U2 - 10.1111/nyas.14593
DO - 10.1111/nyas.14593
M3 - Review article
C2 - 33754368
AN - SCOPUS:85116921760
SN - 0077-8923
VL - 1505
SP - 55
EP - 78
JO - Annals of the New York Academy of Sciences
JF - Annals of the New York Academy of Sciences
IS - 1
ER -