TY - JOUR
T1 - Scaling up psychology via Scientific Regret Minimization
AU - Agrawal, Mayank
AU - Peterson, Joshua C.
AU - Griffiths, Thomas L.
N1 - Funding Information:
ACKNOWLEDGMENTS. We thank Edmond Awad for providing guidance on navigating the Moral Machine dataset. M.A. is supported by the National Defense Science and Engineering Graduate Fellowship Program.
Publisher Copyright:
© 2020 National Academy of Sciences. All rights reserved.
PY - 2020/4/21
Y1 - 2020/4/21
N2 - Do large datasets provide value to psychologists? Without a systematic methodology for working with such datasets, there is a valid concern that analyses will produce noise artifacts rather than true effects. In this paper, we offer a way to enable researchers to systematically build models and identify novel phenomena in large datasets. One traditional approach is to analyze the residuals of models—the biggest errors they make in predicting the data—to discover what might be missing from those models. However, once a dataset is sufficiently large, machine learning algorithms approximate the true underlying function better than the data, suggesting, instead, that the predictions of these data-driven models should be used to guide model building. We call this approach “Scientific Regret Minimization” (SRM), as it focuses on minimizing errors for cases that we know should have been predictable. We apply this exploratory method on a subset of the Moral Machine dataset, a public collection of roughly 40 million moral decisions. Using SRM, we find that incorporating a set of deontological principles that capture dimensions along which groups of agents can vary (e.g., sex and age) improves a computational model of human moral judgment. Furthermore, we are able to identify and independently validate three interesting moral phenomena: criminal dehumanization, age of responsibility, and asymmetric notions of responsibility.
AB - Do large datasets provide value to psychologists? Without a systematic methodology for working with such datasets, there is a valid concern that analyses will produce noise artifacts rather than true effects. In this paper, we offer a way to enable researchers to systematically build models and identify novel phenomena in large datasets. One traditional approach is to analyze the residuals of models—the biggest errors they make in predicting the data—to discover what might be missing from those models. However, once a dataset is sufficiently large, machine learning algorithms approximate the true underlying function better than the data, suggesting, instead, that the predictions of these data-driven models should be used to guide model building. We call this approach “Scientific Regret Minimization” (SRM), as it focuses on minimizing errors for cases that we know should have been predictable. We apply this exploratory method on a subset of the Moral Machine dataset, a public collection of roughly 40 million moral decisions. Using SRM, we find that incorporating a set of deontological principles that capture dimensions along which groups of agents can vary (e.g., sex and age) improves a computational model of human moral judgment. Furthermore, we are able to identify and independently validate three interesting moral phenomena: criminal dehumanization, age of responsibility, and asymmetric notions of responsibility.
KW - Decision-making
KW - Machine learning
KW - Moral psychology
KW - Scientific regret
UR - http://www.scopus.com/inward/record.url?scp=85083482980&partnerID=8YFLogxK
UR - http://www.scopus.com/inward/citedby.url?scp=85083482980&partnerID=8YFLogxK
U2 - 10.1073/pnas.1915841117
DO - 10.1073/pnas.1915841117
M3 - Article
C2 - 32241896
AN - SCOPUS:85083482980
SN - 0027-8424
VL - 117
SP - 8825
EP - 8835
JO - Proceedings of the National Academy of Sciences of the United States of America
JF - Proceedings of the National Academy of Sciences of the United States of America
IS - 16
ER -