TY - JOUR
T1 - Beyond playing 20 questions with nature
T2 - Integrative experiment design in the social and behavioral sciences
AU - Almaatouq, Abdullah
AU - Griffiths, Thomas L.
AU - Suchow, Jordan W.
AU - Whiting, Mark E.
AU - Evans, James
AU - Watts, Duncan J.
N1 - Publisher Copyright:
Copyright © The Author(s), 2022. Published by Cambridge University Press.
PY - 2022
Y1 - 2022
N2 - The dominant paradigm of experiments in the social and behavioral sciences views an experiment as a test of a theory, where the theory is assumed to generalize beyond the experiment's specific conditions. According to this view, which Alan Newell once characterized as playing twenty questions with nature, theory is advanced one experiment at a time, and the integration of disparate findings is assumed to happen via the scientific publishing process. In this article, we argue that the process of integration is at best inefficient, and at worst it does not, in fact, occur. We further show that the challenge of integration cannot be adequately addressed by recently proposed reforms that focus on the reliability and replicability of individual findings, nor simply by conducting more or larger experiments. Rather, the problem arises from the imprecise nature of social and behavioral theories and, consequently, a lack of commensurability across experiments conducted under different conditions. Therefore, researchers must fundamentally rethink how they design experiments and how the experiments relate to theory. We specifically describe an alternative framework, integrative experiment design, which intrinsically promotes commensurability and continuous integration of knowledge. In this paradigm, researchers explicitly map the design space of possible experiments associated with a given research question, embracing many potentially relevant theories rather than focusing on just one. Researchers then iteratively generate theories and test them with experiments explicitly sampled from the design space, allowing results to be integrated across experiments. Given recent methodological and technological developments, we conclude that this approach is feasible and would generate more-reliable, more-cumulative empirical and theoretical knowledge than the current paradigm - and with far greater efficiency.
AB - The dominant paradigm of experiments in the social and behavioral sciences views an experiment as a test of a theory, where the theory is assumed to generalize beyond the experiment's specific conditions. According to this view, which Alan Newell once characterized as playing twenty questions with nature, theory is advanced one experiment at a time, and the integration of disparate findings is assumed to happen via the scientific publishing process. In this article, we argue that the process of integration is at best inefficient, and at worst it does not, in fact, occur. We further show that the challenge of integration cannot be adequately addressed by recently proposed reforms that focus on the reliability and replicability of individual findings, nor simply by conducting more or larger experiments. Rather, the problem arises from the imprecise nature of social and behavioral theories and, consequently, a lack of commensurability across experiments conducted under different conditions. Therefore, researchers must fundamentally rethink how they design experiments and how the experiments relate to theory. We specifically describe an alternative framework, integrative experiment design, which intrinsically promotes commensurability and continuous integration of knowledge. In this paradigm, researchers explicitly map the design space of possible experiments associated with a given research question, embracing many potentially relevant theories rather than focusing on just one. Researchers then iteratively generate theories and test them with experiments explicitly sampled from the design space, allowing results to be integrated across experiments. Given recent methodological and technological developments, we conclude that this approach is feasible and would generate more-reliable, more-cumulative empirical and theoretical knowledge than the current paradigm - and with far greater efficiency.
KW - (in)commensurability
KW - cumulative knowledge
KW - experiments
KW - generalizability
UR - http://www.scopus.com/inward/record.url?scp=85144222227&partnerID=8YFLogxK
UR - http://www.scopus.com/inward/citedby.url?scp=85144222227&partnerID=8YFLogxK
U2 - 10.1017/S0140525X22002874
DO - 10.1017/S0140525X22002874
M3 - Article
C2 - 36539303
AN - SCOPUS:85144222227
SN - 0140-525X
VL - 47
JO - Behavioral and Brain Sciences
JF - Behavioral and Brain Sciences
M1 - e33
ER -