Quantifying idiosyncratic and shared contributions to judgment

Joel E. Martinez, Friederike Funk, Alexander Todorov

Research output: Contribution to journalArticlepeer-review

29 Scopus citations


Identifying relative idiosyncratic and shared contributions to judgments is a fundamental challenge to the study of human behavior, yet there is no established method for estimating these contributions. Using edge cases of stimuli varying in intrarater reliability and interrater agreement—faces (high on both), objects (high on the former, low on the latter), and complex patterns (low on both)—we showed that variance component analyses (VCAs) accurately captured the psychometric properties of the data (Study 1). Simulations showed that the VCA generalizes to any arbitrary continuous rating and that both sample and stimulus set size affect estimate precision (Study 2). Generally, a minimum of 60 raters and 30 stimuli provided reasonable estimates within our simulations. Furthermore, VCA estimates stabilized given more than two repeated measures, consistent with the finding that both intrarater reliability and interrater agreement increased nonlinearly with repeated measures (Study 3). The VCA provides a rigorous examination of where variance lies in data, can be implemented using mixed models with crossed random effects, and is general enough to be useful in any judgment domain in which agreement and disagreement are important to quantify and in which multiple raters independently rate multiple stimuli.

Original languageEnglish (US)
Pages (from-to)1428-1444
Number of pages17
JournalBehavior Research Methods
Issue number4
StatePublished - Aug 1 2020
Externally publishedYes

All Science Journal Classification (ASJC) codes

  • Experimental and Cognitive Psychology
  • Developmental and Educational Psychology
  • Arts and Humanities (miscellaneous)
  • Psychology (miscellaneous)
  • General Psychology


  • Interrater agreement
  • Judgment
  • Measurement error
  • Multilevel modeling
  • Variance component analysis


Dive into the research topics of 'Quantifying idiosyncratic and shared contributions to judgment'. Together they form a unique fingerprint.

Cite this