Trick or treat: Putting peer prediction to the test

Xi Alice Gao, Andrew Mao, Yiling Chen, Ryan Prescott Adams

Research output: Chapter in Book/Report/Conference proceedingConference contribution

14 Scopus citations

Abstract

Collecting truthful subjective information from multiple individuals is an important problem in many social and online systems. While peer prediction mechanisms promise to elicit truthful information by rewarding participants with carefully constructed payments, they also admit uninformative equilibria where coordinating participants provide no useful information. To understand how participants behave towards such mechanisms in practice, we conduct the first controlled online experiment of a peer prediction mechanism, engaging the participants in a multiplayer, real-time and repeated game. Using a hidden Markov model to capture players' strategies from their actions, our results show that participants successfully coordinate on uninformative equilibria and the truthful equilibrium is not focal, even when some uninformative equilibria do not exist or are undesirable. In contrast, most players are consistently truthful in the absence of peer prediction, suggesting that these mechanisms may be harmful when truthful reporting has similar cost to strategic behavior.

Original languageEnglish (US)
Title of host publicationEC 2014 - Proceedings of the 15th ACM Conference on Economics and Computation
PublisherAssociation for Computing Machinery
Pages507-524
Number of pages18
ISBN (Print)9781450325653
DOIs
StatePublished - Jan 1 2014
Externally publishedYes
Event15th ACM Conference on Economics and Computation, EC 2014 - Palo Alto, CA, United States
Duration: Jun 8 2014Jun 12 2014

Publication series

NameEC 2014 - Proceedings of the 15th ACM Conference on Economics and Computation

Other

Other15th ACM Conference on Economics and Computation, EC 2014
CountryUnited States
CityPalo Alto, CA
Period6/8/146/12/14

All Science Journal Classification (ASJC) codes

  • Computer Science (miscellaneous)

Keywords

  • hidden markov models
  • online behavioral experiment
  • peer prediction

Fingerprint Dive into the research topics of 'Trick or treat: Putting peer prediction to the test'. Together they form a unique fingerprint.

  • Cite this

    Gao, X. A., Mao, A., Chen, Y., & Adams, R. P. (2014). Trick or treat: Putting peer prediction to the test. In EC 2014 - Proceedings of the 15th ACM Conference on Economics and Computation (pp. 507-524). (EC 2014 - Proceedings of the 15th ACM Conference on Economics and Computation). Association for Computing Machinery. https://doi.org/10.1145/2600057.2602865