Strategyproofing Peer Assessment via Partitioning: The Price in Terms of Evaluators’ Expertise

Komal Dhull, Steven Jecmen, Pravesh Kothari, Nihar B. Shah

Research output: Chapter in Book/Report/Conference proceedingConference contribution

4 Scopus citations

Abstract

Strategic behavior is a fundamental problem in a variety of real-world applications that require some form of peer assessment, such as peer grading of home works, grant proposal review, conference peer review of scientific papers, and peer assessment of employees in organizations. Since an individual’ sown work is in competition with the submissions the yare evaluating, they may provide dishonest evaluations to increase the relative standing of their own submission. This issue is typically addressed by partitioning the individuals and assigning them to evaluate the work of only those from different subsets. Although this method ensures strategy proofness, each submission may require a different type of expertise for effective evaluation. In this paper, we focus on finding an assignment of evaluators to submissions that maximizes assigned evaluators’ expertise subject to the constraint of strategy proofness. We analyze the price of strategy proofness: that is, the amount of compromise on the assigned evaluators’ expertise required in order to get strategy proofness. We establishesveral polynomial-time algorithms for strategy proof assignment long with assignment-quality guarantees. Finally, we evaluate the methods on a dataset from conference peer review.

Original languageEnglish (US)
Title of host publicationHCOMP 2022 - Proceedings of the 10th AAAI Conference on Human Computation and Crowdsourcing
EditorsJane Hsu, Ming Yin
PublisherAssociation for the Advancement of Artificial Intelligence
Pages53-63
Number of pages11
ISBN (Print)9781577358787
DOIs
StatePublished - 2022
Externally publishedYes
Event10th AAAI Conference on Human Computation and Crowdsourcing, HCOMP 2022 - Virtual, Online
Duration: Nov 6 2022Nov 10 2022

Publication series

NameProceedings of the AAAI Conference on Human Computation and Crowdsourcing
Volume10
ISSN (Print)2769-1330
ISSN (Electronic)2769-1349

Conference

Conference10th AAAI Conference on Human Computation and Crowdsourcing, HCOMP 2022
CityVirtual, Online
Period11/6/2211/10/22

All Science Journal Classification (ASJC) codes

  • Computational Theory and Mathematics
  • Human-Computer Interaction

Fingerprint

Dive into the research topics of 'Strategyproofing Peer Assessment via Partitioning: The Price in Terms of Evaluators’ Expertise'. Together they form a unique fingerprint.

Cite this