Prevalence and Prevention of Large Language Model Use in Crowd Work

Crowd work platforms, such as Prolific and Amazon Mechanical Turk, play an important part in academia and industry, empowering the creation, annotation, and summarization of data,¹¹ as well as surveys and experiments.²¹ At the same time, large language models (LLMs), such as ChatGPT, Gemini, and Claude, promise similar capabilities. They are remarkable data annotators¹⁰ and can, in some cases, accurately simulate human behavior, enabling in-silico experiments and surveys that yield human-like results.² Yet, if crowd workers were to start using LLMs, this could threaten the validity of data generated using crowd work platforms. Sometimes, researchers seek to observe unaided human responses (even if LLMs could provide a good proxy), and LLMs still often fail to accurately simulate human behavior.²² Further, LLM-generated data may degrade subsequent models trained on it.²³ Here, we investigate the extent to which crowd workers use LLMs in a text-production task and whether targeted mitigation strategies can prevent LLM use.