Using Imperfect Surrogates for Downstream Inference: Design-based Supervised Learning for Social Science Applications of Large Language Models

Naoki Egami, Musashi Hinck, Brandon M. Stewart, Hanying Wei

Research output: Contribution to journalConference articlepeer-review

Abstract

In computational social science (CSS), researchers analyze documents to explain social and political phenomena.In most scenarios, CSS researchers first obtain labels for documents and then explain labels using interpretable regression analyses in the second step.One increasingly common way to annotate documents cheaply at scale is through large language models (LLMs).However, like other scalable ways of producing annotations, such surrogate labels are often imperfect and biased.We present a new algorithm for using imperfect annotation surrogates for downstream statistical analyses while guaranteeing statistical properties-like asymptotic unbiasedness and proper uncertainty quantification-which are fundamental to CSS research.We show that direct use of surrogate labels in downstream statistical analyses leads to substantial bias and invalid confidence intervals, even with high surrogate accuracy of 80-90%.To address this, we build on debiased machine learning to propose the design-based supervised learning (DSL) estimator.DSL employs a doubly-robust procedure to combine surrogate labels with a smaller number of high-quality, gold-standard labels.Our approach guarantees valid inference for downstream statistical analyses, even when surrogates are arbitrarily biased and without requiring stringent assumptions, by controlling the probability of sampling documents for gold-standard labeling.Both our theoretical analysis and experimental results show that DSL provides valid statistical inference while achieving root mean squared errors comparable to existing alternatives that focus only on prediction without inferential guarantees.

Original languageEnglish (US)
JournalAdvances in Neural Information Processing Systems
Volume36
StatePublished - 2023
Event37th Conference on Neural Information Processing Systems, NeurIPS 2023 - New Orleans, United States
Duration: Dec 10 2023Dec 16 2023

All Science Journal Classification (ASJC) codes

  • Computer Networks and Communications
  • Information Systems
  • Signal Processing

Fingerprint

Dive into the research topics of 'Using Imperfect Surrogates for Downstream Inference: Design-based Supervised Learning for Social Science Applications of Large Language Models'. Together they form a unique fingerprint.

Cite this