Word Embeddings: What Works, What Doesn’t, and How to Tell the Difference for Applied Research

Pedro L. Rodriguez, Arthur Spirling

Research output: Contribution to journalArticlepeer-review

62 Scopus citations

Abstract

Word embeddings are becoming popular for political science research, yet we know little about their properties and performance. To help scholars seeking to use these techniques, we explore the effects of key parameter choices—including context window length, embedding vector dimensions, and pretrained versus locally fit variants—on the efficiency and quality of inferences possible with these models. Reassuringly we show that results are generally robust to such choices for political corpora of various sizes and in various languages. Beyond reporting extensive technical findings, we provide a novel crowdsourced “Turing test”–style method for examining the relative performance of any two models that produce substantive, text-based outputs. Our results are encouraging: popular, easily available pretrained embeddings perform at a level close to—or surpassing—both human coders and more complicated locally fit models. For completeness, we provide best practice advice for cases where local fitting is required.

Original languageEnglish (US)
Pages (from-to)101-115
Number of pages15
JournalJournal of Politics
Volume84
Issue number1
DOIs
StatePublished - Jan 2022
Externally publishedYes

All Science Journal Classification (ASJC) codes

  • Sociology and Political Science

Fingerprint

Dive into the research topics of 'Word Embeddings: What Works, What Doesn’t, and How to Tell the Difference for Applied Research'. Together they form a unique fingerprint.

Cite this