Effectively Crowdsourcing Radiology Report Annotations

Anne Cocos, Aaron J. Masino, Ting Qian, Ellie Pavlick, Chris Callison-Burch

Research output: Chapter in Book/Report/Conference proceedingConference contribution

9 Scopus citations

Abstract

Crowdsourcing platforms are a popular choice for researchers to gather text annotations quickly at scale. We investigate whether crowdsourced annotations are useful when the labeling task requires medical domain knowledge. Comparing a sentence classification model trained with expert-annotated sentences to the same model trained on crowd-labeled sentences, we find the crowdsourced training data to be just as effective as the manually produced dataset. We can improve the accuracy of the crowd-fueled model without collecting further labels by filtering out worker labels applied with low confidence.

Original languageEnglish (US)
Title of host publicationEMNLP 2015 - 6th International Workshop on Health Text Mining and Information Analysis, LOUHI 2015 - Proceedings of the Workshop
PublisherAssociation for Computational Linguistics (ACL)
Pages109-114
Number of pages6
ISBN (Electronic)9781941643327
StatePublished - 2015
Externally publishedYes
Event6th International Workshop on Health Text Mining and Information Analysis, LOUHI 2015, co-located with EMNLP 2015 - Lisbon, Portugal
Duration: Sep 17 2015 → …

Publication series

NameEMNLP 2015 - 6th International Workshop on Health Text Mining and Information Analysis, LOUHI 2015 - Proceedings of the Workshop

Conference

Conference6th International Workshop on Health Text Mining and Information Analysis, LOUHI 2015, co-located with EMNLP 2015
Country/TerritoryPortugal
CityLisbon
Period9/17/15 → …

All Science Journal Classification (ASJC) codes

  • Computational Theory and Mathematics
  • Computer Science Applications
  • Civil and Structural Engineering
  • Health Informatics

Fingerprint

Dive into the research topics of 'Effectively Crowdsourcing Radiology Report Annotations'. Together they form a unique fingerprint.

Cite this