TY - GEN
T1 - Effectively Crowdsourcing Radiology Report Annotations
AU - Cocos, Anne
AU - Masino, Aaron J.
AU - Qian, Ting
AU - Pavlick, Ellie
AU - Callison-Burch, Chris
N1 - Publisher Copyright:
© 2015 Association for Computational Linguistics.
PY - 2015
Y1 - 2015
N2 - Crowdsourcing platforms are a popular choice for researchers to gather text annotations quickly at scale. We investigate whether crowdsourced annotations are useful when the labeling task requires medical domain knowledge. Comparing a sentence classification model trained with expert-annotated sentences to the same model trained on crowd-labeled sentences, we find the crowdsourced training data to be just as effective as the manually produced dataset. We can improve the accuracy of the crowd-fueled model without collecting further labels by filtering out worker labels applied with low confidence.
AB - Crowdsourcing platforms are a popular choice for researchers to gather text annotations quickly at scale. We investigate whether crowdsourced annotations are useful when the labeling task requires medical domain knowledge. Comparing a sentence classification model trained with expert-annotated sentences to the same model trained on crowd-labeled sentences, we find the crowdsourced training data to be just as effective as the manually produced dataset. We can improve the accuracy of the crowd-fueled model without collecting further labels by filtering out worker labels applied with low confidence.
UR - http://www.scopus.com/inward/record.url?scp=85040610826&partnerID=8YFLogxK
UR - http://www.scopus.com/inward/citedby.url?scp=85040610826&partnerID=8YFLogxK
M3 - Conference contribution
AN - SCOPUS:85040610826
T3 - EMNLP 2015 - 6th International Workshop on Health Text Mining and Information Analysis, LOUHI 2015 - Proceedings of the Workshop
SP - 109
EP - 114
BT - EMNLP 2015 - 6th International Workshop on Health Text Mining and Information Analysis, LOUHI 2015 - Proceedings of the Workshop
PB - Association for Computational Linguistics (ACL)
T2 - 6th International Workshop on Health Text Mining and Information Analysis, LOUHI 2015, co-located with EMNLP 2015
Y2 - 17 September 2015
ER -