Mrqa 2019 shared task: Evaluating generalization in reading comprehension

Adam Fisch, Alon Talmor, Robin Jia, Minjoon Seo, Eunsol Choi, Danqi Chen

Research output: Chapter in Book/Report/Conference proceedingConference contribution

72 Scopus citations

Abstract

We present the results of the Machine Reading for Question Answering (MRQA) 2019 shared task on evaluating the generalization capabilities of reading comprehension systems. 1 In this task we adapted and unified 18 distinct question answering datasets into the same format. Among them six datasets were made available for training six datasets were made available for development and the final six were hidden for final evaluation. Ten teams submitted systems which explored various ideas including data sampling multi-task learning adversarial training and ensembling. The best system achieved an average F1 score of 72.5 on the 12 held-out datasets 10.7 absolute points higher than our initial baseline based on BERT.

Original languageEnglish (US)
Title of host publicationMRQA@EMNLP 2019 - Proceedings of the 2nd Workshop on Machine Reading for Question Answering
PublisherAssociation for Computational Linguistics (ACL)
Pages1-13
Number of pages13
ISBN (Electronic)9781950737819
StatePublished - 2019
Event2nd Workshop on Machine Reading for Question Answering, MRQA@EMNLP 2019 - Hong Kong, China
Duration: Nov 4 2019 → …

Publication series

NameMRQA@EMNLP 2019 - Proceedings of the 2nd Workshop on Machine Reading for Question Answering

Conference

Conference2nd Workshop on Machine Reading for Question Answering, MRQA@EMNLP 2019
Country/TerritoryChina
CityHong Kong
Period11/4/19 → …

All Science Journal Classification (ASJC) codes

  • Computer Graphics and Computer-Aided Design
  • Artificial Intelligence

Fingerprint

Dive into the research topics of 'Mrqa 2019 shared task: Evaluating generalization in reading comprehension'. Together they form a unique fingerprint.

Cite this