MoQA: Benchmarking Multi-Type Open-Domain Question Answering

Howard Yen, Tianyu Gao, Jinhyuk Lee, Danqi Chen

Research output: Chapter in Book/Report/Conference proceedingConference contribution

Abstract

Previous research on open-domain question answering (QA) focuses mainly on short-answered questions. However, information-seeking QA often requires various formats of answers depending on the nature of the questions, e.g., why/how questions typically require a long answer. In this paper, we present MOQA1, a benchmark for open-domain QA that requires building one system that can provide short, medium, long, and yes/no answers to different questions accordingly. MOQA builds upon Natural Questions (Kwiatkowski et al., 2019) with multiple types of questions and additional crowd-sourcing efforts to ensure high data quality. We adapt state-of-the-art models, and reveal unique findings in multi-type open-domain QA: (1) For retriever-reader models, training one retriever on all types achieves the overall best performance, but it is challenging to train one reader model to output answers of different formats, or to train a question classifier to distinguish between types; (2) An end-to-end closed-book QA model trained on multiple types struggles with the task across the board; (3) State-of-the-art large language models such as the largest GPT-3 models (Brown et al., 2020; Ouyang et al., 2022) also lag behind open-book QA models. Our benchmark and analysis call for more effort to build versatile open-domain QA models in the future.

Original languageEnglish (US)
Title of host publicationDialDoc 2023 - Proceedings of the 3rd DialDoc Workshop on Document-Grounded Dialogue and Conversational Question Answering, Proceedings of the Workshop
EditorsSmaranda Muresan, Vivian Chen, Casey Kennington, David Vandyke, Nina Dethlefs, Koji Inoue, Erik Ekstedt, Stefan Ultes
PublisherAssociation for Computational Linguistics (ACL)
Pages8-29
Number of pages22
ISBN (Electronic)9781959429982
StatePublished - 2023
Externally publishedYes
Event3rd Workshop on Document-grounded Dialogue and Conversational Question Answering, DialDoc 2023, co-located with ACL 2023 - Toronto, Canada
Duration: Jul 13 2023 → …

Publication series

NameProceedings of the Annual Meeting of the Association for Computational Linguistics
ISSN (Print)0736-587X

Conference

Conference3rd Workshop on Document-grounded Dialogue and Conversational Question Answering, DialDoc 2023, co-located with ACL 2023
Country/TerritoryCanada
CityToronto
Period7/13/23 → …

All Science Journal Classification (ASJC) codes

  • Computer Science Applications
  • Linguistics and Language
  • Language and Linguistics

Fingerprint

Dive into the research topics of 'MoQA: Benchmarking Multi-Type Open-Domain Question Answering'. Together they form a unique fingerprint.

Cite this