Spanbert: Improving pre-training by representing and predicting spans

Mandar Joshi; Danqi Chen; Yinhan Liu; Daniel S. Weld; Luke Zettlemoyer; Omer Levy

doi:10.1162/tacl_a_00300

Spanbert: Improving pre-training by representing and predicting spans

Mandar Joshi, Danqi Chen, Yinhan Liu, Daniel S. Weld, Luke Zettlemoyer, Omer Levy

Research output: Contribution to journal › Article › peer-review

1264 Scopus citations

Abstract

We present SpanBERT, a pre-training method that is designed to better represent and predict spans of text. Our approach extends BERT by (1) masking contiguous random spans, rather than random tokens, and (2) training the span boundary representations to predict the entire content of the masked span, without relying on the individual token representations within it. SpanBERT consistently outperforms BERT and our better-tuned baselines, with substantial gains on span selection tasks such as question answering and coreference resolution. In particular, with the same training data and model size as BERT_large, our single model obtains 94.6% and 88.7% F1 on SQuAD 1.1 and 2.0 respectively. We also achieve a new state of the art on the OntoNotes coreference resolution task (79.6% F1), strong performance on the TACRED relation extraction benchmark, and even gains on GLUE.¹

Original language	English (US)
Pages (from-to)	64-77
Number of pages	14
Journal	Transactions of the Association for Computational Linguistics
Volume	8
DOIs	https://doi.org/10.1162/tacl_a_00300
State	Published - 2020
Externally published	Yes

All Science Journal Classification (ASJC) codes

Artificial Intelligence
Communication
Human-Computer Interaction
Computer Science Applications
Linguistics and Language

Access to Document

10.1162/tacl_a_00300

Cite this

@article{be58495adf1a4ecbb8bef7a6361bf65a,

title = "Spanbert: Improving pre-training by representing and predicting spans",

abstract = "We present SpanBERT, a pre-training method that is designed to better represent and predict spans of text. Our approach extends BERT by (1) masking contiguous random spans, rather than random tokens, and (2) training the span boundary representations to predict the entire content of the masked span, without relying on the individual token representations within it. SpanBERT consistently outperforms BERT and our better-tuned baselines, with substantial gains on span selection tasks such as question answering and coreference resolution. In particular, with the same training data and model size as BERTlarge, our single model obtains 94.6% and 88.7% F1 on SQuAD 1.1 and 2.0 respectively. We also achieve a new state of the art on the OntoNotes coreference resolution task (79.6% F1), strong performance on the TACRED relation extraction benchmark, and even gains on GLUE.1",

author = "Mandar Joshi and Danqi Chen and Yinhan Liu and Weld, {Daniel S.} and Luke Zettlemoyer and Omer Levy",

note = "Publisher Copyright: {\textcopyright} 2020 Association for Computational Linguistics.",

year = "2020",

doi = "10.1162/tacl_a_00300",

language = "English (US)",

volume = "8",

pages = "64--77",

journal = "Transactions of the Association for Computational Linguistics",

issn = "2307-387X",

publisher = "MIT Press Journals",

}

TY - JOUR

T1 - Spanbert

T2 - Improving pre-training by representing and predicting spans

AU - Joshi, Mandar

AU - Chen, Danqi

AU - Liu, Yinhan

AU - Weld, Daniel S.

AU - Zettlemoyer, Luke

AU - Levy, Omer

PY - 2020

Y1 - 2020

N2 - We present SpanBERT, a pre-training method that is designed to better represent and predict spans of text. Our approach extends BERT by (1) masking contiguous random spans, rather than random tokens, and (2) training the span boundary representations to predict the entire content of the masked span, without relying on the individual token representations within it. SpanBERT consistently outperforms BERT and our better-tuned baselines, with substantial gains on span selection tasks such as question answering and coreference resolution. In particular, with the same training data and model size as BERTlarge, our single model obtains 94.6% and 88.7% F1 on SQuAD 1.1 and 2.0 respectively. We also achieve a new state of the art on the OntoNotes coreference resolution task (79.6% F1), strong performance on the TACRED relation extraction benchmark, and even gains on GLUE.1

AB - We present SpanBERT, a pre-training method that is designed to better represent and predict spans of text. Our approach extends BERT by (1) masking contiguous random spans, rather than random tokens, and (2) training the span boundary representations to predict the entire content of the masked span, without relying on the individual token representations within it. SpanBERT consistently outperforms BERT and our better-tuned baselines, with substantial gains on span selection tasks such as question answering and coreference resolution. In particular, with the same training data and model size as BERTlarge, our single model obtains 94.6% and 88.7% F1 on SQuAD 1.1 and 2.0 respectively. We also achieve a new state of the art on the OntoNotes coreference resolution task (79.6% F1), strong performance on the TACRED relation extraction benchmark, and even gains on GLUE.1

UR - http://www.scopus.com/inward/record.url?scp=85098395373&partnerID=8YFLogxK

UR - http://www.scopus.com/inward/citedby.url?scp=85098395373&partnerID=8YFLogxK

U2 - 10.1162/tacl_a_00300

DO - 10.1162/tacl_a_00300

M3 - Article

AN - SCOPUS:85098395373

SN - 2307-387X

VL - 8

SP - 64

EP - 77

JO - Transactions of the Association for Computational Linguistics

JF - Transactions of the Association for Computational Linguistics

ER -

Spanbert: Improving pre-training by representing and predicting spans

Abstract

All Science Journal Classification (ASJC) codes

Access to Document

Other files and links

Fingerprint

Cite this