A la carte embedding: Cheap but effective induction of semantic feature vectors

Mikhail Khodak, Nikunj Saunshi, Yingyu Liang, Tengyu Ma, Brandon Michael Stewart, Sanjeev Arora

Research output: Chapter in Book/Report/Conference proceedingConference contribution

13 Scopus citations

Abstract

Motivations like domain adaptation, transfer learning, and feature learning have fueled interest in inducing embeddings for rare or unseen words, n-grams, synsets, and other textual features. This paper introduces à la carte embedding, a simple and general alternative to the usual word2vec-based approaches for building such representations that is based upon recent theoretical results for GloVe-like embeddings. Our method relies mainly on a linear transformation that is efficiently learnable using pretrained word vectors and linear regression. This transform is applicable “on the fly” in the future when a new text feature or rare word is encountered, even if only a single usage example is available. We introduce a new dataset showing how the à la carte method requires fewer examples of words in context to learn high-quality embeddings and we obtain state-of-the-art results on a nonce task and some unsupervised document classification tasks.

Original languageEnglish (US)
Title of host publicationACL 2018 - 56th Annual Meeting of the Association for Computational Linguistics, Proceedings of the Conference (Long Papers)
PublisherAssociation for Computational Linguistics (ACL)
Pages12-22
Number of pages11
ISBN (Electronic)9781948087322
DOIs
StatePublished - 2018
Event56th Annual Meeting of the Association for Computational Linguistics, ACL 2018 - Melbourne, Australia
Duration: Jul 15 2018Jul 20 2018

Publication series

NameACL 2018 - 56th Annual Meeting of the Association for Computational Linguistics, Proceedings of the Conference (Long Papers)
Volume1

Conference

Conference56th Annual Meeting of the Association for Computational Linguistics, ACL 2018
CountryAustralia
CityMelbourne
Period7/15/187/20/18

All Science Journal Classification (ASJC) codes

  • Software
  • Computational Theory and Mathematics

Fingerprint Dive into the research topics of 'A la carte embedding: Cheap but effective induction of semantic feature vectors'. Together they form a unique fingerprint.

  • Cite this

    Khodak, M., Saunshi, N., Liang, Y., Ma, T., Stewart, B. M., & Arora, S. (2018). A la carte embedding: Cheap but effective induction of semantic feature vectors. In ACL 2018 - 56th Annual Meeting of the Association for Computational Linguistics, Proceedings of the Conference (Long Papers) (pp. 12-22). (ACL 2018 - 56th Annual Meeting of the Association for Computational Linguistics, Proceedings of the Conference (Long Papers); Vol. 1). Association for Computational Linguistics (ACL). https://doi.org/10.18653/v1/p18-1002