A Bayesian framework for word segmentation: Exploring the effects of context

Sharon Goldwater, Thomas L. Griffiths, Mark Johnson

Research output: Contribution to journalArticle

214 Scopus citations

Abstract

Since the experiments of Saffran et al. [Saffran, J., Aslin, R., & Newport, E. (1996). Statistical learning in 8-month-old infants. Science, 274, 1926-1928], there has been a great deal of interest in the question of how statistical regularities in the speech stream might be used by infants to begin to identify individual words. In this work, we use computational modeling to explore the effects of different assumptions the learner might make regarding the nature of words - in particular, how these assumptions affect the kinds of words that are segmented from a corpus of transcribed child-directed speech. We develop several models within a Bayesian ideal observer framework, and use them to examine the consequences of assuming either that words are independent units, or units that help to predict other units. We show through empirical and theoretical results that the assumption of independence causes the learner to undersegment the corpus, with many two- and three-word sequences (e.g. what's that, do you, in the house) misidentified as individual words. In contrast, when the learner assumes that words are predictive, the resulting segmentation is far more accurate. These results indicate that taking context into account is important for a statistical word segmentation strategy to be successful, and raise the possibility that even young infants may be able to exploit more subtle statistical patterns than have usually been considered.

Original languageEnglish (US)
Pages (from-to)21-54
Number of pages34
JournalCognition
Volume112
Issue number1
DOIs
StatePublished - Jul 2009

All Science Journal Classification (ASJC) codes

  • Experimental and Cognitive Psychology
  • Language and Linguistics
  • Developmental and Educational Psychology
  • Linguistics and Language
  • Cognitive Neuroscience

Keywords

  • Bayesian
  • Computational modeling
  • Language acquisition
  • Word segmentation

Fingerprint Dive into the research topics of 'A Bayesian framework for word segmentation: Exploring the effects of context'. Together they form a unique fingerprint.

  • Cite this