A Neural Signature of Hierarchical Reinforcement Learning

José J.F. Ribas-Fernandes, Alec Solway, Carlos Diuk, Joseph T. McGuire, Andrew G. Barto, Yael Niv, Matthew M. Botvinick

Research output: Contribution to journalArticle

97 Scopus citations

Abstract

Human behavior displays hierarchical structure: simple actions cohere into subtask sequences, which work together to accomplish overall task goals. Although the neural substrates of such hierarchy have been the target of increasing research, they remain poorly understood. We propose that the computations supporting hierarchical behavior may relate to those in hierarchical reinforcement learning (HRL), a machine-learning framework that extends reinforcement-learning mechanisms into hierarchical domains. To test this, we leveraged a distinctive prediction arising from HRL. In ordinary reinforcement learning, reward prediction errors are computed when there is an unanticipated change in the prospects for accomplishing overall task goals. HRL entails that prediction errors should also occur in relation to task subgoals. In three neuroimaging studies we observed neural responses consistent with such subgoal-related reward prediction errors, within structures previously implicated in reinforcement learning. The results reported support the relevance of HRL to the neural processes underlying hierarchical behavior.

Original languageEnglish (US)
Pages (from-to)370-379
Number of pages10
JournalNeuron
Volume71
Issue number2
DOIs
StatePublished - Jul 28 2011

All Science Journal Classification (ASJC) codes

  • Neuroscience(all)

Fingerprint Dive into the research topics of 'A Neural Signature of Hierarchical Reinforcement Learning'. Together they form a unique fingerprint.

  • Cite this

    Ribas-Fernandes, J. J. F., Solway, A., Diuk, C., McGuire, J. T., Barto, A. G., Niv, Y., & Botvinick, M. M. (2011). A Neural Signature of Hierarchical Reinforcement Learning. Neuron, 71(2), 370-379. https://doi.org/10.1016/j.neuron.2011.05.042