Having multiple selves helps learning agents explore and adapt in complex changing worlds

Zack Dulberg, Rachit Dubey, Isabel M. Berwian, Jonathan D. Cohen

Research output: Contribution to journalArticlepeer-review

1 Scopus citations

Abstract

Satisfying a variety of conflicting needs in a changing environment is a fundamental challenge for any adaptive agent. Here, we show that designing an agent in a modular fashion as a collection of subagents, each dedicated to a separate need, powerfully enhanced the agent’s capacity to satisfy its overall needs. We used the formalism of deep reinforcement learning to investigate a biologically relevant multiobjective task: continually maintaining homeostasis of a set of physiologic variables. We then conducted simulations in a variety of environments and compared how modular agents performed relative to standard monolithic agents (i.e., agents that aimed to satisfy all needs in an integrated manner using a single aggregate measure of success). Simulations revealed that modular agents a) exhibited a form of exploration that was intrinsic and emergent rather than extrinsically imposed; b) were robust to changes in nonstationary environments, and c) scaled gracefully in their ability to maintain homeostasis as the number of conflicting objectives increased. Supporting analysis suggested that the robustness to changing environments and increasing numbers of needs were due to intrinsic exploration and efficiency of representation afforded by the modular architecture. These results suggest that the normative principles by which agents have adapted to complex changing environments may also explain why humans have long been described as consisting of “multiple selves.”

Original languageEnglish (US)
Article numbere2221180120
JournalProceedings of the National Academy of Sciences of the United States of America
Volume120
Issue number28
DOIs
StatePublished - Jul 11 2023

All Science Journal Classification (ASJC) codes

  • General

Keywords

  • conflict
  • exploration
  • modularity
  • multiobjective decision-making
  • reinforcement learning

Fingerprint

Dive into the research topics of 'Having multiple selves helps learning agents explore and adapt in complex changing worlds'. Together they form a unique fingerprint.

Cite this