An efficient, generalized bellman update for cooperative inverse reinforcement learning

Dhruv Malik, Malayandi Palaniappan, Jaime F. Fisac, Dylan Hadfield-Menell, Stuart Russell, Anca D. Dragan

Research output: Chapter in Book/Report/Conference proceedingConference contribution

1 Scopus citations

Abstract

Our goal is for AI systems to correctly identify and act according to their human user's objec-tives. Cooperative Inverse Reinforcement Learning (CIRL) formalizes this value alignment problem as a two-player game between a human and robot, in which only the human knows the pa-rameters of the reward function: The robot needs to learn them as the interaction unfolds. Previ-ous work showed that CIRL can be solved as a POMDP, but with an action space size exponential in the size of the reward parameter space. In this work, we exploit a specific property of CIRL-the human is a full information agent-to derive an optimality-preserving modification to the standard Bellman update; this reduces the complexity of the problem by an exponential factor and allows us to relax CIRL's assumption of human rationality. We apply this update to a variety of POMDP solvers and find that it enables us to scale CIRL to non-trivial problems, with larger reward parame-ter spaces, and larger action spaces for both robot and human. In solutions to these larger problems, the human exhibits pedagogic behavior, while the robot interprets it as such and attains higher value for the human.

Original languageEnglish (US)
Title of host publication35th International Conference on Machine Learning, ICML 2018
EditorsJennifer Dy, Andreas Krause
PublisherInternational Machine Learning Society (IMLS)
Pages5435-5443
Number of pages9
ISBN (Electronic)9781510867963
StatePublished - 2018
Externally publishedYes
Event35th International Conference on Machine Learning, ICML 2018 - Stockholm, Sweden
Duration: Jul 10 2018Jul 15 2018

Publication series

Name35th International Conference on Machine Learning, ICML 2018
Volume8

Other

Other35th International Conference on Machine Learning, ICML 2018
CountrySweden
CityStockholm
Period7/10/187/15/18

All Science Journal Classification (ASJC) codes

  • Computational Theory and Mathematics
  • Human-Computer Interaction
  • Software

Fingerprint Dive into the research topics of 'An efficient, generalized bellman update for cooperative inverse reinforcement learning'. Together they form a unique fingerprint.

  • Cite this

    Malik, D., Palaniappan, M., Fisac, J. F., Hadfield-Menell, D., Russell, S., & Dragan, A. D. (2018). An efficient, generalized bellman update for cooperative inverse reinforcement learning. In J. Dy, & A. Krause (Eds.), 35th International Conference on Machine Learning, ICML 2018 (pp. 5435-5443). (35th International Conference on Machine Learning, ICML 2018; Vol. 8). International Machine Learning Society (IMLS).