TY - JOUR
T1 - A generalized algorithm for multi-objective reinforcement learning and policy adaptation
AU - Yang, Runzhe
AU - Sun, Xingyuan
AU - Narasimhan, Karthik
N1 - Funding Information:
The authors would like to thank Yexiang Xue at Purdue University, Carla Gomes at Cornell University for helpful discussions on multi-objective optimization, Lu Chen, Kai Yu at Shanghai Jiao Tong University for discussing dialogue applications, Haoran Cai at MIT for helping running a part of synthetic experiments, and anonymous reviewers for constructive suggestions.
Publisher Copyright:
© 2019 Neural information processing systems foundation. All rights reserved.
PY - 2019
Y1 - 2019
N2 - We introduce a new algorithm for multi-objective reinforcement learning (MORL) with linear preferences, with the goal of enabling few-shot adaptation to new tasks. In MORL, the aim is to learn policies over multiple competing objectives whose relative importance (preferences) is unknown to the agent. While this alleviates dependence on scalar reward design, the expected return of a policy can change significantly with varying preferences, making it challenging to learn a single model to produce optimal policies under different preference conditions. We propose a generalized version of the Bellman equation to learn a single parametric representation for optimal policies over the space of all possible preferences. After an initial learning phase, our agent can execute the optimal policy under any given preference, or automatically infer an underlying preference with very few samples. Experiments across four different domains demonstrate the effectiveness of our approach.
AB - We introduce a new algorithm for multi-objective reinforcement learning (MORL) with linear preferences, with the goal of enabling few-shot adaptation to new tasks. In MORL, the aim is to learn policies over multiple competing objectives whose relative importance (preferences) is unknown to the agent. While this alleviates dependence on scalar reward design, the expected return of a policy can change significantly with varying preferences, making it challenging to learn a single model to produce optimal policies under different preference conditions. We propose a generalized version of the Bellman equation to learn a single parametric representation for optimal policies over the space of all possible preferences. After an initial learning phase, our agent can execute the optimal policy under any given preference, or automatically infer an underlying preference with very few samples. Experiments across four different domains demonstrate the effectiveness of our approach.
UR - http://www.scopus.com/inward/record.url?scp=85084150576&partnerID=8YFLogxK
UR - http://www.scopus.com/inward/citedby.url?scp=85084150576&partnerID=8YFLogxK
M3 - Conference article
AN - SCOPUS:85084150576
SN - 1049-5258
VL - 32
JO - Advances in Neural Information Processing Systems
JF - Advances in Neural Information Processing Systems
T2 - 33rd Annual Conference on Neural Information Processing Systems, NeurIPS 2019
Y2 - 8 December 2019 through 14 December 2019
ER -