Bias-corrected Q-learning to control max-operator bias in Q-learning

Donghun Lee; Boris Defourny; Warren Buckler Powell

doi:10.1109/ADPRL.2013.6614994

Bias-corrected Q-learning to control max-operator bias in Q-learning

Donghun Lee, Boris Defourny, Warren Buckler Powell

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution

43 Scopus citations

Abstract

We identify a class of stochastic control problems with highly random rewards and high discount factor which induce high levels of statistical error in the estimated action-value function. This produces significant levels of max-operator bias in Q-learning, which can induce the algorithm to diverge for millions of iterations. We present a bias-corrected Q-learning algorithm with asymptotically unbiased resistance against the max-operator bias, and show that the algorithm asymptotically converges to the optimal policy, as Q-learning does. We show experimentally that bias-corrected Q-learning performs well in a domain with highly random rewards where Q-learning and other related algorithms suffer from the max-operator bias.

Original language	English (US)
Title of host publication	Proceedings of the 2013 IEEE Symposium on Adaptive Dynamic Programming and Reinforcement Learning, ADPRL 2013 - 2013 IEEE Symposium Series on Computational Intelligence, SSCI 2013
Pages	93-99
Number of pages	7
DOIs	https://doi.org/10.1109/ADPRL.2013.6614994
State	Published - 2013
Event	2013 4th IEEE Symposium on Adaptive Dynamic Programming and Reinforcement Learning, ADPRL 2013 - Singapore, Singapore Duration: Apr 16 2013 → Apr 19 2013

Publication series

Name	IEEE Symposium on Adaptive Dynamic Programming and Reinforcement Learning, ADPRL
ISSN (Print)	2325-1824
ISSN (Electronic)	2325-1867

Other

Other	2013 4th IEEE Symposium on Adaptive Dynamic Programming and Reinforcement Learning, ADPRL 2013
Country/Territory	Singapore
City	Singapore
Period	4/16/13 → 4/19/13

All Science Journal Classification (ASJC) codes

Computational Theory and Mathematics
Computer Science Applications
Software

Access to Document

10.1109/ADPRL.2013.6614994

Cite this

Lee, D., Defourny, B., & Powell, W. B. (2013). Bias-corrected Q-learning to control max-operator bias in Q-learning. In Proceedings of the 2013 IEEE Symposium on Adaptive Dynamic Programming and Reinforcement Learning, ADPRL 2013 - 2013 IEEE Symposium Series on Computational Intelligence, SSCI 2013 (pp. 93-99). Article 6614994 (IEEE Symposium on Adaptive Dynamic Programming and Reinforcement Learning, ADPRL). https://doi.org/10.1109/ADPRL.2013.6614994

Lee, Donghun ; Defourny, Boris ; Powell, Warren Buckler. / Bias-corrected Q-learning to control max-operator bias in Q-learning. Proceedings of the 2013 IEEE Symposium on Adaptive Dynamic Programming and Reinforcement Learning, ADPRL 2013 - 2013 IEEE Symposium Series on Computational Intelligence, SSCI 2013. 2013. pp. 93-99 (IEEE Symposium on Adaptive Dynamic Programming and Reinforcement Learning, ADPRL).

@inproceedings{71d4325d1a3f49958da82ddb1b9cb40a,

title = "Bias-corrected Q-learning to control max-operator bias in Q-learning",

abstract = "We identify a class of stochastic control problems with highly random rewards and high discount factor which induce high levels of statistical error in the estimated action-value function. This produces significant levels of max-operator bias in Q-learning, which can induce the algorithm to diverge for millions of iterations. We present a bias-corrected Q-learning algorithm with asymptotically unbiased resistance against the max-operator bias, and show that the algorithm asymptotically converges to the optimal policy, as Q-learning does. We show experimentally that bias-corrected Q-learning performs well in a domain with highly random rewards where Q-learning and other related algorithms suffer from the max-operator bias.",

author = "Donghun Lee and Boris Defourny and Powell, {Warren Buckler}",

year = "2013",

doi = "10.1109/ADPRL.2013.6614994",

language = "English (US)",

isbn = "9781467359252",

series = "IEEE Symposium on Adaptive Dynamic Programming and Reinforcement Learning, ADPRL",

pages = "93--99",

booktitle = "Proceedings of the 2013 IEEE Symposium on Adaptive Dynamic Programming and Reinforcement Learning, ADPRL 2013 - 2013 IEEE Symposium Series on Computational Intelligence, SSCI 2013",

note = "2013 4th IEEE Symposium on Adaptive Dynamic Programming and Reinforcement Learning, ADPRL 2013 ; Conference date: 16-04-2013 Through 19-04-2013",

}

Lee, D, Defourny, B & Powell, WB 2013, Bias-corrected Q-learning to control max-operator bias in Q-learning. in Proceedings of the 2013 IEEE Symposium on Adaptive Dynamic Programming and Reinforcement Learning, ADPRL 2013 - 2013 IEEE Symposium Series on Computational Intelligence, SSCI 2013., 6614994, IEEE Symposium on Adaptive Dynamic Programming and Reinforcement Learning, ADPRL, pp. 93-99, 2013 4th IEEE Symposium on Adaptive Dynamic Programming and Reinforcement Learning, ADPRL 2013, Singapore, Singapore, 4/16/13. https://doi.org/10.1109/ADPRL.2013.6614994

Bias-corrected Q-learning to control max-operator bias in Q-learning. / Lee, Donghun; Defourny, Boris; Powell, Warren Buckler.
Proceedings of the 2013 IEEE Symposium on Adaptive Dynamic Programming and Reinforcement Learning, ADPRL 2013 - 2013 IEEE Symposium Series on Computational Intelligence, SSCI 2013. 2013. p. 93-99 6614994 (IEEE Symposium on Adaptive Dynamic Programming and Reinforcement Learning, ADPRL).

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution

TY - GEN

T1 - Bias-corrected Q-learning to control max-operator bias in Q-learning

AU - Lee, Donghun

AU - Defourny, Boris

AU - Powell, Warren Buckler

PY - 2013

Y1 - 2013

N2 - We identify a class of stochastic control problems with highly random rewards and high discount factor which induce high levels of statistical error in the estimated action-value function. This produces significant levels of max-operator bias in Q-learning, which can induce the algorithm to diverge for millions of iterations. We present a bias-corrected Q-learning algorithm with asymptotically unbiased resistance against the max-operator bias, and show that the algorithm asymptotically converges to the optimal policy, as Q-learning does. We show experimentally that bias-corrected Q-learning performs well in a domain with highly random rewards where Q-learning and other related algorithms suffer from the max-operator bias.

AB - We identify a class of stochastic control problems with highly random rewards and high discount factor which induce high levels of statistical error in the estimated action-value function. This produces significant levels of max-operator bias in Q-learning, which can induce the algorithm to diverge for millions of iterations. We present a bias-corrected Q-learning algorithm with asymptotically unbiased resistance against the max-operator bias, and show that the algorithm asymptotically converges to the optimal policy, as Q-learning does. We show experimentally that bias-corrected Q-learning performs well in a domain with highly random rewards where Q-learning and other related algorithms suffer from the max-operator bias.

UR - http://www.scopus.com/inward/record.url?scp=84891539165&partnerID=8YFLogxK

UR - http://www.scopus.com/inward/citedby.url?scp=84891539165&partnerID=8YFLogxK

U2 - 10.1109/ADPRL.2013.6614994

DO - 10.1109/ADPRL.2013.6614994

M3 - Conference contribution

AN - SCOPUS:84891539165

SN - 9781467359252

T3 - IEEE Symposium on Adaptive Dynamic Programming and Reinforcement Learning, ADPRL

SP - 93

EP - 99

BT - Proceedings of the 2013 IEEE Symposium on Adaptive Dynamic Programming and Reinforcement Learning, ADPRL 2013 - 2013 IEEE Symposium Series on Computational Intelligence, SSCI 2013

T2 - 2013 4th IEEE Symposium on Adaptive Dynamic Programming and Reinforcement Learning, ADPRL 2013

Y2 - 16 April 2013 through 19 April 2013

ER -

Lee D, Defourny B, Powell WB. Bias-corrected Q-learning to control max-operator bias in Q-learning. In Proceedings of the 2013 IEEE Symposium on Adaptive Dynamic Programming and Reinforcement Learning, ADPRL 2013 - 2013 IEEE Symposium Series on Computational Intelligence, SSCI 2013. 2013. p. 93-99. 6614994. (IEEE Symposium on Adaptive Dynamic Programming and Reinforcement Learning, ADPRL). doi: 10.1109/ADPRL.2013.6614994

Bias-corrected Q-learning to control max-operator bias in Q-learning

Abstract

Publication series

Other

All Science Journal Classification (ASJC) codes

Access to Document

Other files and links

Fingerprint

Cite this