Model-based adaptive critic designs

This chapter provides an overview of model-based adaptive critic designs including background, general algorithms, implementations, and comparisons. The authors begin by introducing the mathematical background of model-reference adaptive critic designs. Various ADP designs such as Heuristic Dynamic Programming (HDP), Dual HDP (DHP), Globalized DHP (GDHP), and Action-Dependent (AD) designs are examined from both a mathematical and implementation standpoint and put into perspective. Pseudocode is provided for many aspects of the algorithms. The chapter concludes with applications and examples. For another overview perspective that focuses more on implementation issues read Chapter 4: Guidance in the Use of Adaptive Critics for Control. Chapter 15 contains a comparison of DHP with back-propagation through time, building a common framework for comparing these methods.