OFFLINE REINFORCEMENT LEARNING WITH DIFFERENTIABLE FUNCTION APPROXIMATION IS PROVABLY EFFICIENT

Offline reinforcement learning, which aims at optimizing sequential decision-making strategies with historical data, has been extensively applied in real-life applications.State-Of-The-Art algorithms usually leverage powerful function approximators (e.g.neural networks) to alleviate the sample complexity hurdle for better empirical performances.Despite the successes, a more systematic understanding of the statistical complexity for function approximation remains lacking.Towards bridging the gap, we take a step by considering offline reinforcement learning with differentiable function class approximation (DFA).This function class naturally incorporates a wide range of models with nonlinear/nonconvex structures.We show offline RL with differentiable function approximation is provably efficient by analyzing the pessimistic fitted Q-learning (PFQL) algorithm, and our results provide the theoretical basis for understanding a variety of practical heuristics that rely on Fitted Q-Iteration style design.In addition, we further improve our guarantee with a tighter instance-dependent characterization.We hope our work could draw interest in studying reinforcement learning with differentiable function approximation beyond the scope of current research.