Dopamine and inference about timing

Several investigators have suggested that the primate dopamine system carries an error signal for learning to predict future rewards. These models, based on temporal-difference (TD) learning, explain most phasic responses of primate dopamine neurons in appetitive conditioning; moreover, they suggest a neurophysiological account of animal conditioning behavior. But because existing models are based in the simple formal setting of Markov processes, they are deficient in at least two areas relevant to physiological and behavioral data. They do not provide a realistic account of the partial observability of the state of the world, nor of how the system tracks the timing of events. In this paper, we introduce a version of TD learning grounded in a richer formal model to better address both issues and, consequently, to explain some data that challenge existing models.