A Resource-Rational Account of Human Eye Movements During Immersive Visual Search

The nature of eye movements during visual search has been widely studied in cognitive science. Virtual reality (VR) paradigms are an opportunity to test whether computational models of search can predict naturalistic search behavior. However, existing ideal observer models are constrained by strong assumptions about the structure of the world, rendering them impractical for modeling the complexity of environments which can be studied in VR. To address these limitations, we modeled immersive visual search as a reinforcement learning problem, in which sequential decisions are made over a multidimensional representation of the environment learned by a convolutional neural network. In our formulation, RL agents learned a policy over latent states—effectively solving what is known as a meta–Markov decision process (meta-MDP), where each decision concerns how to allocate attention to information in the environment. Training deep-RL agents on the meta-MDP showed that learned (i.e., optimal) search policies converge to a classic ideal-observer model of search developed for simple (1D) stimuli. We compared the learned resource-rational policy with human gaze data from a visual-search experiment conducted in VR and found qualitative and quantitative alignment between model predictions and human behavior. However, both the model’s simulated performance and its correspondence with human behavior depended strongly on the representational features available to the policy. These results suggest that naturalistic visual search behavior can partially be explained by resource-rational allocation of limited cognitive resources, and the choice of representation influences the degree of alignment between model and human behavior.