We study stochastic games with a large number of players, where players are coupled via their payoff functions. A standard solution concept for such games is Markov perfect equilibrium (MPE). It is well known that the computation of MPE suffers from the "curse of dimensionality." Recently an approximate solution concept called "oblivious equilibrium" (OE) was developed by Weintraub et. al, where each player reacts to only the average behavior of other players. In this work, we characterize a set of games in which OE approximates MPE. Specifically, we show that if system dynamics and payoff functions are concave in state and action and have decreasing differences in state and action, then an oblivious equilibrium of such a game approximates MPE. These exogenous conditions on model primitives allow us to characterize a set of games where OE can be used as an approximate solution concept.