Holography is perhaps the most promising technology to achieve wide field of view compact eye-glasses style near-eye displays. However, the digital hologram computation algorithms are still not perfect and resort to heuristic encoding or iterative methods relying on varying relaxations. In this paper, we deviate from such heuristic solutions to holographic phase retrieval but instead rely on formal optimization that is enabled by complex Wirtinger gradients. We pose the entire hologram computation forward model as a differentiable forward model and formulate a quadratic loss function that is solved via first-order optimization methods. Using this framework, we achieve holographic reconstructions with an order of magnitude improved image quality, both in simulation and on an experimental prototype.