In a multi-user system with multiple antennas at the base station, precoding techniques in the downlink broadcast channel allow users to detect their respective data in a non-cooperative manner. Vector Perturbation Precoding (VPP) is a non-linear variant of transmit-side channel inversion that perturbs user data to achieve full diversity order. While promising, finding an optimal perturbation in VPP is known to be an NP-hard problem, demanding heavy computational support at the base station and limiting the feasibility of the approach to small MIMO systems. This work proposes a radically different processing architecture for the downlink VPP problem, one based on Quantum Annealing (QA), to enable the applicability of VPP to large MIMO systems. Our design reduces VPP to a quadratic polynomial form amenable to QA, then refines the problem coefficients to mitigate the adverse effects of QA hardware noise. We evaluate our proposed QA based VPP (QAVP) technique on a real Quantum Annealing device over a variety of design and machine parameter settings. With existing hardware, QAVP can achieve a BER of 10-4 with 100µs compute time, for a 6 × 6 MIMO system using 64 QAM modulation at 32 dB SNR.