DIFFERENTIALLY PRIVATE IMAGE CLASSIFICATION BY LEARNING PRIORS FROM RANDOM PROCESSES

In privacy-preserving machine learning, differentially private stochastic gradient descent (DP-SGD) performs worse than SGD due to per-sample gradient clipping and noise addition. A recent focus in private learning research is improving the performance of DP-SGD on private data by incorporating priors that are learned from real-world public data. In this work, we explore how we can improve the privacy-utility trade-off of DP-SGD by learning priors from images generated by random processes and transferring these priors to private data. We propose DP-RandP, a three-phase approach, to combine the benefits of linear probing and full fine-tuning for the synthetic prior. We attain new state-of-the-art accuracy when training from scratch on CIFAR10, CIFAR100, MedMNIST, Camelyon17 and ImageNet for a range of privacy budgets ε ∈ [1, 10]. In particular, we improve the previous best reported accuracy on CIFAR10 from 60.6% to 72.3% for ε = 1.