LoRA Training in the NTK Regime has No Spurious Local Minima

Low-rank adaptation (LoRA) has become the standard approach for parameter-efficient fine-tuning of large language models (LLM), but our theoretical understanding of LoRA has been limited. In this work, we theoretically analyze LoRA fine-tuning in the neural tangent kernel (NTK) regime with N data points, showing: (i) full finetuning (without LoRA) admits a low-rank solution of rank r ≲ √N; (ii) using LoRA with rank r ≳ √N eliminates spurious local minima, allowing (stochastic) gradient descent to find the low-rank solutions; (iii) the low-rank solution found using LoRA generalizes well.