Neural Networks can Learn Representations with Gradient Descent

Alex Damian, Jason D. Lee, Mahdi Soltanolkotabi

Research output: Contribution to journalConference articlepeer-review

7 Scopus citations


Significant theoretical work has established that in specific regimes, neural networks trained by gradient descent behave like kernel methods. However, in practice, it is known that neural networks strongly outperform their associated kernels. In this work, we explain this gap by demonstrating that there is a large class of functions which cannot be efficiently learned by kernel methods but can be easily learned with gradient descent on a two layer neural network outside the kernel regime by learning representations that are relevant to the target task. We also demonstrate that these representations allow for efficient transfer learning, which is impossible in the kernel regime. Specifically, we consider the problem of learning polynomials which depend on only a few relevant directions, i.e. of the form f?(x) = g(Ux) where U : Rd → Rr with d ≫ r. When the degree of f? is p, it is known that n dp samples are necessary to learn f? in the kernel regime. Our primary result is that gradient descent learns a representation of the data which depends only on the directions relevant to f?. This results in an improved sample complexity of n d2 and enables transfer learning with sample complexity independent of d.

Original languageEnglish (US)
Pages (from-to)5413-5452
Number of pages40
JournalProceedings of Machine Learning Research
StatePublished - 2022
Event35th Conference on Learning Theory, COLT 2022 - London, United Kingdom
Duration: Jul 2 2022Jul 5 2022

All Science Journal Classification (ASJC) codes

  • Artificial Intelligence
  • Software
  • Control and Systems Engineering
  • Statistics and Probability


  • gradient descent
  • kernel
  • neural network
  • representation learning
  • transfer learning


Dive into the research topics of 'Neural Networks can Learn Representations with Gradient Descent'. Together they form a unique fingerprint.

Cite this