Abstract
Significant theoretical work has established that in specific regimes, neural networks trained by gradient descent behave like kernel methods. However, in practice, it is known that neural networks strongly outperform their associated kernels. In this work, we explain this gap by demonstrating that there is a large class of functions which cannot be efficiently learned by kernel methods but can be easily learned with gradient descent on a two layer neural network outside the kernel regime by learning representations that are relevant to the target task. We also demonstrate that these representations allow for efficient transfer learning, which is impossible in the kernel regime. Specifically, we consider the problem of learning polynomials which depend on only a few relevant directions, i.e. of the form f?(x) = g(Ux) where U : Rd → Rr with d ≫ r. When the degree of f? is p, it is known that n dp samples are necessary to learn f? in the kernel regime. Our primary result is that gradient descent learns a representation of the data which depends only on the directions relevant to f?. This results in an improved sample complexity of n d2 and enables transfer learning with sample complexity independent of d.
Original language | English (US) |
---|---|
Pages (from-to) | 5413-5452 |
Number of pages | 40 |
Journal | Proceedings of Machine Learning Research |
Volume | 178 |
State | Published - 2022 |
Event | 35th Conference on Learning Theory, COLT 2022 - London, United Kingdom Duration: Jul 2 2022 → Jul 5 2022 |
All Science Journal Classification (ASJC) codes
- Artificial Intelligence
- Software
- Control and Systems Engineering
- Statistics and Probability
Keywords
- gradient descent
- kernel
- neural network
- representation learning
- transfer learning