Kolmogorov width decay and poor approximators in machine learning: shallow neural networks, random feature models and neural tangent kernels

E. Weinan; Stephan Wojtowytsch

doi:10.1007/s40687-020-00233-4

Kolmogorov width decay and poor approximators in machine learning: shallow neural networks, random feature models and neural tangent kernels

E. Weinan, Stephan Wojtowytsch

Mathematics

Research output: Contribution to journal › Article › peer-review

9 Scopus citations

Abstract

We establish a scale separation of Kolmogorov width type between subspaces of a given Banach space under the condition that a sequence of linear maps converges much faster on one of the subspaces. The general technique is then applied to show that reproducing kernel Hilbert spaces are poor L²-approximators for the class of two-layer neural networks in high dimension, and that multi-layer networks with small path norm are poor approximators for certain Lipschitz functions, also in the L²-topology.

Original language	English (US)
Article number	5
Journal	Research in Mathematical Sciences
Volume	8
Issue number	1
DOIs	https://doi.org/10.1007/s40687-020-00233-4
State	Published - Mar 2021

All Science Journal Classification (ASJC) codes

Theoretical Computer Science
Mathematics (miscellaneous)
Computational Mathematics
Applied Mathematics

Keywords

Approximation theory
Barron space
Curse of dimensionality
Kolmogorov width
Multi-layer network
Neural tangent kernel
Population risk
Random feature model
Reproducing kernel Hilbert space
Two-layer network

Access to Document

10.1007/s40687-020-00233-4

Cite this

@article{0e8c566e0582479786e098040728abd6,

title = "Kolmogorov width decay and poor approximators in machine learning: shallow neural networks, random feature models and neural tangent kernels",

abstract = "We establish a scale separation of Kolmogorov width type between subspaces of a given Banach space under the condition that a sequence of linear maps converges much faster on one of the subspaces. The general technique is then applied to show that reproducing kernel Hilbert spaces are poor L2-approximators for the class of two-layer neural networks in high dimension, and that multi-layer networks with small path norm are poor approximators for certain Lipschitz functions, also in the L2-topology.",

keywords = "Approximation theory, Barron space, Curse of dimensionality, Kolmogorov width, Multi-layer network, Neural tangent kernel, Population risk, Random feature model, Reproducing kernel Hilbert space, Two-layer network",

author = "E. Weinan and Stephan Wojtowytsch",

note = "Publisher Copyright: {\textcopyright} 2021, Springer Nature Switzerland AG.",

year = "2021",

month = mar,

doi = "10.1007/s40687-020-00233-4",

language = "English (US)",

volume = "8",

journal = "Research in Mathematical Sciences",

issn = "2522-0144",

publisher = "SpringerOpen",

number = "1",

}

TY - JOUR

T1 - Kolmogorov width decay and poor approximators in machine learning

T2 - shallow neural networks, random feature models and neural tangent kernels

AU - Weinan, E.

AU - Wojtowytsch, Stephan

PY - 2021/3

Y1 - 2021/3

N2 - We establish a scale separation of Kolmogorov width type between subspaces of a given Banach space under the condition that a sequence of linear maps converges much faster on one of the subspaces. The general technique is then applied to show that reproducing kernel Hilbert spaces are poor L2-approximators for the class of two-layer neural networks in high dimension, and that multi-layer networks with small path norm are poor approximators for certain Lipschitz functions, also in the L2-topology.

AB - We establish a scale separation of Kolmogorov width type between subspaces of a given Banach space under the condition that a sequence of linear maps converges much faster on one of the subspaces. The general technique is then applied to show that reproducing kernel Hilbert spaces are poor L2-approximators for the class of two-layer neural networks in high dimension, and that multi-layer networks with small path norm are poor approximators for certain Lipschitz functions, also in the L2-topology.

KW - Approximation theory

KW - Barron space

KW - Curse of dimensionality

KW - Kolmogorov width

KW - Multi-layer network

KW - Neural tangent kernel

KW - Population risk

KW - Random feature model

KW - Reproducing kernel Hilbert space

KW - Two-layer network

UR - http://www.scopus.com/inward/record.url?scp=85098730662&partnerID=8YFLogxK

UR - http://www.scopus.com/inward/citedby.url?scp=85098730662&partnerID=8YFLogxK

U2 - 10.1007/s40687-020-00233-4

DO - 10.1007/s40687-020-00233-4

M3 - Article

AN - SCOPUS:85098730662

SN - 2522-0144

VL - 8

JO - Research in Mathematical Sciences

JF - Research in Mathematical Sciences

IS - 1

M1 - 5

ER -

Kolmogorov width decay and poor approximators in machine learning: shallow neural networks, random feature models and neural tangent kernels

Abstract

All Science Journal Classification (ASJC) codes

Keywords

Access to Document

Other files and links

Fingerprint

Cite this