Approximation by combinations of ReLU and squared ReLU ridge functions with ℓ1 and ℓ0 controls

Jason M. Klusowski, Andrew R. Barron

Research output: Contribution to journalArticlepeer-review

11 Scopus citations

Abstract

We establish L and L2 error bounds for functions of many variables that are approximated by linear combinations of rectified linear unit (ReLU) and squared ReLU ridge functions with 1 and 0 controls on their inner and outer parameters. With the squared ReLU ridge function, we show that the L2 approximation error is inversely proportional to the inner layer 0 sparsity and it need only be sublinear in the outer layer ℓ0 sparsity. Our constructions are obtained using a variant of the Maurey-Jones-Barron probabilistic method, which can be interpreted as either stratified sampling with proportionate allocation or two-stage cluster sampling. We also provide companion error lower bounds that reveal near optimality of our constructions. Despite the sparsity assumptions, we showcase the richness and flexibility of these ridge combinations by defining a large family of functions, in terms of certain spectral conditions, that are particularly well approximated by them.

Original languageEnglish (US)
Article number8485650
Pages (from-to)7649-7656
Number of pages8
JournalIEEE Transactions on Information Theory
Volume64
Issue number12
DOIs
StatePublished - Dec 2018
Externally publishedYes

All Science Journal Classification (ASJC) codes

  • Information Systems
  • Computer Science Applications
  • Library and Information Sciences

Keywords

  • Ridge combinations
  • approximation error
  • rectified linear unit
  • sparse models
  • spline
  • stratified sampling

Fingerprint

Dive into the research topics of 'Approximation by combinations of ReLU and squared ReLU ridge functions with ℓ<sup>1</sup> and ℓ<sup>0</sup> controls'. Together they form a unique fingerprint.

Cite this