No spurious local minima in a two hidden unit Relu network

Jiajun Luo; Chenwei Wu; Jason D. Lee

No spurious local minima in a two hidden unit Relu network

Jiajun Luo, Chenwei Wu, Jason D. Lee

Research output: Contribution to conference › Paper › peer-review

Abstract

Deep learning models can be efficiently optimized via stochastic gradient descent, but there is little theoretical evidence to support this. A key question in optimization is to understand when the optimization landscape of a neural network is amenable to gradient-based optimization. We focus on a simple neural network two-layer ReLU network with two hidden units, and show that all local minimizers are global. This combined with recent work of Lee et al. (2017); Lee et al. (2016) show that gradient descent converges to the global minimizer.

Original language	English (US)
State	Published - Jan 1 2018
Externally published	Yes
Event	6th International Conference on Learning Representations, ICLR 2018 - Vancouver, Canada Duration: Apr 30 2018 → May 3 2018

Conference

Conference	6th International Conference on Learning Representations, ICLR 2018
Country/Territory	Canada
City	Vancouver
Period	4/30/18 → 5/3/18

All Science Journal Classification (ASJC) codes

Education
Computer Science Applications
Linguistics and Language
Language and Linguistics

Cite this

@conference{c5ef18cff0a346f0a20cb1cf33ccb2b3,

title = "No spurious local minima in a two hidden unit Relu network",

abstract = "Deep learning models can be efficiently optimized via stochastic gradient descent, but there is little theoretical evidence to support this. A key question in optimization is to understand when the optimization landscape of a neural network is amenable to gradient-based optimization. We focus on a simple neural network two-layer ReLU network with two hidden units, and show that all local minimizers are global. This combined with recent work of Lee et al. (2017); Lee et al. (2016) show that gradient descent converges to the global minimizer.",

author = "Jiajun Luo and Chenwei Wu and Lee, \{Jason D.\}",

year = "2018",

month = jan,

day = "1",

language = "English (US)",

note = "6th International Conference on Learning Representations, ICLR 2018 ; Conference date: 30-04-2018 Through 03-05-2018",

}

TY - CONF

T1 - No spurious local minima in a two hidden unit Relu network

AU - Luo, Jiajun

AU - Wu, Chenwei

AU - Lee, Jason D.

PY - 2018/1/1

Y1 - 2018/1/1

N2 - Deep learning models can be efficiently optimized via stochastic gradient descent, but there is little theoretical evidence to support this. A key question in optimization is to understand when the optimization landscape of a neural network is amenable to gradient-based optimization. We focus on a simple neural network two-layer ReLU network with two hidden units, and show that all local minimizers are global. This combined with recent work of Lee et al. (2017); Lee et al. (2016) show that gradient descent converges to the global minimizer.

AB - Deep learning models can be efficiently optimized via stochastic gradient descent, but there is little theoretical evidence to support this. A key question in optimization is to understand when the optimization landscape of a neural network is amenable to gradient-based optimization. We focus on a simple neural network two-layer ReLU network with two hidden units, and show that all local minimizers are global. This combined with recent work of Lee et al. (2017); Lee et al. (2016) show that gradient descent converges to the global minimizer.

UR - https://www.scopus.com/pages/publications/85070979362

UR - http://www.scopus.com/inward/citedby.url?scp=85070979362&partnerID=8YFLogxK

M3 - Paper

T2 - 6th International Conference on Learning Representations, ICLR 2018

Y2 - 30 April 2018 through 3 May 2018

ER -

No spurious local minima in a two hidden unit Relu network

Abstract

Conference

All Science Journal Classification (ASJC) codes

Other files and links

Fingerprint

Cite this