Adding one neuron can eliminate all bad local minima

Shiyu Liang; Ruoyu Sun; Jason D. Lee; R. Srikant

Adding one neuron can eliminate all bad local minima

Shiyu Liang, Ruoyu Sun, Jason D. Lee, R. Srikant

Research output: Contribution to journal › Conference article › peer-review

Abstract

One of the main difficulties in analyzing neural networks is the non-convexity of the loss function which may have many bad local minima. In this paper, we study the landscape of neural networks for binary classification tasks. Under mild assumptions, we prove that after adding one special neuron with a skip connection to the output, or one special neuron per layer, every local minimum is a global minimum.

Original language	English (US)
Pages (from-to)	4350-4360
Number of pages	11
Journal	Advances in Neural Information Processing Systems
Volume	2018-December
State	Published - 2018
Externally published	Yes
Event	32nd Conference on Neural Information Processing Systems, NeurIPS 2018 - Montreal, Canada Duration: Dec 2 2018 → Dec 8 2018

All Science Journal Classification (ASJC) codes

Computer Networks and Communications
Information Systems
Signal Processing

Cite this

@article{d32491c9ebc04f61b08bd20e03afe9c6,

title = "Adding one neuron can eliminate all bad local minima",

abstract = "One of the main difficulties in analyzing neural networks is the non-convexity of the loss function which may have many bad local minima. In this paper, we study the landscape of neural networks for binary classification tasks. Under mild assumptions, we prove that after adding one special neuron with a skip connection to the output, or one special neuron per layer, every local minimum is a global minimum.",

author = "Shiyu Liang and Ruoyu Sun and Lee, {Jason D.} and R. Srikant",

note = "Publisher Copyright: {\textcopyright} 2018 Curran Associates Inc..All rights reserved.; 32nd Conference on Neural Information Processing Systems, NeurIPS 2018 ; Conference date: 02-12-2018 Through 08-12-2018",

year = "2018",

language = "English (US)",

volume = "2018-December",

pages = "4350--4360",

journal = "Advances in Neural Information Processing Systems",

issn = "1049-5258",

}

TY - JOUR

T1 - Adding one neuron can eliminate all bad local minima

AU - Liang, Shiyu

AU - Sun, Ruoyu

AU - Lee, Jason D.

AU - Srikant, R.

PY - 2018

Y1 - 2018

N2 - One of the main difficulties in analyzing neural networks is the non-convexity of the loss function which may have many bad local minima. In this paper, we study the landscape of neural networks for binary classification tasks. Under mild assumptions, we prove that after adding one special neuron with a skip connection to the output, or one special neuron per layer, every local minimum is a global minimum.

AB - One of the main difficulties in analyzing neural networks is the non-convexity of the loss function which may have many bad local minima. In this paper, we study the landscape of neural networks for binary classification tasks. Under mild assumptions, we prove that after adding one special neuron with a skip connection to the output, or one special neuron per layer, every local minimum is a global minimum.

UR - http://www.scopus.com/inward/record.url?scp=85064842273&partnerID=8YFLogxK

UR - http://www.scopus.com/inward/citedby.url?scp=85064842273&partnerID=8YFLogxK

M3 - Conference article

AN - SCOPUS:85064842273

SN - 1049-5258

VL - 2018-December

SP - 4350

EP - 4360

JO - Advances in Neural Information Processing Systems

JF - Advances in Neural Information Processing Systems

T2 - 32nd Conference on Neural Information Processing Systems, NeurIPS 2018

Y2 - 2 December 2018 through 8 December 2018

ER -

Adding one neuron can eliminate all bad local minima

Abstract

All Science Journal Classification (ASJC) codes

Other files and links

Fingerprint

Cite this