Implicit bias of gradient descent on linear convolutional networks

Suriya Gunasekar; Jason D. Lee; Nathan Srebro; Daniel Soudry

Implicit bias of gradient descent on linear convolutional networks

Suriya Gunasekar, Jason D. Lee, Nathan Srebro, Daniel Soudry

Research output: Contribution to journal › Conference article › peer-review

Abstract

We show that gradient descent on full width linear convolutional networks of depth L converges to a linear predictor related to the `_2/L bridge penalty in the frequency domain. This is in contrast to fully connected linear networks, where regardless of depth, gradient descent converges to the `₂ maximum margin solution.

Original language	English (US)
Pages (from-to)	9461-9471
Number of pages	11
Journal	Advances in Neural Information Processing Systems
Volume	2018-December
State	Published - 2018
Externally published	Yes
Event	32nd Conference on Neural Information Processing Systems, NeurIPS 2018 - Montreal, Canada Duration: Dec 2 2018 → Dec 8 2018

All Science Journal Classification (ASJC) codes

Computer Networks and Communications
Information Systems
Signal Processing

Cite this

@article{3c7e2a1d4ce0441ba9793c66afee2874,

title = "Implicit bias of gradient descent on linear convolutional networks",

abstract = "We show that gradient descent on full width linear convolutional networks of depth L converges to a linear predictor related to the `2/L bridge penalty in the frequency domain. This is in contrast to fully connected linear networks, where regardless of depth, gradient descent converges to the `2 maximum margin solution.",

author = "Suriya Gunasekar and Lee, \{Jason D.\} and Nathan Srebro and Daniel Soudry",

note = "Publisher Copyright: {\textcopyright} 2018 Curran Associates Inc.All rights reserved.; 32nd Conference on Neural Information Processing Systems, NeurIPS 2018 ; Conference date: 02-12-2018 Through 08-12-2018",

year = "2018",

language = "English (US)",

volume = "2018-December",

pages = "9461--9471",

journal = "Advances in Neural Information Processing Systems",

issn = "1049-5258",

publisher = "Neural information processing systems foundation",

}

TY - JOUR

T1 - Implicit bias of gradient descent on linear convolutional networks

AU - Gunasekar, Suriya

AU - Lee, Jason D.

AU - Srebro, Nathan

AU - Soudry, Daniel

PY - 2018

Y1 - 2018

N2 - We show that gradient descent on full width linear convolutional networks of depth L converges to a linear predictor related to the `2/L bridge penalty in the frequency domain. This is in contrast to fully connected linear networks, where regardless of depth, gradient descent converges to the `2 maximum margin solution.

AB - We show that gradient descent on full width linear convolutional networks of depth L converges to a linear predictor related to the `2/L bridge penalty in the frequency domain. This is in contrast to fully connected linear networks, where regardless of depth, gradient descent converges to the `2 maximum margin solution.

UR - https://www.scopus.com/pages/publications/85057771475

UR - https://www.scopus.com/inward/citedby.url?scp=85057771475&partnerID=8YFLogxK

M3 - Conference article

AN - SCOPUS:85057771475

SN - 1049-5258

VL - 2018-December

SP - 9461

EP - 9471

JO - Advances in Neural Information Processing Systems

JF - Advances in Neural Information Processing Systems

T2 - 32nd Conference on Neural Information Processing Systems, NeurIPS 2018

Y2 - 2 December 2018 through 8 December 2018

ER -

Implicit bias of gradient descent on linear convolutional networks

Abstract

All Science Journal Classification (ASJC) codes

Other files and links

Fingerprint

Cite this