Characterizing implicit bias in terms of optimization geometry

Suriya Gunasekar; Jason Lee; Daniel Soudry; Nathan Srebro

Characterizing implicit bias in terms of optimization geometry

Suriya Gunasekar, Jason Lee, Daniel Soudry, Nathan Srebro

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution

Abstract

We study the implicit bias of generic optimization methods, including mirror descent, natural gradient descent, and steepest descent with respect to different potentials and norms, when optimizing under determined linear regression or separable linear classification problems. We explore the question of whether the specific global minimum (among the many possible global minima) reached by optimization can be characterized in terms of the potential or norm of the optimization geometry, and independently of hypcrparameter choices such as step size and momentum.

Original language	English (US)
Title of host publication	35th International Conference on Machine Learning, ICML 2018
Editors	Jennifer Dy, Andreas Krause
Publisher	International Machine Learning Society (IMLS)
Pages	2932-2955
Number of pages	24
ISBN (Electronic)	9781510867963
State	Published - 2018
Externally published	Yes
Event	35th International Conference on Machine Learning, ICML 2018 - Stockholm, Sweden Duration: Jul 10 2018 → Jul 15 2018

Publication series

Name	35th International Conference on Machine Learning, ICML 2018
Volume	4

Other

Other	35th International Conference on Machine Learning, ICML 2018
Country/Territory	Sweden
City	Stockholm
Period	7/10/18 → 7/15/18

All Science Journal Classification (ASJC) codes

Computational Theory and Mathematics
Human-Computer Interaction
Software

Cite this

@inproceedings{6dd4befff4fe4cdc974fe7b828b2bbe0,

title = "Characterizing implicit bias in terms of optimization geometry",

abstract = "We study the implicit bias of generic optimization methods, including mirror descent, natural gradient descent, and steepest descent with respect to different potentials and norms, when optimizing under determined linear regression or separable linear classification problems. We explore the question of whether the specific global minimum (among the many possible global minima) reached by optimization can be characterized in terms of the potential or norm of the optimization geometry, and independently of hypcrparameter choices such as step size and momentum.",

author = "Suriya Gunasekar and Jason Lee and Daniel Soudry and Nathan Srebro",

note = "Publisher Copyright: {\textcopyright} 2018 35th International Conference on Machine Learning, ICML 2018. All rights reserved.; 35th International Conference on Machine Learning, ICML 2018 ; Conference date: 10-07-2018 Through 15-07-2018",

year = "2018",

language = "English (US)",

series = "35th International Conference on Machine Learning, ICML 2018",

publisher = "International Machine Learning Society (IMLS)",

pages = "2932--2955",

editor = "Jennifer Dy and Andreas Krause",

booktitle = "35th International Conference on Machine Learning, ICML 2018",

}

Gunasekar, S, Lee, J, Soudry, D & Srebro, N 2018, Characterizing implicit bias in terms of optimization geometry. in J Dy & A Krause (eds), 35th International Conference on Machine Learning, ICML 2018. 35th International Conference on Machine Learning, ICML 2018, vol. 4, International Machine Learning Society (IMLS), pp. 2932-2955, 35th International Conference on Machine Learning, ICML 2018, Stockholm, Sweden, 7/10/18.

Characterizing implicit bias in terms of optimization geometry. / Gunasekar, Suriya; Lee, Jason; Soudry, Daniel et al.
35th International Conference on Machine Learning, ICML 2018. ed. / Jennifer Dy; Andreas Krause. International Machine Learning Society (IMLS), 2018. p. 2932-2955 (35th International Conference on Machine Learning, ICML 2018; Vol. 4).

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution

TY - GEN

T1 - Characterizing implicit bias in terms of optimization geometry

AU - Gunasekar, Suriya

AU - Lee, Jason

AU - Soudry, Daniel

AU - Srebro, Nathan

PY - 2018

Y1 - 2018

N2 - We study the implicit bias of generic optimization methods, including mirror descent, natural gradient descent, and steepest descent with respect to different potentials and norms, when optimizing under determined linear regression or separable linear classification problems. We explore the question of whether the specific global minimum (among the many possible global minima) reached by optimization can be characterized in terms of the potential or norm of the optimization geometry, and independently of hypcrparameter choices such as step size and momentum.

AB - We study the implicit bias of generic optimization methods, including mirror descent, natural gradient descent, and steepest descent with respect to different potentials and norms, when optimizing under determined linear regression or separable linear classification problems. We explore the question of whether the specific global minimum (among the many possible global minima) reached by optimization can be characterized in terms of the potential or norm of the optimization geometry, and independently of hypcrparameter choices such as step size and momentum.

UR - http://www.scopus.com/inward/record.url?scp=85057321382&partnerID=8YFLogxK

UR - http://www.scopus.com/inward/citedby.url?scp=85057321382&partnerID=8YFLogxK

M3 - Conference contribution

AN - SCOPUS:85057321382

T3 - 35th International Conference on Machine Learning, ICML 2018

SP - 2932

EP - 2955

BT - 35th International Conference on Machine Learning, ICML 2018

A2 - Dy, Jennifer

A2 - Krause, Andreas

PB - International Machine Learning Society (IMLS)

T2 - 35th International Conference on Machine Learning, ICML 2018

Y2 - 10 July 2018 through 15 July 2018

ER -

Characterizing implicit bias in terms of optimization geometry

Abstract

Publication series

Other

All Science Journal Classification (ASJC) codes

Other files and links

Fingerprint

Cite this