Stochastic modified equations and adaptive stochastic gradient algorithms

Qianxiao Li; Cheng Tai; E. Weinan

Stochastic modified equations and adaptive stochastic gradient algorithms

Qianxiao Li, Cheng Tai, E. Weinan

Mathematics

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution

20 Scopus citations

Abstract

We develop the method of stochastic modified equations (SME), in which stochastic gradient algorithms are approximated in the weak sense by continuous-time stochastic differential equations. We exploit the continuous formulation together with optimal control theory to derive novel adaptive hyper-parameter adjustment policies. Our algorithms have competitive performance with the added benefit of being robust to varying models and datasets. This provides a general methodology for the analysis and design of stochastic gradient algorithms.

Original language	English (US)
Title of host publication	34th International Conference on Machine Learning, ICML 2017
Publisher	International Machine Learning Society (IMLS)
Pages	3306-3340
Number of pages	35
ISBN (Electronic)	9781510855144
State	Published - 2017
Event	34th International Conference on Machine Learning, ICML 2017 - Sydney, Australia Duration: Aug 6 2017 → Aug 11 2017

Publication series

Name	34th International Conference on Machine Learning, ICML 2017
Volume	5

Other

Other	34th International Conference on Machine Learning, ICML 2017
Country/Territory	Australia
City	Sydney
Period	8/6/17 → 8/11/17

All Science Journal Classification (ASJC) codes

Computational Theory and Mathematics
Human-Computer Interaction
Software

Cite this

@inproceedings{ab021c6b39774e35b99e59a1ded55c5c,

title = "Stochastic modified equations and adaptive stochastic gradient algorithms",

abstract = "We develop the method of stochastic modified equations (SME), in which stochastic gradient algorithms are approximated in the weak sense by continuous-time stochastic differential equations. We exploit the continuous formulation together with optimal control theory to derive novel adaptive hyper-parameter adjustment policies. Our algorithms have competitive performance with the added benefit of being robust to varying models and datasets. This provides a general methodology for the analysis and design of stochastic gradient algorithms.",

author = "Qianxiao Li and Cheng Tai and E. Weinan",

note = "Publisher Copyright: {\textcopyright} 2017 by the author(s).; 34th International Conference on Machine Learning, ICML 2017 ; Conference date: 06-08-2017 Through 11-08-2017",

year = "2017",

language = "English (US)",

series = "34th International Conference on Machine Learning, ICML 2017",

publisher = "International Machine Learning Society (IMLS)",

pages = "3306--3340",

booktitle = "34th International Conference on Machine Learning, ICML 2017",

}

Li, Q, Tai, C & Weinan, E 2017, Stochastic modified equations and adaptive stochastic gradient algorithms. in 34th International Conference on Machine Learning, ICML 2017. 34th International Conference on Machine Learning, ICML 2017, vol. 5, International Machine Learning Society (IMLS), pp. 3306-3340, 34th International Conference on Machine Learning, ICML 2017, Sydney, Australia, 8/6/17.

Stochastic modified equations and adaptive stochastic gradient algorithms. / Li, Qianxiao; Tai, Cheng; Weinan, E.
34th International Conference on Machine Learning, ICML 2017. International Machine Learning Society (IMLS), 2017. p. 3306-3340 (34th International Conference on Machine Learning, ICML 2017; Vol. 5).

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution

TY - GEN

T1 - Stochastic modified equations and adaptive stochastic gradient algorithms

AU - Li, Qianxiao

AU - Tai, Cheng

AU - Weinan, E.

PY - 2017

Y1 - 2017

N2 - We develop the method of stochastic modified equations (SME), in which stochastic gradient algorithms are approximated in the weak sense by continuous-time stochastic differential equations. We exploit the continuous formulation together with optimal control theory to derive novel adaptive hyper-parameter adjustment policies. Our algorithms have competitive performance with the added benefit of being robust to varying models and datasets. This provides a general methodology for the analysis and design of stochastic gradient algorithms.

AB - We develop the method of stochastic modified equations (SME), in which stochastic gradient algorithms are approximated in the weak sense by continuous-time stochastic differential equations. We exploit the continuous formulation together with optimal control theory to derive novel adaptive hyper-parameter adjustment policies. Our algorithms have competitive performance with the added benefit of being robust to varying models and datasets. This provides a general methodology for the analysis and design of stochastic gradient algorithms.

UR - http://www.scopus.com/inward/record.url?scp=85048468735&partnerID=8YFLogxK

UR - http://www.scopus.com/inward/citedby.url?scp=85048468735&partnerID=8YFLogxK

M3 - Conference contribution

AN - SCOPUS:85048468735

T3 - 34th International Conference on Machine Learning, ICML 2017

SP - 3306

EP - 3340

BT - 34th International Conference on Machine Learning, ICML 2017

PB - International Machine Learning Society (IMLS)

T2 - 34th International Conference on Machine Learning, ICML 2017

Y2 - 6 August 2017 through 11 August 2017

ER -

Stochastic modified equations and adaptive stochastic gradient algorithms

Abstract

Publication series

Other

All Science Journal Classification (ASJC) codes

Other files and links

Fingerprint

Cite this