TY - GEN
T1 - Stochastic modified equations and adaptive stochastic gradient algorithms
AU - Li, Qianxiao
AU - Tai, Cheng
AU - Weinan, E.
N1 - Funding Information:
We would like to thank the anonymous reviewers for their constructive comments. We are also grateful for the many discussions with Dr Sixin Zhang. This work is supported in part by Major Program of NNSFC under grant 91130005, DOE DE-SC0009248, and ONR N00014-13-1-0338.
Publisher Copyright:
© 2017 by the author(s).
PY - 2017
Y1 - 2017
N2 - We develop the method of stochastic modified equations (SME), in which stochastic gradient algorithms are approximated in the weak sense by continuous-time stochastic differential equations. We exploit the continuous formulation together with optimal control theory to derive novel adaptive hyper-parameter adjustment policies. Our algorithms have competitive performance with the added benefit of being robust to varying models and datasets. This provides a general methodology for the analysis and design of stochastic gradient algorithms.
AB - We develop the method of stochastic modified equations (SME), in which stochastic gradient algorithms are approximated in the weak sense by continuous-time stochastic differential equations. We exploit the continuous formulation together with optimal control theory to derive novel adaptive hyper-parameter adjustment policies. Our algorithms have competitive performance with the added benefit of being robust to varying models and datasets. This provides a general methodology for the analysis and design of stochastic gradient algorithms.
UR - http://www.scopus.com/inward/record.url?scp=85048468735&partnerID=8YFLogxK
UR - http://www.scopus.com/inward/citedby.url?scp=85048468735&partnerID=8YFLogxK
M3 - Conference contribution
AN - SCOPUS:85048468735
T3 - 34th International Conference on Machine Learning, ICML 2017
SP - 3306
EP - 3340
BT - 34th International Conference on Machine Learning, ICML 2017
PB - International Machine Learning Society (IMLS)
T2 - 34th International Conference on Machine Learning, ICML 2017
Y2 - 6 August 2017 through 11 August 2017
ER -