Convergence of gradient descent on separable data

Mor Shpigel Nacson, Jason D. Lee, Suriya Gunasekar, Pedro H.P. Savarese, Nathan Srebro, Daniel Soudry

Research output: Contribution to conferencePaperpeer-review

42 Scopus citations


We provide a detailed study on the implicit bias of gradient descent when optimizing loss functions with strictly monotone tails, such as the logistic loss, over separable datasets. We look at two basic questions: (a) what are the conditions on the tail of the loss function under which gradient descent converges in the direction of the L2 maximum-margin separator? (b) how does the rate of margin convergence depend on the tail of the loss function and the choice of the step size? We show that for a large family of super-polynomial tailed losses, gradient descent iterates on linear networks of any depth converge in the direction of L2 maximum-margin solution, while this does not hold for losses with heavier tails. Within this family, for simple linear models we show that the optimal rates with fixed step size is indeed obtained for the commonly used exponentially tailed losses such as logistic loss. However, with a fixed step size the optimal convergence rate is extremely slow as 1/log(t), as also proved in Soudry et al. (2018a). For linear models with exponential loss, we further prove that the convergence rate could be improved to log(t)/√t by using aggressive step sizes that compensates for the rapidly vanishing gradients. Numerical results suggest this method might be useful for deep networks.

Original languageEnglish (US)
StatePublished - 2020
Externally publishedYes
Event22nd International Conference on Artificial Intelligence and Statistics, AISTATS 2019 - Naha, Japan
Duration: Apr 16 2019Apr 18 2019


Conference22nd International Conference on Artificial Intelligence and Statistics, AISTATS 2019

All Science Journal Classification (ASJC) codes

  • Artificial Intelligence
  • Statistics and Probability


Dive into the research topics of 'Convergence of gradient descent on separable data'. Together they form a unique fingerprint.

Cite this