A convergence analysis of gradient descent for deep linear neural networks

Sanjeev Arora, Noah Golowich, Nadav Cohen, Wei Hu

Research output: Contribution to conferencePaper

6 Scopus citations

Abstract

We analyze speed of convergence to global optimum for gradient descent training a deep linear neural network (parameterized as x 7→ WNWN−1 · · · W1x) by minimizing the `2 loss over whitened data. Convergence at a linear rate is guaranteed when the following hold: (i) dimensions of hidden layers are at least the minimum of the input and output dimensions; (ii) weight matrices at initialization are approximately balanced; and (iii) the initial loss is smaller than the loss of any rank-deficient solution. The assumptions on initialization (conditions (ii) and (iii)) are necessary, in the sense that violating any one of them may lead to convergence failure. Moreover, in the important case of output dimension 1, i.e. scalar regression, they are met, and thus convergence to global optimum holds, with constant probability under a random initialization scheme. Our results significantly extend previous analyses, e.g., of deep linear residual networks (Bartlett et al., 2018).

Original languageEnglish (US)
StatePublished - Jan 1 2019
Event7th International Conference on Learning Representations, ICLR 2019 - New Orleans, United States
Duration: May 6 2019May 9 2019

Conference

Conference7th International Conference on Learning Representations, ICLR 2019
CountryUnited States
CityNew Orleans
Period5/6/195/9/19

All Science Journal Classification (ASJC) codes

  • Education
  • Computer Science Applications
  • Linguistics and Language
  • Language and Linguistics

Fingerprint Dive into the research topics of 'A convergence analysis of gradient descent for deep linear neural networks'. Together they form a unique fingerprint.

  • Cite this

    Arora, S., Golowich, N., Cohen, N., & Hu, W. (2019). A convergence analysis of gradient descent for deep linear neural networks. Paper presented at 7th International Conference on Learning Representations, ICLR 2019, New Orleans, United States.