TY - JOUR
T1 - Over-parameterized adversarial training
T2 - 34th Conference on Neural Information Processing Systems, NeurIPS 2020
AU - Zhang, Yi
AU - Plevrakis, Orestis
AU - Du, Simon S.
AU - Li, Xingguo
AU - Song, Zhao
AU - Arora, Sanjeev
N1 - Funding Information:
The authors acknowledge fundings from ONR, NSF, Simons Foundation, DARPA/SRC, AWS, Schmidt Foundation and IAS.
Publisher Copyright:
© 2020 Neural information processing systems foundation. All rights reserved.
PY - 2020
Y1 - 2020
N2 - Adversarial training is a popular method to give neural nets robustness against adversarial perturbations. In practice adversarial training leads to low robust training loss. However, a rigorous explanation for why this happens under natural conditions is still missing. Recently a convergence theory for standard (non-adversarial) training was developed by various groups for very over-parametrized nets. It is unclear how to extend these results to adversarial training because of the min-max objective. Recently, a first step towards this direction was made by [14] using tools from online learning, but they require the width of the net and the running time to be exponential in input dimension d, and they consider an activation function that is not used in practice. Our work proves convergence to low robust training loss for polynomial width and running time, instead of exponential, under natural assumptions and with ReLU activation. Key element of our proof is showing that ReLU networks near initialization can approximate the step function, which may be of independent interest.
AB - Adversarial training is a popular method to give neural nets robustness against adversarial perturbations. In practice adversarial training leads to low robust training loss. However, a rigorous explanation for why this happens under natural conditions is still missing. Recently a convergence theory for standard (non-adversarial) training was developed by various groups for very over-parametrized nets. It is unclear how to extend these results to adversarial training because of the min-max objective. Recently, a first step towards this direction was made by [14] using tools from online learning, but they require the width of the net and the running time to be exponential in input dimension d, and they consider an activation function that is not used in practice. Our work proves convergence to low robust training loss for polynomial width and running time, instead of exponential, under natural assumptions and with ReLU activation. Key element of our proof is showing that ReLU networks near initialization can approximate the step function, which may be of independent interest.
UR - http://www.scopus.com/inward/record.url?scp=85098811785&partnerID=8YFLogxK
UR - http://www.scopus.com/inward/citedby.url?scp=85098811785&partnerID=8YFLogxK
M3 - Conference article
AN - SCOPUS:85098811785
SN - 1049-5258
VL - 2020-December
JO - Advances in Neural Information Processing Systems
JF - Advances in Neural Information Processing Systems
Y2 - 6 December 2020 through 12 December 2020
ER -