TY - GEN
T1 - Parameterizing Activation Functions for Adversarial Robustness
AU - Dai, Sihui
AU - Mahloujifar, Saeed
AU - Mittal, Prateek
N1 - Funding Information:
This work was supported in part by the National Science Foundation under grants CNS-1553437 and CNS-1704105, the ARL's Army Artificial Intelligence Innovation Institute (A2I2), the Office of Naval Research Young Investigator Award, the Army Re-search Office Young Investigator Prize, Schmidt DataX award, and Princeton E-ffiliates Award. This material is also based upon work supported by the National Science Foundation Graduate Research Fellowship under Grant No. DGE-2039656
Funding Information:
We would like to thank Vikash Sehwag and Chong Xiang for their discussions on this project and feedback on the paper draft. This work was supported in part by the National Science Foundation under grants CNS-1553437 and CNS-1704105, the ARL’s Army Artificial Intelligence InnovationInstitute (A2I2), the Office of Naval Research Young Investigator Award, the Army Research Office Young Investigator Prize, Schmidt DataX award, and Princeton E-ffiliates Award. This material is also based upon work supported by the National Science Foundation Graduate Research Fellowship under Grant No. DGE-2039656. Any opinions, findings, and conclusions or recommendations expressed in this material are those of the author(s) and do not necessarily reflect the views of the National Science Foundation.
Publisher Copyright:
© 2022 IEEE.
PY - 2022
Y1 - 2022
N2 - Deep neural networks are known to be vulnerable to adversarially perturbed inputs. A commonly used defense is adversarial training, whose performance is influenced by model architecture. While previous works have studied the impact of varying model width and depth on robustness, the impact of using learnable parametric activation functions (PAFs) has not been studied. We study how using learnable PAFs can improve robustness in conjunction with adversarial training. We first ask the question: Can changing activation function shape improve robustness? To address this, we choose a set of PAFs with parameters that allow us to independently control behavior on negative inputs, inputs near zero, and positive inputs. Using these PAFs, we train models using adversarial training with fixed PAF shape parameter values. We find that all regions of PAF shape influence the robustness of obtained models, however only variation in certain regions (inputs near zero, positive inputs) can improve robustness over ReLU. We then combine learnable PAFs with adversarial training and analyze robust performance. We find that choice of activation function can significantly impact the robustness of the trained model. We find that only certain PAFs, such as smooth PAFs, are able to improve robustness significantly over ReLU. Overall, our work puts into context the importance of activation functions in adversarially trained models.
AB - Deep neural networks are known to be vulnerable to adversarially perturbed inputs. A commonly used defense is adversarial training, whose performance is influenced by model architecture. While previous works have studied the impact of varying model width and depth on robustness, the impact of using learnable parametric activation functions (PAFs) has not been studied. We study how using learnable PAFs can improve robustness in conjunction with adversarial training. We first ask the question: Can changing activation function shape improve robustness? To address this, we choose a set of PAFs with parameters that allow us to independently control behavior on negative inputs, inputs near zero, and positive inputs. Using these PAFs, we train models using adversarial training with fixed PAF shape parameter values. We find that all regions of PAF shape influence the robustness of obtained models, however only variation in certain regions (inputs near zero, positive inputs) can improve robustness over ReLU. We then combine learnable PAFs with adversarial training and analyze robust performance. We find that choice of activation function can significantly impact the robustness of the trained model. We find that only certain PAFs, such as smooth PAFs, are able to improve robustness significantly over ReLU. Overall, our work puts into context the importance of activation functions in adversarially trained models.
KW - activation functions
KW - adversarial robustness
KW - adversarial training
UR - http://www.scopus.com/inward/record.url?scp=85136149927&partnerID=8YFLogxK
UR - http://www.scopus.com/inward/citedby.url?scp=85136149927&partnerID=8YFLogxK
U2 - 10.1109/SPW54247.2022.9833884
DO - 10.1109/SPW54247.2022.9833884
M3 - Conference contribution
AN - SCOPUS:85136149927
T3 - Proceedings - 43rd IEEE Symposium on Security and Privacy Workshops, SPW 2022
SP - 80
EP - 87
BT - Proceedings - 43rd IEEE Symposium on Security and Privacy Workshops, SPW 2022
PB - Institute of Electrical and Electronics Engineers Inc.
T2 - 43rd IEEE Symposium on Security and Privacy Workshops, SPW 2022
Y2 - 23 May 2022 through 26 May 2022
ER -