A Light Recipe to Train Robust Vision Transformers

Edoardo Debenedetti, Vikash Sehwag, Prateek Mittal

Research output: Chapter in Book/Report/Conference proceedingConference contribution

2 Scopus citations

Abstract

In this paper, we ask whether Vision Transformers (ViTs) can serve as an underlying architecture for improving the adversarial robustness of machine learning models against evasion attacks. While earlier works have focused on improving Convolutional Neural Networks, we show that also ViTs are highly suitable for adversarial training to achieve competitive performance. We achieve this objective using a custom adversarial training recipe, discovered using rigorous ablation studies on a subset of the ImageNet dataset. The canonical training recipe for ViTs recommends strong data augmentation, in part to compensate for the lack of vision inductive bias of attention modules, when compared to convolutions. We show that this recipe achieves suboptimal performance when used for adversarial training. In contrast, we find that omitting all heavy data augmentation, and adding some additional bag-of-tricks ϵ- warmup and larger weight decay), significantly boosts the performance of robust ViTs. We show that our recipe generalizes to different classes of ViT architectures and large-scale models on full ImageNet-1k. Additionally, investigating the reasons for the robustness of our models, we show that it is easier to generate strong attacks during training when using our recipe and that this leads to better robustness at test time. Finally, we further study one consequence of adversarial training by proposing a way to quantify the semantic nature of adversarial perturbations and highlight its correlation with the robustness of the model. Overall, we recommend that the community should avoid translating the canonical training recipes in ViTs to robust training and rethink common training choices in the context of adversarial training. We share the code for your experiments at the following URL: https://github.com/dedeswim/vits-robustness-torch.

Original languageEnglish (US)
Title of host publicationProceedings - 2023 IEEE Conference on Secure and Trustworthy Machine Learning, SaTML 2023
PublisherInstitute of Electrical and Electronics Engineers Inc.
Pages225-253
Number of pages29
ISBN (Electronic)9781665462990
DOIs
StatePublished - 2023
Externally publishedYes
Event2023 IEEE Conference on Secure and Trustworthy Machine Learning, SaTML 2023 - Raleigh, United States
Duration: Feb 8 2023Feb 10 2023

Publication series

NameProceedings - 2023 IEEE Conference on Secure and Trustworthy Machine Learning, SaTML 2023

Conference

Conference2023 IEEE Conference on Secure and Trustworthy Machine Learning, SaTML 2023
Country/TerritoryUnited States
CityRaleigh
Period2/8/232/10/23

All Science Journal Classification (ASJC) codes

  • Computer Vision and Pattern Recognition
  • Safety, Risk, Reliability and Quality
  • Artificial Intelligence

Keywords

  • Adversarial Robustness
  • Adversarial Training
  • Computer Vision
  • Vision Transformer

Fingerprint

Dive into the research topics of 'A Light Recipe to Train Robust Vision Transformers'. Together they form a unique fingerprint.

Cite this