Bamboo: Making Preemptible Instances Resilient for Affordable Training of Large DNNs

John Thorpe, Pengzhan Zhao, Jonathan Eyolfson, Yifan Qiao, Zhihao Jia, Minjia Zhang, Ravi Netravali, Guoqing Harry Xu

Research output: Chapter in Book/Report/Conference proceedingConference contribution

6 Scopus citations

Abstract

DNN models across many domains continue to grow in size, resulting in high resource requirements for effective training, and unpalatable (and often unaffordable) costs for organizations and research labs across scales. This paper aims to significantly reduce training costs with effective use of preemptible instances, i.e., those that can be obtained at a much cheaper price while idle, but may be preempted whenever requested by priority users. Doing so, however, requires new forms of resiliency and efficiency to cope with the possibility of frequent preemptions - a failure model that is drastically different from the occasional failures in normal cluster settings that existing checkpointing techniques target. We present Bamboo, a distributed system that tackles these challenges by introducing redundant computations into the training pipeline, i.e., whereby one node performs computations over not only its own layers but also over some layers in its neighbor. Our key insight is that training large models often requires pipeline parallelism where “pipeline bubbles” naturally exist. Bamboo carefully fills redundant computations into these bubbles, providing resilience at a low cost. Across a variety of widely used DNN models, Bamboo outperforms traditional checkpointing by 3.7× in training throughput, and reduces costs by 2.4× compared to a setting where on-demand instances are used.

Original languageEnglish (US)
Title of host publicationProceedings of the 20th USENIX Symposium on Networked Systems Design and Implementation, NSDI 2023
PublisherUSENIX Association
Pages497-513
Number of pages17
ISBN (Electronic)9781939133335
StatePublished - 2023
Event20th USENIX Symposium on Networked Systems Design and Implementation, NSDI 2023 - Boston, United States
Duration: Apr 17 2023Apr 19 2023

Publication series

NameProceedings of the 20th USENIX Symposium on Networked Systems Design and Implementation, NSDI 2023

Conference

Conference20th USENIX Symposium on Networked Systems Design and Implementation, NSDI 2023
Country/TerritoryUnited States
CityBoston
Period4/17/234/19/23

All Science Journal Classification (ASJC) codes

  • Computer Networks and Communications
  • Control and Systems Engineering

Fingerprint

Dive into the research topics of 'Bamboo: Making Preemptible Instances Resilient for Affordable Training of Large DNNs'. Together they form a unique fingerprint.

Cite this