Dorylus: Affordable, scalable, and accurate GNN training with distributed CPU servers and serverless threads

John Thorpe, Yifan Qiao, Jonathan Eyolfson, Shen Teng, Guanzhou Hu, Zhihao Jia, Jinliang Wei, Keval Vora, Ravi Netravali, Miryung Kim, Guoqing Harry Xu

Research output: Chapter in Book/Report/Conference proceedingConference contribution

Abstract

A graph neural network (GNN) enables deep learning on structured graph data. There are two major GNN training obstacles: 1) it relies on high-end servers with many GPUs which are expensive to purchase and maintain, and 2) limited memory on GPUs cannot scale to today’s billion-edge graphs. This paper presents Dorylus: a distributed system for training GNNs. Uniquely, Dorylus can take advantage of serverless computing to increase scalability at a low cost. The key insight guiding our design is computation separation. Computation separation makes it possible to construct a deep, bounded-asynchronous pipeline where graph and tensor parallel tasks can fully overlap, effectively hiding the network latency incurred by Lambdas. With the help of thousands of Lambda threads, Dorylus scales GNN training to billion-edge graphs. Currently, for large graphs, CPU servers offer the best performance per dollar over GPU servers. Just using Lambdas on top of Dorylus offers up to 2.75× more performance-per-dollar than CPU-only servers. Concretely, Dorylus is 1.22× faster and 4.83× cheaper than GPU servers for massive sparse graphs. Dorylus is up to 3.8× faster and 10.7× cheaper compared to existing sampling-based systems.

Original languageEnglish (US)
Title of host publicationProceedings of the 15th USENIX Symposium on Operating Systems Design and Implementation, OSDI 2021
PublisherUSENIX Association
Pages495-514
Number of pages20
ISBN (Electronic)9781939133229
StatePublished - 2021
Event15th USENIX Symposium on Operating Systems Design and Implementation, OSDI 2021 - Virtual, Online
Duration: Jul 14 2021Jul 16 2021

Publication series

NameProceedings of the 15th USENIX Symposium on Operating Systems Design and Implementation, OSDI 2021

Conference

Conference15th USENIX Symposium on Operating Systems Design and Implementation, OSDI 2021
CityVirtual, Online
Period7/14/217/16/21

All Science Journal Classification (ASJC) codes

  • Computer Networks and Communications
  • Hardware and Architecture
  • Information Systems

Fingerprint

Dive into the research topics of 'Dorylus: Affordable, scalable, and accurate GNN training with distributed CPU servers and serverless threads'. Together they form a unique fingerprint.

Cite this