Dorylus: Affordable, scalable, and accurate GNN training with distributed CPU servers and serverless threads

John Thorpe; Yifan Qiao; Jonathan Eyolfson; Shen Teng; Guanzhou Hu; Zhihao Jia; Jinliang Wei; Keval Vora; Ravi Netravali; Miryung Kim; Guoqing Harry Xu

Dorylus: Affordable, scalable, and accurate GNN training with distributed CPU servers and serverless threads

John Thorpe, Yifan Qiao, Jonathan Eyolfson, Shen Teng, Guanzhou Hu, Zhihao Jia, Jinliang Wei, Keval Vora, Ravi Netravali, Miryung Kim, Guoqing Harry Xu

Computer Science

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution

79 Scopus citations

Abstract

A graph neural network (GNN) enables deep learning on structured graph data. There are two major GNN training obstacles: 1) it relies on high-end servers with many GPUs which are expensive to purchase and maintain, and 2) limited memory on GPUs cannot scale to today’s billion-edge graphs. This paper presents Dorylus: a distributed system for training GNNs. Uniquely, Dorylus can take advantage of serverless computing to increase scalability at a low cost. The key insight guiding our design is computation separation. Computation separation makes it possible to construct a deep, bounded-asynchronous pipeline where graph and tensor parallel tasks can fully overlap, effectively hiding the network latency incurred by Lambdas. With the help of thousands of Lambda threads, Dorylus scales GNN training to billion-edge graphs. Currently, for large graphs, CPU servers offer the best performance per dollar over GPU servers. Just using Lambdas on top of Dorylus offers up to 2.75× more performance-per-dollar than CPU-only servers. Concretely, Dorylus is 1.22× faster and 4.83× cheaper than GPU servers for massive sparse graphs. Dorylus is up to 3.8× faster and 10.7× cheaper compared to existing sampling-based systems.

Original language	English (US)
Title of host publication	Proceedings of the 15th USENIX Symposium on Operating Systems Design and Implementation, OSDI 2021
Publisher	USENIX Association
Pages	495-514
Number of pages	20
ISBN (Electronic)	9781939133229
State	Published - 2021
Event	15th USENIX Symposium on Operating Systems Design and Implementation, OSDI 2021 - Virtual, Online Duration: Jul 14 2021 → Jul 16 2021

Publication series

Name	Proceedings of the 15th USENIX Symposium on Operating Systems Design and Implementation, OSDI 2021

Conference

Conference	15th USENIX Symposium on Operating Systems Design and Implementation, OSDI 2021
City	Virtual, Online
Period	7/14/21 → 7/16/21

All Science Journal Classification (ASJC) codes

Computer Networks and Communications
Hardware and Architecture
Information Systems

Cite this

Thorpe, J., Qiao, Y., Eyolfson, J., Teng, S., Hu, G., Jia, Z., Wei, J., Vora, K., Netravali, R., Kim, M., & Xu, G. H. (2021). Dorylus: Affordable, scalable, and accurate GNN training with distributed CPU servers and serverless threads. In Proceedings of the 15th USENIX Symposium on Operating Systems Design and Implementation, OSDI 2021 (pp. 495-514). (Proceedings of the 15th USENIX Symposium on Operating Systems Design and Implementation, OSDI 2021). USENIX Association.

Thorpe, John ; Qiao, Yifan ; Eyolfson, Jonathan et al. / Dorylus : Affordable, scalable, and accurate GNN training with distributed CPU servers and serverless threads. Proceedings of the 15th USENIX Symposium on Operating Systems Design and Implementation, OSDI 2021. USENIX Association, 2021. pp. 495-514 (Proceedings of the 15th USENIX Symposium on Operating Systems Design and Implementation, OSDI 2021).

@inproceedings{914e840ce0a9427eae708500d3c67b3d,

title = "Dorylus: Affordable, scalable, and accurate GNN training with distributed CPU servers and serverless threads",

abstract = "A graph neural network (GNN) enables deep learning on structured graph data. There are two major GNN training obstacles: 1) it relies on high-end servers with many GPUs which are expensive to purchase and maintain, and 2) limited memory on GPUs cannot scale to today{\textquoteright}s billion-edge graphs. This paper presents Dorylus: a distributed system for training GNNs. Uniquely, Dorylus can take advantage of serverless computing to increase scalability at a low cost. The key insight guiding our design is computation separation. Computation separation makes it possible to construct a deep, bounded-asynchronous pipeline where graph and tensor parallel tasks can fully overlap, effectively hiding the network latency incurred by Lambdas. With the help of thousands of Lambda threads, Dorylus scales GNN training to billion-edge graphs. Currently, for large graphs, CPU servers offer the best performance per dollar over GPU servers. Just using Lambdas on top of Dorylus offers up to 2.75× more performance-per-dollar than CPU-only servers. Concretely, Dorylus is 1.22× faster and 4.83× cheaper than GPU servers for massive sparse graphs. Dorylus is up to 3.8× faster and 10.7× cheaper compared to existing sampling-based systems.",

author = "John Thorpe and Yifan Qiao and Jonathan Eyolfson and Shen Teng and Guanzhou Hu and Zhihao Jia and Jinliang Wei and Keval Vora and Ravi Netravali and Miryung Kim and Xu, {Guoqing Harry}",

note = "Publisher Copyright: {\textcopyright} 2021 by The USENIX Association. All rights reserved.; 15th USENIX Symposium on Operating Systems Design and Implementation, OSDI 2021 ; Conference date: 14-07-2021 Through 16-07-2021",

year = "2021",

language = "English (US)",

series = "Proceedings of the 15th USENIX Symposium on Operating Systems Design and Implementation, OSDI 2021",

publisher = "USENIX Association",

pages = "495--514",

booktitle = "Proceedings of the 15th USENIX Symposium on Operating Systems Design and Implementation, OSDI 2021",

}

Thorpe, J, Qiao, Y, Eyolfson, J, Teng, S, Hu, G, Jia, Z, Wei, J, Vora, K, Netravali, R, Kim, M & Xu, GH 2021, Dorylus: Affordable, scalable, and accurate GNN training with distributed CPU servers and serverless threads. in Proceedings of the 15th USENIX Symposium on Operating Systems Design and Implementation, OSDI 2021. Proceedings of the 15th USENIX Symposium on Operating Systems Design and Implementation, OSDI 2021, USENIX Association, pp. 495-514, 15th USENIX Symposium on Operating Systems Design and Implementation, OSDI 2021, Virtual, Online, 7/14/21.

Dorylus: Affordable, scalable, and accurate GNN training with distributed CPU servers and serverless threads. / Thorpe, John; Qiao, Yifan; Eyolfson, Jonathan et al.
Proceedings of the 15th USENIX Symposium on Operating Systems Design and Implementation, OSDI 2021. USENIX Association, 2021. p. 495-514 (Proceedings of the 15th USENIX Symposium on Operating Systems Design and Implementation, OSDI 2021).

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution

TY - GEN

T1 - Dorylus

T2 - 15th USENIX Symposium on Operating Systems Design and Implementation, OSDI 2021

AU - Thorpe, John

AU - Qiao, Yifan

AU - Eyolfson, Jonathan

AU - Teng, Shen

AU - Hu, Guanzhou

AU - Jia, Zhihao

AU - Wei, Jinliang

AU - Vora, Keval

AU - Netravali, Ravi

AU - Kim, Miryung

AU - Xu, Guoqing Harry

PY - 2021

Y1 - 2021

N2 - A graph neural network (GNN) enables deep learning on structured graph data. There are two major GNN training obstacles: 1) it relies on high-end servers with many GPUs which are expensive to purchase and maintain, and 2) limited memory on GPUs cannot scale to today’s billion-edge graphs. This paper presents Dorylus: a distributed system for training GNNs. Uniquely, Dorylus can take advantage of serverless computing to increase scalability at a low cost. The key insight guiding our design is computation separation. Computation separation makes it possible to construct a deep, bounded-asynchronous pipeline where graph and tensor parallel tasks can fully overlap, effectively hiding the network latency incurred by Lambdas. With the help of thousands of Lambda threads, Dorylus scales GNN training to billion-edge graphs. Currently, for large graphs, CPU servers offer the best performance per dollar over GPU servers. Just using Lambdas on top of Dorylus offers up to 2.75× more performance-per-dollar than CPU-only servers. Concretely, Dorylus is 1.22× faster and 4.83× cheaper than GPU servers for massive sparse graphs. Dorylus is up to 3.8× faster and 10.7× cheaper compared to existing sampling-based systems.

AB - A graph neural network (GNN) enables deep learning on structured graph data. There are two major GNN training obstacles: 1) it relies on high-end servers with many GPUs which are expensive to purchase and maintain, and 2) limited memory on GPUs cannot scale to today’s billion-edge graphs. This paper presents Dorylus: a distributed system for training GNNs. Uniquely, Dorylus can take advantage of serverless computing to increase scalability at a low cost. The key insight guiding our design is computation separation. Computation separation makes it possible to construct a deep, bounded-asynchronous pipeline where graph and tensor parallel tasks can fully overlap, effectively hiding the network latency incurred by Lambdas. With the help of thousands of Lambda threads, Dorylus scales GNN training to billion-edge graphs. Currently, for large graphs, CPU servers offer the best performance per dollar over GPU servers. Just using Lambdas on top of Dorylus offers up to 2.75× more performance-per-dollar than CPU-only servers. Concretely, Dorylus is 1.22× faster and 4.83× cheaper than GPU servers for massive sparse graphs. Dorylus is up to 3.8× faster and 10.7× cheaper compared to existing sampling-based systems.

UR - http://www.scopus.com/inward/record.url?scp=85113859558&partnerID=8YFLogxK

UR - http://www.scopus.com/inward/citedby.url?scp=85113859558&partnerID=8YFLogxK

M3 - Conference contribution

AN - SCOPUS:85113859558

T3 - Proceedings of the 15th USENIX Symposium on Operating Systems Design and Implementation, OSDI 2021

SP - 495

EP - 514

BT - Proceedings of the 15th USENIX Symposium on Operating Systems Design and Implementation, OSDI 2021

PB - USENIX Association

Y2 - 14 July 2021 through 16 July 2021

ER -

Thorpe J, Qiao Y, Eyolfson J, Teng S, Hu G, Jia Z et al. Dorylus: Affordable, scalable, and accurate GNN training with distributed CPU servers and serverless threads. In Proceedings of the 15th USENIX Symposium on Operating Systems Design and Implementation, OSDI 2021. USENIX Association. 2021. p. 495-514. (Proceedings of the 15th USENIX Symposium on Operating Systems Design and Implementation, OSDI 2021).

Dorylus: Affordable, scalable, and accurate GNN training with distributed CPU servers and serverless threads

Abstract

Publication series

Conference

All Science Journal Classification (ASJC) codes

Other files and links

Fingerprint

Cite this