Fireworks: A dynamic workflow system designed for highthroughput applications

Anubhav Jain; Shyue Ping Ong; Wei Chen; Bharat Medasani; Xiaohui Qu; Michael Kocher; Miriam Brafman; Guido Petretto; Gian Marco Rignanese; Geoffroy Hautier; Daniel Gunter; Kristin A. Persson

doi:10.1002/cpe.3505

Fireworks: A dynamic workflow system designed for highthroughput applications

Anubhav Jain, Shyue Ping Ong, Wei Chen, Bharat Medasani, Xiaohui Qu, Michael Kocher, Miriam Brafman, Guido Petretto, Gian Marco Rignanese, Geoffroy Hautier, Daniel Gunter, Kristin A. Persson

Research output: Contribution to journal › Article › peer-review

438 Scopus citations

Abstract

This paper introduces FireWorks, a workflow software for running high-throughput calculation workflows at supercomputing centers. FireWorks has been used to complete over 50 million CPU-hours worth of computational chemistry and materials science calculations at the National Energy Research Supercomputing Center. It has been designed to serve the demanding high-throughput computing needs of these applications, with extensive support for (i) concurrent execution through job packing, (ii) failure detection and correction, (iii) provenance and reporting for long-running projects, (iv) automated duplicate detection, and (v) dynamic workflows (i.e., modifying the workflow graph during runtime). We have found that these features are highly relevant to enabling modern data-driven and high-throughput science applications, and we discuss our implementation strategy that rests on Python and NoSQL databases (MongoDB). Finally, we present performance data and limitations of our approach along with planned future work.

Original language	English (US)
Pages (from-to)	5037-5059
Number of pages	23
Journal	Concurrency and Computation: Practice and Experience
Volume	27
Issue number	17
DOIs	https://doi.org/10.1002/cpe.3505
State	Published - Dec 10 2015
Externally published	Yes

All Science Journal Classification (ASJC) codes

Software
Theoretical Computer Science
Computer Science Applications
Computer Networks and Communications
Computational Theory and Mathematics

Keywords

Fault-tolerant computing
High-throughput computing
Scientific workflows

Access to Document

10.1002/cpe.3505

Cite this

@article{644a1d0c0a794e24b02ff116610bc3bd,

title = "Fireworks: A dynamic workflow system designed for highthroughput applications",

abstract = "This paper introduces FireWorks, a workflow software for running high-throughput calculation workflows at supercomputing centers. FireWorks has been used to complete over 50 million CPU-hours worth of computational chemistry and materials science calculations at the National Energy Research Supercomputing Center. It has been designed to serve the demanding high-throughput computing needs of these applications, with extensive support for (i) concurrent execution through job packing, (ii) failure detection and correction, (iii) provenance and reporting for long-running projects, (iv) automated duplicate detection, and (v) dynamic workflows (i.e., modifying the workflow graph during runtime). We have found that these features are highly relevant to enabling modern data-driven and high-throughput science applications, and we discuss our implementation strategy that rests on Python and NoSQL databases (MongoDB). Finally, we present performance data and limitations of our approach along with planned future work.",

keywords = "Fault-tolerant computing, High-throughput computing, Scientific workflows",

author = "Anubhav Jain and Ong, \{Shyue Ping\} and Wei Chen and Bharat Medasani and Xiaohui Qu and Michael Kocher and Miriam Brafman and Guido Petretto and Rignanese, \{Gian Marco\} and Geoffroy Hautier and Daniel Gunter and Persson, \{Kristin A.\}",

note = "Publisher Copyright: {\textcopyright} 2015 John Wiley \& Sons, Ltd.",

year = "2015",

month = dec,

day = "10",

doi = "10.1002/cpe.3505",

language = "English (US)",

volume = "27",

pages = "5037--5059",

journal = "Concurrency and Computation: Practice and Experience",

issn = "1532-0626",

publisher = "John Wiley and Sons Ltd",

number = "17",

}

TY - JOUR

T1 - Fireworks

T2 - A dynamic workflow system designed for highthroughput applications

AU - Jain, Anubhav

AU - Ong, Shyue Ping

AU - Chen, Wei

AU - Medasani, Bharat

AU - Qu, Xiaohui

AU - Kocher, Michael

AU - Brafman, Miriam

AU - Petretto, Guido

AU - Rignanese, Gian Marco

AU - Hautier, Geoffroy

AU - Gunter, Daniel

AU - Persson, Kristin A.

PY - 2015/12/10

Y1 - 2015/12/10

N2 - This paper introduces FireWorks, a workflow software for running high-throughput calculation workflows at supercomputing centers. FireWorks has been used to complete over 50 million CPU-hours worth of computational chemistry and materials science calculations at the National Energy Research Supercomputing Center. It has been designed to serve the demanding high-throughput computing needs of these applications, with extensive support for (i) concurrent execution through job packing, (ii) failure detection and correction, (iii) provenance and reporting for long-running projects, (iv) automated duplicate detection, and (v) dynamic workflows (i.e., modifying the workflow graph during runtime). We have found that these features are highly relevant to enabling modern data-driven and high-throughput science applications, and we discuss our implementation strategy that rests on Python and NoSQL databases (MongoDB). Finally, we present performance data and limitations of our approach along with planned future work.

AB - This paper introduces FireWorks, a workflow software for running high-throughput calculation workflows at supercomputing centers. FireWorks has been used to complete over 50 million CPU-hours worth of computational chemistry and materials science calculations at the National Energy Research Supercomputing Center. It has been designed to serve the demanding high-throughput computing needs of these applications, with extensive support for (i) concurrent execution through job packing, (ii) failure detection and correction, (iii) provenance and reporting for long-running projects, (iv) automated duplicate detection, and (v) dynamic workflows (i.e., modifying the workflow graph during runtime). We have found that these features are highly relevant to enabling modern data-driven and high-throughput science applications, and we discuss our implementation strategy that rests on Python and NoSQL databases (MongoDB). Finally, we present performance data and limitations of our approach along with planned future work.

KW - Fault-tolerant computing

KW - High-throughput computing

KW - Scientific workflows

UR - https://www.scopus.com/pages/publications/84929448814

UR - https://www.scopus.com/inward/citedby.url?scp=84929448814&partnerID=8YFLogxK

U2 - 10.1002/cpe.3505

DO - 10.1002/cpe.3505

M3 - Article

AN - SCOPUS:84929448814

SN - 1532-0626

VL - 27

SP - 5037

EP - 5059

JO - Concurrency and Computation: Practice and Experience

JF - Concurrency and Computation: Practice and Experience

IS - 17

ER -

Fireworks: A dynamic workflow system designed for highthroughput applications

Abstract

All Science Journal Classification (ASJC) codes

Keywords

Access to Document

Other files and links

Fingerprint

Cite this