TY - JOUR
T1 - Optimizing CMS build infrastructure via Apache Mesos
AU - Abdurachmanov, David
AU - Degano, Alessandro
AU - Elmer, Peter
AU - Eulisse, Giulio
AU - Mendez, David
AU - Muzaffar, Shahzad
N1 - Funding Information:
This work was partially supported by the National Science Foundation, under Cooperative Agreement PHY-1120138, and by the U.S. Department of Energy.
PY - 2015
Y1 - 2015
N2 - The Offline Software of the CMS Experiment at the Large Hadron Collider (LHC) at CERN consists of 6M lines of in-house code, developed over a decade by nearly 1000 physicists, as well as a comparable amount of general use open-source code. A critical ingredient to the success of the construction and early operation of the WLCG was the convergence, around the year 2000, on the use of a homogeneous environment of commodity x86-64 processors and Linux. Apache Mesos is a cluster manager that provides efficient resource isolation and sharing across distributed applications, or frameworks. It can run Hadoop, Jenkins, Spark, Aurora, and other applications on a dynamically shared pool of nodes. We present how we migrated our continuous integration system to schedule jobs on a relatively small Apache Mesos enabled cluster and how this resulted in better resource usage, higher peak performance and lower latency thanks to the dynamic scheduling capabilities of Mesos.
AB - The Offline Software of the CMS Experiment at the Large Hadron Collider (LHC) at CERN consists of 6M lines of in-house code, developed over a decade by nearly 1000 physicists, as well as a comparable amount of general use open-source code. A critical ingredient to the success of the construction and early operation of the WLCG was the convergence, around the year 2000, on the use of a homogeneous environment of commodity x86-64 processors and Linux. Apache Mesos is a cluster manager that provides efficient resource isolation and sharing across distributed applications, or frameworks. It can run Hadoop, Jenkins, Spark, Aurora, and other applications on a dynamically shared pool of nodes. We present how we migrated our continuous integration system to schedule jobs on a relatively small Apache Mesos enabled cluster and how this resulted in better resource usage, higher peak performance and lower latency thanks to the dynamic scheduling capabilities of Mesos.
UR - http://www.scopus.com/inward/record.url?scp=84962296891&partnerID=8YFLogxK
UR - http://www.scopus.com/inward/citedby.url?scp=84962296891&partnerID=8YFLogxK
U2 - 10.1088/1742-6596/664/6/062013
DO - 10.1088/1742-6596/664/6/062013
M3 - Conference article
AN - SCOPUS:84962296891
SN - 1742-6588
VL - 664
JO - Journal of Physics: Conference Series
JF - Journal of Physics: Conference Series
IS - 6
M1 - 062013
T2 - 21st International Conference on Computing in High Energy and Nuclear Physics, CHEP 2015
Y2 - 13 April 2015 through 17 April 2015
ER -