Striped Data Server for Scalable Parallel Data Analysis

Jin Chang, Oliver Gutsche, Igor Mandrichenko, James Pivarski

Research output: Contribution to journalConference articlepeer-review


A columnar data representation is known to be an efficient way for data storage, specifically in cases when the analysis is often done based only on a small fragment of the available data structures. A data representation like Apache Parquet is a step forward from a columnar representation, which splits data horizontally to allow for easy parallelization of data analysis. Based on the general idea of columnar data storage, working on the FNAL LDRD Project FNAL-LDRD-2016-032, we have developed a striped data representation, which, we believe, is better suited to the needs of High Energy Physics data analysis. A traditional columnar approach allows for efficient data analysis of complex structures. While keeping all the benefits of columnar data representations, the striped mechanism goes further by enabling easy parallelization of computations without requiring special hardware. We will present an implementation and some performance characteristics of such a data representation mechanism using a distributed no-SQL database or a local file system, unified under the same API and data representation model. The representation is efficient and at the same time simple so that it allows for a common data model and APIs for wide range of underlying storage mechanisms such as distributed no-SQL databases and local file systems. Striped storage adopts Numpy arrays as its basic data representation format, which makes it easy and efficient to use in Python applications. The Striped Data Server is a web service, which allows to hide the server implementation details from the end user, easily exposes data to WAN users, and allows to utilize well known and developed data caching solutions to further increase data access efficiency. We are considering the Striped Data Server as the core of an enterprise scale data analysis platform for High Energy Physics and similar areas of data processing. We have been testing this architecture with a 2TB dataset from a CMS dark matter search and plan to expand it to multiple 100 TB or even PB scale. We will present the striped format, Striped Data Server architecture and performance test results.

Original languageEnglish (US)
Article number042035
JournalJournal of Physics: Conference Series
Issue number4
StatePublished - Oct 18 2018
Event18th International Workshop on Advanced Computing and Analysis Techniques in Physics Research, ACAT 2017 - Seattle, United States
Duration: Aug 21 2017Aug 25 2017

All Science Journal Classification (ASJC) codes

  • General Physics and Astronomy


Dive into the research topics of 'Striped Data Server for Scalable Parallel Data Analysis'. Together they form a unique fingerprint.

Cite this