Riffle: Optimized Shuffle Service for Large-Scale Data Analytics

Haoyu Zhang, Brian Cho, Ergin Seyfe, Avery Ching, Michael J. Freedman

Research output: Chapter in Book/Report/Conference proceedingConference contribution

51 Scopus citations

Abstract

The rapidly growing size of data and complexity of analytics present new challenges for large-scale data processing systems. Modern systems keep data partitions in memory for pipelined operators, and persist data across stages with wide dependencies on disks for fault tolerance. While processing can often scale well by splitting jobs into smaller tasks for better parallelism, all-to-all data transfer—called shuffle operations—become the scaling bottleneck when running many small tasks in multi-stage data analytics jobs. Our key observation is that this bottleneck is due to the superlinear increase in disk I/O operations as data volume increases. We present Riffle, an optimized shuffle service for big-data analytics frameworks that significantly improves I/O efficiency and scales to process petabytes of data. To do so, Riffle efficiently merges fragmented intermediate shuffle files into larger block files, and thus converts small, random disk I/O requests into large, sequential ones. Riffle further improves performance and fault tolerance by mixing both merged and unmerged block files to minimize merge operation overhead. Using Riffle, Facebook production jobs on Spark clusters with over 1,000 executors experience up to a 10x reduction in the number of shuffle I/O requests and 40% improvement in the end-to-end job completion time.

Original languageEnglish (US)
Title of host publicationProceedings of the 13th EuroSys Conference, EuroSys 2018
PublisherAssociation for Computing Machinery, Inc
ISBN (Electronic)9781450355841
DOIs
StatePublished - Apr 23 2018
Event13th EuroSys Conference, EuroSys 2018 - Porto, Portugal
Duration: Apr 23 2018Apr 26 2018

Publication series

NameProceedings of the 13th EuroSys Conference, EuroSys 2018
Volume2018-January

Other

Other13th EuroSys Conference, EuroSys 2018
Country/TerritoryPortugal
CityPorto
Period4/23/184/26/18

All Science Journal Classification (ASJC) codes

  • Electrical and Electronic Engineering
  • Hardware and Architecture
  • Computer Networks and Communications

Keywords

  • Big-Data Analytics Frameworks
  • I/O Optimization
  • Shuffle Service
  • Storage

Fingerprint

Dive into the research topics of 'Riffle: Optimized Shuffle Service for Large-Scale Data Analytics'. Together they form a unique fingerprint.

Cite this