All-pairs: An abstraction for data-intensive cloud computing

Christopher Moretti, Jared Bulosan, Douglas Thain, Patrick J. Flynn

Research output: Chapter in Book/Report/Conference proceedingConference contribution

71 Scopus citations

Abstract

Although modern parallel and distributed computing systems provide easy access to large amounts of computing power, it is not always easy for non-expert users to harness these large systems effectively. A large workload composed in what seems to be the obvious way by a naive user may accidentally abuse shared resources and achieve very poor performance. To address this problem, we propose that production systems should provide end users with high-level abstractions that allow for the easy expression and efficient execution of data intensive workloads. We present one example of an abstraction - All-Pairs - that fits the needs of several data-intensive scientific applications. We demonstrate that an optimized All-Pairs abstraction is both easier to use than the underlying system, and achieves performance orders of magnitude better than the obvious but naive approach, and twice as fast as a hand-optimized conventional approach.

Original languageEnglish (US)
Title of host publicationIPDPS Miami 2008 - Proceedings of the 22nd IEEE International Parallel and Distributed Processing Symposium, Program and CD-ROM
DOIs
StatePublished - Sep 10 2008
Externally publishedYes
EventIPDPS 2008 - 22nd IEEE International Parallel and Distributed Processing Symposium - Miami, FL, United States
Duration: Apr 14 2008Apr 18 2008

Publication series

NameIPDPS Miami 2008 - Proceedings of the 22nd IEEE International Parallel and Distributed Processing Symposium, Program and CD-ROM

Other

OtherIPDPS 2008 - 22nd IEEE International Parallel and Distributed Processing Symposium
CountryUnited States
CityMiami, FL
Period4/14/084/18/08

All Science Journal Classification (ASJC) codes

  • Hardware and Architecture
  • Software
  • Electrical and Electronic Engineering

Fingerprint Dive into the research topics of 'All-pairs: An abstraction for data-intensive cloud computing'. Together they form a unique fingerprint.

  • Cite this

    Moretti, C., Bulosan, J., Thain, D., & Flynn, P. J. (2008). All-pairs: An abstraction for data-intensive cloud computing. In IPDPS Miami 2008 - Proceedings of the 22nd IEEE International Parallel and Distributed Processing Symposium, Program and CD-ROM [4536311] (IPDPS Miami 2008 - Proceedings of the 22nd IEEE International Parallel and Distributed Processing Symposium, Program and CD-ROM). https://doi.org/10.1109/IPDPS.2008.4536311