Aggregation and degradation in Jetstream: Streaming analytics in the wide area

Ariel Rabkin, Matvey Arye, Siddhartha Sen, Vivek S. Pai, Michael J. Freedman

Research output: Chapter in Book/Report/Conference proceedingConference contribution

78 Scopus citations

Abstract

We present JetStream, a system that allows real-time analysis of large, widely-distributed changing data sets. Traditional approaches to distributed analytics require users to specify in advance which data is to be backhauled to a central location for analysis. This is a poor match for domains where available bandwidth is scarce and it is infeasible to collect all potentially useful data. JetStream addresses bandwidth limits in two ways, both of which are explicit in the programming model. The system incorporates structured storage in the form of OLAP data cubes, so data can be stored for analysis near where it is generated. Using cubes, queries can aggregate data in ways and locations of their choosing. The system also includes adaptive filtering and other transformations that adjusts data quality to match available bandwidth. Many bandwidth-saving transformations are possible; we discuss which are appropriate for which data and how they can best be combined. We implemented a range of analytic queries on web request logs and image data. Queries could be expressed in a few lines of code. Using structured storage on source nodes conserved network bandwidth by allowing data to be collected only when needed to fulfill queries. Our adaptive control mechanisms are responsive enough to keep end-to-end latency within a few seconds, even when available bandwidth drops by a factor of two, and are flexible enough to express practical policies.

Original languageEnglish (US)
Title of host publicationProceedings of the 11th USENIX Symposium on Networked Systems Design and Implementation, NSDI 2014
PublisherUSENIX Association
Pages275-288
Number of pages14
ISBN (Electronic)9781931971096
StatePublished - Jan 1 2014
Event11th USENIX Symposium on Networked Systems Design and Implementation, NSDI 2014 - Seattle, United States
Duration: Apr 2 2014Apr 4 2014

Publication series

NameProceedings of the 11th USENIX Symposium on Networked Systems Design and Implementation, NSDI 2014

Conference

Conference11th USENIX Symposium on Networked Systems Design and Implementation, NSDI 2014
CountryUnited States
CitySeattle
Period4/2/144/4/14

All Science Journal Classification (ASJC) codes

  • Control and Systems Engineering
  • Computer Networks and Communications

Fingerprint Dive into the research topics of 'Aggregation and degradation in Jetstream: Streaming analytics in the wide area'. Together they form a unique fingerprint.

  • Cite this

    Rabkin, A., Arye, M., Sen, S., Pai, V. S., & Freedman, M. J. (2014). Aggregation and degradation in Jetstream: Streaming analytics in the wide area. In Proceedings of the 11th USENIX Symposium on Networked Systems Design and Implementation, NSDI 2014 (pp. 275-288). (Proceedings of the 11th USENIX Symposium on Networked Systems Design and Implementation, NSDI 2014). USENIX Association.