F4: Facebook's warm blob storage system

Subramanian Muralidhar, Wyatt Lloyd, Sabyasachi Roy, Cory Hill, Ernest Lin, Weiwen Liu, Satadru Pan, Shiva Shankar, Viswanath Sivakumar, Linpeng Tang, Sanjeev Kumar

Research output: Chapter in Book/Report/Conference proceedingConference contribution

171 Scopus citations

Abstract

Facebook's corpus of photos, videos, and other Binary Large OBjects (BLOBs) that need to be reliably stored and quickly accessible is massive and continues to grow. As the footprint of BLOBs increases, storing them in our traditional storage system, Haystack, is becoming increasingly inefficient. To increase our storage efficiency, measured in the effective-replication-factor of BLOBs, we examine the underlying access patterns of BLOBs and identify temperature zones that include hot BLOBs that are accessed frequently and warm BLOBs that are accessed far less often. Our overall BLOB storage system is designed to isolate warm BLOBs and enable us to use a specialized warm BLOB storage system, f4. f4 is a new system that lowers the effective-replication-factor of warm BLOBs while remaining fault tolerant and able to support the lower throughput demands. f4 currently stores over 65PBs of logical BLOBs and reduces their effective-replication-factor from 3.6 to either 2.8 or 2.1. f4 provides low latency; is resilient to disk, host, rack, and datacenter failures; and provides sufficient throughput for warm BLOBs.

Original languageEnglish (US)
Title of host publicationProceedings of the 11th USENIX Symposium on Operating Systems Design and Implementation, OSDI 2014
PublisherUSENIX Association
Pages383-398
Number of pages16
ISBN (Electronic)9781931971164
StatePublished - Jan 1 2014
Externally publishedYes
Event11th USENIX Symposium on Operating Systems Design and Implementation, OSDI 2014 - Broomfield, United States
Duration: Oct 6 2014Oct 8 2014

Publication series

NameProceedings of the 11th USENIX Symposium on Operating Systems Design and Implementation, OSDI 2014

Conference

Conference11th USENIX Symposium on Operating Systems Design and Implementation, OSDI 2014
CountryUnited States
CityBroomfield
Period10/6/1410/8/14

All Science Journal Classification (ASJC) codes

  • Information Systems
  • Computer Networks and Communications
  • Hardware and Architecture

Fingerprint Dive into the research topics of 'F4: Facebook's warm blob storage system'. Together they form a unique fingerprint.

  • Cite this

    Muralidhar, S., Lloyd, W., Roy, S., Hill, C., Lin, E., Liu, W., Pan, S., Shankar, S., Sivakumar, V., Tang, L., & Kumar, S. (2014). F4: Facebook's warm blob storage system. In Proceedings of the 11th USENIX Symposium on Operating Systems Design and Implementation, OSDI 2014 (pp. 383-398). (Proceedings of the 11th USENIX Symposium on Operating Systems Design and Implementation, OSDI 2014). USENIX Association.