Coherence domain restriction on large scale systems

Yaosheng Fu, Tri M. Nguyen, David Wentzlaff

Research output: Chapter in Book/Report/Conference proceedingConference contribution

31 Scopus citations


Designing massive scale cache coherence systems has been an elusive goal. Whether it be on large-scale GPUs, future thousand-core chips, or across million-core warehouse scale computers, having shared memory, even to a limited extent, improves programmability. This work sidesteps the traditional challenges of creating massively scalable cache coherence by restricting coherence to flexible subsets (domains) of a system's total cores and home nodes. This paper proposes Coherence Domain Restriction (CDR), a novel coherence framework that enables the creation of thousand to million core systems that use shared memory while maintaining low storage and energy overhead. Inspired by the observation that the majority of cache lines are only shared by a subset of cores either due to limited application parallelism or limited page sharing, CDR restricts the coherence domain from global cache coherence to VM-level, application-level, or page-level. We explore two types of restriction, one which limits the total number of sharers that can access a coherence domain and one which limits the number and location of home nodes that partake in a coherence domain. Each independent coherence domain only tracks the cores in its domain instead of the whole system, thereby removing the need for a coherence scheme built on top of CDR to scale. Sharer Restriction achieves constant storage overhead as core count increases while Home Restriction provides localized communication enabling higher performance. Unlike previous systems, CDR is flexible and does not restrict the location of the home nodes or sharers within a domain. We evaluate CDR in the context of a 1024-core chip and in the novel application of shared memory to a 1,000,000-core warehouse scale computer. Sharer Restriction results in significant area savings, while Home Restriction in the 1024-core chip and 1,000,000-core system increases performance by 29% and 23.04x respectively when comparing with global home placement. We implemented the entire CDR framework in a 25-core processor taped out in IBM's 32nm SOI process and present a detailed area characterization.

Original languageEnglish (US)
Title of host publicationProceedings - 48th Annual IEEE/ACM International Symposium on Microarchitecture, MICRO 2015
PublisherIEEE Computer Society
Number of pages13
ISBN (Electronic)9781450340342
StatePublished - Dec 5 2015
Event48th Annual IEEE/ACM International Symposium on Microarchitecture, MICRO 2015 - Waikiki, United States
Duration: Dec 5 2015Dec 9 2015

Publication series

NameProceedings of the Annual International Symposium on Microarchitecture, MICRO
ISSN (Print)1072-4451


Other48th Annual IEEE/ACM International Symposium on Microarchitecture, MICRO 2015
Country/TerritoryUnited States

All Science Journal Classification (ASJC) codes

  • Hardware and Architecture


  • cache coherence
  • home placement
  • shared memory


Dive into the research topics of 'Coherence domain restriction on large scale systems'. Together they form a unique fingerprint.

Cite this