Tiny but Mighty: Designing and Realizing Scalable Latency Tolerance for Manycore SoCs

Marcelo Orenes-Vera, Aninda Manocha, Jonathan Balkind, Fei Gao, Juan L. Aragón, David Wentzlaff, Margaret Martonosi

Research output: Chapter in Book/Report/Conference proceedingConference contribution

12 Scopus citations


Modern computing systems employ signifcant heterogeneity and specialization to meet performance targets at manageable power. However, memory latency bottlenecks remain problematic, particularly for sparse neural network and graph analytic applications where indirect memory accesses (IMAs) challenge the memory hierarchy. Decades of prior art have proposed hardware and software mechanisms to mitigate IMA latency, but they fail to analyze real-chip considerations, especially when used in SoCs and manycores. In this paper, we revisit many of these techniques while taking into account manycore integration and verifcation. We present the frst system implementation of latency tolerance hardware that provides signifcant speedups without requiring any memory hierarchy or processor tile modifcations. This is achieved through a Memory Access Parallel-Load Engine (MAPLE), integrated through the Network-on-Chip (NoC) in a scalable manner. Our hardware-software co-design allows programs to perform longlatency memory accesses asynchronously from the core, avoiding pipeline stalls, and enabling greater memory parallelism (MLP). In April 2021 we taped out a manycore chip that includes tens of MAPLE instances for efcient data supply. MAPLE demonstrates a full RTL implementation of out-of-core latency-mitigation hardware, with virtual memory support and automated compilation targetting it. This paper evaluates MAPLE integrated with a dualcore FPGA prototype running applications with full SMP Linux, and demonstrates geomean speedups of 2.35× and 2.27× over softwarebased prefetching and decoupling, respectively. Compared to stateof-the-art hardware, it provides geomean speedups of 1.82× and 1.72× over prefetching and decoupling techniques.

Original languageEnglish (US)
Title of host publicationISCA 2022 - Proceedings of the 49th Annual International Symposium on Computer Architecture
PublisherInstitute of Electrical and Electronics Engineers Inc.
Number of pages14
ISBN (Electronic)9781450386104
StatePublished - Jun 18 2022
Event49th IEEE/ACM International Symposium on Computer Architecture, ISCA 2022 - New York, United States
Duration: Jun 18 2022Jun 22 2022

Publication series

NameProceedings - International Symposium on Computer Architecture
ISSN (Print)1063-6897


Conference49th IEEE/ACM International Symposium on Computer Architecture, ISCA 2022
Country/TerritoryUnited States
CityNew York

All Science Journal Classification (ASJC) codes

  • Hardware and Architecture


  • Decoupling
  • Latency tolerance
  • Memory
  • Modular RTL


Dive into the research topics of 'Tiny but Mighty: Designing and Realizing Scalable Latency Tolerance for Manycore SoCs'. Together they form a unique fingerprint.

Cite this