Home-based SVM protocols for SMP clusters: Design and performance

As small-scale shared memory multiprocessors proliferate in the market, it is very attractive to construct large-scale systems by connecting smaller multiprocessors together in software using efficient commodity network interfaces and networks. Using a shared virtual memory (SVM) layer for this purpose preserves the attractive shared memory programming abstraction across nodes. In this paper: We describe home-based SVM protocols that support symmetric multiprocessor (SMP) nodes, taking advantage of the intra-node hardware cache coherence and synchronization mechanisms. Our protocols take no special advantage of the network interface and network except as a fast communication link, and as such are very portable. We present the key design tradeoffs, discuss our choices, and describe key data structures that enable us to implement these choices quite simply. We present an implementation on a network of 4-way Intel PentiumPro SMPs interconnected with Myrinet, and provide performance results. We explore the advantages of SMP nodes over uniprocessor nodes with this protocol, as well as other performance tradeoffs, through both real implementation and simulation as appropriate, since both have important roles to play. We find one approach to deliver good parallel performance on many real applications (at least at the scale we examine) and to improve performance over SVM across uniprocessor nodes.