Test coverage metrics for the network

Xieyang Xu, Ryan Beckett, Karthick Jayaraman, Ratul Mahajan, David Walker

Research output: Chapter in Book/Report/Conference proceedingConference contribution

Abstract

Testing and verification have emerged as key tools in the battle to improve the reliability of networks and the services they provide. However, the success of even the best technology of this sort is limited by how effectively it is applied, and in today's enormously complex industrial networks, it is surprisingly easy to overlook particular interfaces, routes, or flows when creating a test suite. Moreover, network engineers, unlike their software counterparts, have no help to battle this problem - there are no metrics or systems to compute the quality of their test suites or the extent to which their networks have been verified. To address this gap, we develop a general framework to define and compute network coverage for stateless network data planes. It computes coverage for a range of network components (\EG, interfaces, devices, paths) and supports many types of tests (e.g., concrete versus symbolic; local versus end-to-end; tests that check network state versus those that analyze behavior). Our framework is based on the observation that any network dataplane component can be decomposed into forwarding rules and all types of tests ultimately exercise these rules using one or more packets. We build a system called Yardstick based on this framework and deploy it in Microsoft Azure. Within the first month of its deployment inside one of the production networks, it uncovered several testing gaps and helped improve testing by covering 89% more forwarding rules and 17% more network interfaces.

Original languageEnglish (US)
Title of host publicationSIGCOMM 2021 - Proceedings of the ACM SIGCOMM 2021 Conference
PublisherAssociation for Computing Machinery, Inc
Pages775-787
Number of pages13
ISBN (Electronic)9781450383837
DOIs
StatePublished - Aug 9 2021
Event2021 Annual Conference of the ACM Special Interest Group on Data Communication on the Applications, SIGCOMM 2021 - Virtual, Online, United States
Duration: Aug 23 2021Aug 27 2021

Publication series

NameSIGCOMM 2021 - Proceedings of the ACM SIGCOMM 2021 Conference

Conference

Conference2021 Annual Conference of the ACM Special Interest Group on Data Communication on the Applications, SIGCOMM 2021
Country/TerritoryUnited States
CityVirtual, Online
Period8/23/218/27/21

All Science Journal Classification (ASJC) codes

  • Computer Networks and Communications
  • Hardware and Architecture
  • Electrical and Electronic Engineering

Keywords

  • coverage metrics
  • network verification
  • reliability

Fingerprint

Dive into the research topics of 'Test coverage metrics for the network'. Together they form a unique fingerprint.

Cite this