TY - GEN
T1 - Test coverage metrics for the network
AU - Xu, Xieyang
AU - Beckett, Ryan
AU - Jayaraman, Karthick
AU - Mahajan, Ratul
AU - Walker, David
N1 - Publisher Copyright:
© 2021 ACM.
PY - 2021/8/9
Y1 - 2021/8/9
N2 - Testing and verification have emerged as key tools in the battle to improve the reliability of networks and the services they provide. However, the success of even the best technology of this sort is limited by how effectively it is applied, and in today's enormously complex industrial networks, it is surprisingly easy to overlook particular interfaces, routes, or flows when creating a test suite. Moreover, network engineers, unlike their software counterparts, have no help to battle this problem - there are no metrics or systems to compute the quality of their test suites or the extent to which their networks have been verified. To address this gap, we develop a general framework to define and compute network coverage for stateless network data planes. It computes coverage for a range of network components (\EG, interfaces, devices, paths) and supports many types of tests (e.g., concrete versus symbolic; local versus end-to-end; tests that check network state versus those that analyze behavior). Our framework is based on the observation that any network dataplane component can be decomposed into forwarding rules and all types of tests ultimately exercise these rules using one or more packets. We build a system called Yardstick based on this framework and deploy it in Microsoft Azure. Within the first month of its deployment inside one of the production networks, it uncovered several testing gaps and helped improve testing by covering 89% more forwarding rules and 17% more network interfaces.
AB - Testing and verification have emerged as key tools in the battle to improve the reliability of networks and the services they provide. However, the success of even the best technology of this sort is limited by how effectively it is applied, and in today's enormously complex industrial networks, it is surprisingly easy to overlook particular interfaces, routes, or flows when creating a test suite. Moreover, network engineers, unlike their software counterparts, have no help to battle this problem - there are no metrics or systems to compute the quality of their test suites or the extent to which their networks have been verified. To address this gap, we develop a general framework to define and compute network coverage for stateless network data planes. It computes coverage for a range of network components (\EG, interfaces, devices, paths) and supports many types of tests (e.g., concrete versus symbolic; local versus end-to-end; tests that check network state versus those that analyze behavior). Our framework is based on the observation that any network dataplane component can be decomposed into forwarding rules and all types of tests ultimately exercise these rules using one or more packets. We build a system called Yardstick based on this framework and deploy it in Microsoft Azure. Within the first month of its deployment inside one of the production networks, it uncovered several testing gaps and helped improve testing by covering 89% more forwarding rules and 17% more network interfaces.
KW - coverage metrics
KW - network verification
KW - reliability
UR - http://www.scopus.com/inward/record.url?scp=85113270887&partnerID=8YFLogxK
UR - http://www.scopus.com/inward/citedby.url?scp=85113270887&partnerID=8YFLogxK
U2 - 10.1145/3452296.3472941
DO - 10.1145/3452296.3472941
M3 - Conference contribution
AN - SCOPUS:85113270887
T3 - SIGCOMM 2021 - Proceedings of the ACM SIGCOMM 2021 Conference
SP - 775
EP - 787
BT - SIGCOMM 2021 - Proceedings of the ACM SIGCOMM 2021 Conference
PB - Association for Computing Machinery, Inc
T2 - 2021 Annual Conference of the ACM Special Interest Group on Data Communication on the Applications, SIGCOMM 2021
Y2 - 23 August 2021 through 27 August 2021
ER -