Failure Prediction with Statistical Guarantees for Vision-Based Robot Control

Alec Farid, David Snyder, Allen Z. Ren, Anirudha Majumdar

Research output: Chapter in Book/Report/Conference proceedingConference contribution

4 Scopus citations


We are motivated by the problem of performing failure prediction for safety-critical robotic systems with highdimensional sensor observations (e.g., vision). Given access to a black-box control policy (e.g., in the form of a neural network) and a dataset of training environments, we present an approach for synthesizing a failure predictor with guaranteed bounds on false-positive and false-negative errors. In order to achieve this, we utilize techniques from Probably Approximately Correct (PAC)-Bayes generalization theory. In addition, we present novel class-conditional bounds that allow us to trade-off the relative rates of false-positive vs. false-negative errors. We propose algorithms that train failure predictors (that take as input the history of sensor observations) by minimizing our theoretical error bounds. We demonstrate the resulting approach using extensive simulation and hardware experiments for vision-based navigation with a drone and grasping objects with a robotic manipulator equipped with a wrist-mounted RGB-D camera. These experiments illustrate the ability of our approach to (1) provide strong bounds on failure prediction error rates (that closely match empirical error rates), and (2) improve safety by predicting failures.

Original languageEnglish (US)
Title of host publicationRobotics
Subtitle of host publicationScience and Systems
EditorsKris Hauser, Dylan Shell, Shoudong Huang
PublisherMIT Press Journals
ISBN (Print)9780992374785
StatePublished - 2022
Event18th Robotics: Science and Systems, RSS 2022 - New York City, United States
Duration: Jun 27 2022 → …

Publication series

NameRobotics: Science and Systems
ISSN (Electronic)2330-765X


Conference18th Robotics: Science and Systems, RSS 2022
Country/TerritoryUnited States
CityNew York City
Period6/27/22 → …

All Science Journal Classification (ASJC) codes

  • Artificial Intelligence
  • Control and Systems Engineering
  • Electrical and Electronic Engineering


Dive into the research topics of 'Failure Prediction with Statistical Guarantees for Vision-Based Robot Control'. Together they form a unique fingerprint.

Cite this