Spatial Action Maps for Mobile Manipulation

Jimmy Wu, Xingyuan Sun, Andy Zeng, Shuran Song, Johnny Lee, Szymon Rusinkiewicz, Thomas Funkhouser

Research output: Chapter in Book/Report/Conference proceedingConference contribution

23 Scopus citations

Abstract

Typical end-to-end formulations for learning robotic navigation involve predicting a small set of steering command actions (e.g., step forward, turn left, turn right, etc.) from images of the current state (e.g., a bird’s-eye view of a SLAM reconstruction). Instead, we show that it can be advantageous to learn with dense action representations defined in the same domain as the state. In this work, we present “spatial action maps,” in which the set of possible actions is represented by a pixel map (aligned with the input image of the current state), where each pixel represents a local navigational endpoint at the corresponding scene location. Using ConvNets to infer spatial action maps from state images, action predictions are thereby spatially anchored on local visual features in the scene, enabling significantly faster learning of complex behaviors for mobile manipulation tasks with reinforcement learning. In our experiments, we task a robot with pushing objects to a goal location, and find that policies learned with spatial action maps achieve much better performance than traditional alternatives.

Original languageEnglish (US)
Title of host publicationRobotics
Subtitle of host publicationScience and Systems XVI
EditorsMarc Toussaint, Antonio Bicchi, Tucker Hermans
PublisherMIT Press Journals
ISBN (Print)9780992374761
DOIs
StatePublished - 2020
Event16th Robotics: Science and Systems, RSS 2020 - Virtual, Online
Duration: Jul 12 2020Jul 16 2020

Publication series

NameRobotics: Science and Systems
ISSN (Electronic)2330-765X

Conference

Conference16th Robotics: Science and Systems, RSS 2020
CityVirtual, Online
Period7/12/207/16/20

All Science Journal Classification (ASJC) codes

  • Artificial Intelligence
  • Control and Systems Engineering
  • Electrical and Electronic Engineering

Fingerprint

Dive into the research topics of 'Spatial Action Maps for Mobile Manipulation'. Together they form a unique fingerprint.

Cite this