Pixels to Plans: Learning Non-Prehensile Manipulation by Imitating a Planner

Tarik Tosun, Eric Mitchell, Ben Eisner, Jinwook Huh, Bhoram Lee, Daewon Lee, Volkan Isler, H. Sebastian Seung, Daniel Lee

Research output: Chapter in Book/Report/Conference proceedingConference contribution

Abstract

We present a novel method enabling robots to quickly learn to manipulate objects by leveraging a motion planner to generate 'expert' training trajectories from a small amount of human-labeled data. In contrast to the traditional sense-plan-act cycle, we propose a deep learning architecture and training regimen called PtPNet that can estimate effective end-effector trajectories for manipulation directly from a single RGB-D image of an object. Additionally, we present a data collection and augmentation pipeline that enables the automatic generation of large numbers (millions) of training image and trajectory examples with almost no human labeling effort.We demonstrate our approach in a non-prehensile tool-based manipulation task, specifically picking up shoes with a hook. In hardware experiments, PtPNet generates motion plans (open-loop trajectories) that reliably (89% success over 189 trials) pick up four very different shoes from a range of positions and orientations, and reliably picks up a shoe it has never seen before. Compared with a traditional sense-plan-act paradigm, our system has the advantages of operating on sparse information (single RGB-D frame), producing high-quality trajectories much faster than the expert planner (300ms versus several seconds), and generalizing effectively to previously unseen shoes. Video available at https://youtu.be/voIkyiBtwn4.

Original languageEnglish (US)
Title of host publication2019 IEEE/RSJ International Conference on Intelligent Robots and Systems, IROS 2019
PublisherInstitute of Electrical and Electronics Engineers Inc.
Pages7431-7438
Number of pages8
ISBN (Electronic)9781728140049
DOIs
StatePublished - Nov 2019
Externally publishedYes
Event2019 IEEE/RSJ International Conference on Intelligent Robots and Systems, IROS 2019 - Macau, China
Duration: Nov 3 2019Nov 8 2019

Publication series

NameIEEE International Conference on Intelligent Robots and Systems
ISSN (Print)2153-0858
ISSN (Electronic)2153-0866

Conference

Conference2019 IEEE/RSJ International Conference on Intelligent Robots and Systems, IROS 2019
CountryChina
CityMacau
Period11/3/1911/8/19

All Science Journal Classification (ASJC) codes

  • Control and Systems Engineering
  • Software
  • Computer Vision and Pattern Recognition
  • Computer Science Applications

Fingerprint Dive into the research topics of 'Pixels to Plans: Learning Non-Prehensile Manipulation by Imitating a Planner'. Together they form a unique fingerprint.

  • Cite this

    Tosun, T., Mitchell, E., Eisner, B., Huh, J., Lee, B., Lee, D., Isler, V., Seung, H. S., & Lee, D. (2019). Pixels to Plans: Learning Non-Prehensile Manipulation by Imitating a Planner. In 2019 IEEE/RSJ International Conference on Intelligent Robots and Systems, IROS 2019 (pp. 7431-7438). [8968224] (IEEE International Conference on Intelligent Robots and Systems). Institute of Electrical and Electronics Engineers Inc.. https://doi.org/10.1109/IROS40897.2019.8968224