Leveraging Language for Accelerated Learning of Tool Manipulation

Allen Z. Ren, Bharat Govil, Tsung Yen Yang, Karthik Narasimhan, Anirudha Majumdar

Research output: Contribution to journalConference articlepeer-review

4 Scopus citations


Robust and generalized tool manipulation requires an understanding of the properties and affordances of different tools. We investigate whether linguistic information about a tool (e.g., its geometry, common uses) can help control policies adapt faster to new tools for a given task. We obtain diverse descriptions of various tools in natural language and use pre-trained language models to generate their feature representations. We then perform language-conditioned meta-learning to learn policies that can efficiently adapt to new tools given their corresponding text descriptions. Our results demonstrate that combining linguistic information and meta-learning significantly accelerates tool learning in several manipulation tasks including pushing, lifting, sweeping, and hammering.

Original languageEnglish (US)
Pages (from-to)1531-1541
Number of pages11
JournalProceedings of Machine Learning Research
StatePublished - 2023
Externally publishedYes
Event6th Conference on Robot Learning, CoRL 2022 - Auckland, New Zealand
Duration: Dec 14 2022Dec 18 2022

All Science Journal Classification (ASJC) codes

  • Artificial Intelligence
  • Software
  • Control and Systems Engineering
  • Statistics and Probability


  • Language for Robotics
  • Meta-learning
  • Tool Manipulation


Dive into the research topics of 'Leveraging Language for Accelerated Learning of Tool Manipulation'. Together they form a unique fingerprint.

Cite this