Text2Shape: Generating Shapes from Natural Language by Learning Joint Embeddings

Kevin Chen, Christopher B. Choy, Manolis Savva, Angel X. Chang, Thomas Funkhouser, Silvio Savarese

Research output: Chapter in Book/Report/Conference proceedingConference contribution

2 Scopus citations

Abstract

We present a method for generating colored 3D shapes from natural language. To this end, we first learn joint embeddings of freeform text descriptions and colored 3D shapes. Our model combines and extends learning by association and metric learning approaches to learn implicit cross-modal connections, and produces a joint representation that captures the many-to-many relations between language and physical properties of 3D shapes such as color and shape. To evaluate our approach, we collect a large dataset of natural language descriptions for physical 3D objects in the ShapeNet dataset. With this learned joint embedding we demonstrate text-to-shape retrieval that outperforms baseline approaches. Using our embeddings with a novel conditional Wasserstein GAN framework, we generate colored 3D shapes from text. Our method is the first to connect natural language text with realistic 3D objects exhibiting rich variations in color, texture, and shape detail.

Original languageEnglish (US)
Title of host publicationComputer Vision - ACCV 2018 - 14th Asian Conference on Computer Vision, Revised Selected Papers
EditorsHongdong Li, C.V. Jawahar, Konrad Schindler, Greg Mori
PublisherSpringer Verlag
Pages100-116
Number of pages17
ISBN (Print)9783030208929
DOIs
StatePublished - Jan 1 2019
Event14th Asian Conference on Computer Vision, ACCV 2018 - Perth, Australia
Duration: Dec 2 2018Dec 6 2018

Publication series

NameLecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)
Volume11363 LNCS
ISSN (Print)0302-9743
ISSN (Electronic)1611-3349

Conference

Conference14th Asian Conference on Computer Vision, ACCV 2018
CountryAustralia
CityPerth
Period12/2/1812/6/18

All Science Journal Classification (ASJC) codes

  • Theoretical Computer Science
  • Computer Science(all)

Fingerprint Dive into the research topics of 'Text2Shape: Generating Shapes from Natural Language by Learning Joint Embeddings'. Together they form a unique fingerprint.

  • Cite this

    Chen, K., Choy, C. B., Savva, M., Chang, A. X., Funkhouser, T., & Savarese, S. (2019). Text2Shape: Generating Shapes from Natural Language by Learning Joint Embeddings. In H. Li, C. V. Jawahar, K. Schindler, & G. Mori (Eds.), Computer Vision - ACCV 2018 - 14th Asian Conference on Computer Vision, Revised Selected Papers (pp. 100-116). (Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics); Vol. 11363 LNCS). Springer Verlag. https://doi.org/10.1007/978-3-030-20893-6_7