Scalable annotation of fine-grained categories without experts

Timnit Gebru, Jonathan Krause, Jia Deng, Fei Fei Li

Research output: Chapter in Book/Report/Conference proceedingConference contribution

9 Scopus citations

Abstract

We present a crowdsourcing workflow to collect image annotations for visually similar synthetic categories without requiring experts. In animals, there is a direct link between taxonomy and visual similarity: e.g. a collie (type of dog) looks more similar to other collies (e.g. smooth collie) than a greyhound (another type of dog). However, in synthetic categories such as cars, objects with similar taxonomy can have very different appearance: e.g. a 2011 Ford F-150 Supercrew-HD looks the same as a 2011 Ford F-150 Supercrew-LL but very different from a 2011 Ford F-150 Supercrew-SVT. We introduce a graph based crowdsourcing algorithm to automatically group visually indistinguishable objects together. Using our workflow, we label 712,430 images by ∼ 1,000 Amazon Mechanical Turk workers; resulting in the largest fine-grained visual dataset reported to date with 2,657 categories of cars annotated at 1/20th the cost of hiring experts.

Original languageEnglish (US)
Title of host publicationCHI 2017 - Proceedings of the 2017 ACM SIGCHI Conference on Human Factors in Computing Systems
Subtitle of host publicationExplore, Innovate, Inspire
PublisherAssociation for Computing Machinery
Pages1877-1881
Number of pages5
ISBN (Electronic)9781450346559
DOIs
StatePublished - May 2 2017
Externally publishedYes
Event2017 ACM SIGCHI Conference on Human Factors in Computing Systems, CHI 2017 - Denver, United States
Duration: May 6 2017May 11 2017

Publication series

NameConference on Human Factors in Computing Systems - Proceedings
Volume2017-May

Other

Other2017 ACM SIGCHI Conference on Human Factors in Computing Systems, CHI 2017
CountryUnited States
CityDenver
Period5/6/175/11/17

All Science Journal Classification (ASJC) codes

  • Software
  • Human-Computer Interaction
  • Computer Graphics and Computer-Aided Design

Keywords

  • Crowdsourcing
  • Fine-grained dataset
  • Human computation

Fingerprint Dive into the research topics of 'Scalable annotation of fine-grained categories without experts'. Together they form a unique fingerprint.

  • Cite this

    Gebru, T., Krause, J., Deng, J., & Li, F. F. (2017). Scalable annotation of fine-grained categories without experts. In CHI 2017 - Proceedings of the 2017 ACM SIGCHI Conference on Human Factors in Computing Systems: Explore, Innovate, Inspire (pp. 1877-1881). (Conference on Human Factors in Computing Systems - Proceedings; Vol. 2017-May). Association for Computing Machinery. https://doi.org/10.1145/3025453.3025930