Scalable annotation of fine-grained categories without experts

Timnit Gebru, Jonathan Krause, Jia Deng, Fei Fei Li

Research output: Chapter in Book/Report/Conference proceedingConference contribution

10 Scopus citations


We present a crowdsourcing workflow to collect image annotations for visually similar synthetic categories without requiring experts. In animals, there is a direct link between taxonomy and visual similarity: e.g. a collie (type of dog) looks more similar to other collies (e.g. smooth collie) than a greyhound (another type of dog). However, in synthetic categories such as cars, objects with similar taxonomy can have very different appearance: e.g. a 2011 Ford F-150 Supercrew-HD looks the same as a 2011 Ford F-150 Supercrew-LL but very different from a 2011 Ford F-150 Supercrew-SVT. We introduce a graph based crowdsourcing algorithm to automatically group visually indistinguishable objects together. Using our workflow, we label 712,430 images by ∼ 1,000 Amazon Mechanical Turk workers; resulting in the largest fine-grained visual dataset reported to date with 2,657 categories of cars annotated at 1/20th the cost of hiring experts.

Original languageEnglish (US)
Title of host publicationCHI 2017 - Proceedings of the 2017 ACM SIGCHI Conference on Human Factors in Computing Systems
Subtitle of host publicationExplore, Innovate, Inspire
PublisherAssociation for Computing Machinery
Number of pages5
ISBN (Electronic)9781450346559
StatePublished - May 2 2017
Externally publishedYes
Event2017 ACM SIGCHI Conference on Human Factors in Computing Systems, CHI 2017 - Denver, United States
Duration: May 6 2017May 11 2017

Publication series

NameConference on Human Factors in Computing Systems - Proceedings


Other2017 ACM SIGCHI Conference on Human Factors in Computing Systems, CHI 2017
Country/TerritoryUnited States

All Science Journal Classification (ASJC) codes

  • Software
  • Human-Computer Interaction
  • Computer Graphics and Computer-Aided Design


  • Crowdsourcing
  • Fine-grained dataset
  • Human computation


Dive into the research topics of 'Scalable annotation of fine-grained categories without experts'. Together they form a unique fingerprint.

Cite this