TY - GEN
T1 - Scalable annotation of fine-grained categories without experts
AU - Gebru, Timnit
AU - Krause, Jonathan
AU - Deng, Jia
AU - Li, Fei Fei
N1 - Publisher Copyright:
© 2017 ACM.
PY - 2017/5/2
Y1 - 2017/5/2
N2 - We present a crowdsourcing workflow to collect image annotations for visually similar synthetic categories without requiring experts. In animals, there is a direct link between taxonomy and visual similarity: e.g. a collie (type of dog) looks more similar to other collies (e.g. smooth collie) than a greyhound (another type of dog). However, in synthetic categories such as cars, objects with similar taxonomy can have very different appearance: e.g. a 2011 Ford F-150 Supercrew-HD looks the same as a 2011 Ford F-150 Supercrew-LL but very different from a 2011 Ford F-150 Supercrew-SVT. We introduce a graph based crowdsourcing algorithm to automatically group visually indistinguishable objects together. Using our workflow, we label 712,430 images by ∼ 1,000 Amazon Mechanical Turk workers; resulting in the largest fine-grained visual dataset reported to date with 2,657 categories of cars annotated at 1/20th the cost of hiring experts.
AB - We present a crowdsourcing workflow to collect image annotations for visually similar synthetic categories without requiring experts. In animals, there is a direct link between taxonomy and visual similarity: e.g. a collie (type of dog) looks more similar to other collies (e.g. smooth collie) than a greyhound (another type of dog). However, in synthetic categories such as cars, objects with similar taxonomy can have very different appearance: e.g. a 2011 Ford F-150 Supercrew-HD looks the same as a 2011 Ford F-150 Supercrew-LL but very different from a 2011 Ford F-150 Supercrew-SVT. We introduce a graph based crowdsourcing algorithm to automatically group visually indistinguishable objects together. Using our workflow, we label 712,430 images by ∼ 1,000 Amazon Mechanical Turk workers; resulting in the largest fine-grained visual dataset reported to date with 2,657 categories of cars annotated at 1/20th the cost of hiring experts.
KW - Crowdsourcing
KW - Fine-grained dataset
KW - Human computation
UR - http://www.scopus.com/inward/record.url?scp=85044867909&partnerID=8YFLogxK
UR - http://www.scopus.com/inward/citedby.url?scp=85044867909&partnerID=8YFLogxK
U2 - 10.1145/3025453.3025930
DO - 10.1145/3025453.3025930
M3 - Conference contribution
AN - SCOPUS:85044867909
T3 - Conference on Human Factors in Computing Systems - Proceedings
SP - 1877
EP - 1881
BT - CHI 2017 - Proceedings of the 2017 ACM SIGCHI Conference on Human Factors in Computing Systems
PB - Association for Computing Machinery
T2 - 2017 ACM SIGCHI Conference on Human Factors in Computing Systems, CHI 2017
Y2 - 6 May 2017 through 11 May 2017
ER -