Scalable multi-label annotation

Jia Deng; Olga Russakovsky; Jonathan Krause; Michael Bernstein; Alex Berg; Li Fei-Fei

doi:10.1145/2556288.2557011

Scalable multi-label annotation

Jia Deng, Olga Russakovsky, Jonathan Krause, Michael Bernstein, Alex Berg, Li Fei-Fei

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution

119 Scopus citations

Abstract

We study strategies for scalable multi-label annotation, or for efficiently acquiring multiple labels from humans for a collection of items. We propose an algorithm that exploits correlation, hierarchy, and sparsity of the label distribution. A case study of labeling 200 objects using 20,000 images demonstrates the effectiveness of our approach. The algorithm results in up to 6x reduction in human computation time compared to the naïve method of querying a human annotator for the presence of every object in every image.

Original language	English (US)
Title of host publication	CHI 2014
Subtitle of host publication	One of a CHInd - Conference Proceedings, 32nd Annual ACM Conference on Human Factors in Computing Systems
Publisher	Association for Computing Machinery
Pages	3099-3102
Number of pages	4
ISBN (Print)	9781450324731
DOIs	https://doi.org/10.1145/2556288.2557011
State	Published - 2014
Externally published	Yes
Event	32nd Annual ACM Conference on Human Factors in Computing Systems, CHI 2014 - Toronto, ON, Canada Duration: Apr 26 2014 → May 1 2014

Publication series

Name	Conference on Human Factors in Computing Systems - Proceedings

Other

Other	32nd Annual ACM Conference on Human Factors in Computing Systems, CHI 2014
Country/Territory	Canada
City	Toronto, ON
Period	4/26/14 → 5/1/14

All Science Journal Classification (ASJC) codes

Software
Human-Computer Interaction
Computer Graphics and Computer-Aided Design

Keywords

Crowdsourcing
Human computation

Access to Document

10.1145/2556288.2557011

Cite this

Deng, J., Russakovsky, O., Krause, J., Bernstein, M., Berg, A., & Fei-Fei, L. (2014). Scalable multi-label annotation. In CHI 2014: One of a CHInd - Conference Proceedings, 32nd Annual ACM Conference on Human Factors in Computing Systems (pp. 3099-3102). (Conference on Human Factors in Computing Systems - Proceedings). Association for Computing Machinery. https://doi.org/10.1145/2556288.2557011

@inproceedings{7201d26e3d7246e1bb39fd91f6b11326,

title = "Scalable multi-label annotation",

abstract = "We study strategies for scalable multi-label annotation, or for efficiently acquiring multiple labels from humans for a collection of items. We propose an algorithm that exploits correlation, hierarchy, and sparsity of the label distribution. A case study of labeling 200 objects using 20,000 images demonstrates the effectiveness of our approach. The algorithm results in up to 6x reduction in human computation time compared to the na{\"i}ve method of querying a human annotator for the presence of every object in every image.",

keywords = "Crowdsourcing, Human computation",

author = "Jia Deng and Olga Russakovsky and Jonathan Krause and Michael Bernstein and Alex Berg and Li Fei-Fei",

year = "2014",

doi = "10.1145/2556288.2557011",

language = "English (US)",

isbn = "9781450324731",

series = "Conference on Human Factors in Computing Systems - Proceedings",

publisher = "Association for Computing Machinery",

pages = "3099--3102",

booktitle = "CHI 2014",

note = "32nd Annual ACM Conference on Human Factors in Computing Systems, CHI 2014 ; Conference date: 26-04-2014 Through 01-05-2014",

}

Deng, J , Russakovsky, O, Krause, J, Bernstein, M, Berg, A & Fei-Fei, L 2014, Scalable multi-label annotation. in CHI 2014: One of a CHInd - Conference Proceedings, 32nd Annual ACM Conference on Human Factors in Computing Systems. Conference on Human Factors in Computing Systems - Proceedings, Association for Computing Machinery, pp. 3099-3102, 32nd Annual ACM Conference on Human Factors in Computing Systems, CHI 2014, Toronto, ON, Canada, 4/26/14. https://doi.org/10.1145/2556288.2557011

Scalable multi-label annotation. / Deng, Jia ; Russakovsky, Olga; Krause, Jonathan et al.
CHI 2014: One of a CHInd - Conference Proceedings, 32nd Annual ACM Conference on Human Factors in Computing Systems. Association for Computing Machinery, 2014. p. 3099-3102 (Conference on Human Factors in Computing Systems - Proceedings).

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution

TY - GEN

T1 - Scalable multi-label annotation

AU - Deng, Jia

AU - Russakovsky, Olga

AU - Krause, Jonathan

AU - Bernstein, Michael

AU - Berg, Alex

AU - Fei-Fei, Li

PY - 2014

Y1 - 2014

N2 - We study strategies for scalable multi-label annotation, or for efficiently acquiring multiple labels from humans for a collection of items. We propose an algorithm that exploits correlation, hierarchy, and sparsity of the label distribution. A case study of labeling 200 objects using 20,000 images demonstrates the effectiveness of our approach. The algorithm results in up to 6x reduction in human computation time compared to the naïve method of querying a human annotator for the presence of every object in every image.

AB - We study strategies for scalable multi-label annotation, or for efficiently acquiring multiple labels from humans for a collection of items. We propose an algorithm that exploits correlation, hierarchy, and sparsity of the label distribution. A case study of labeling 200 objects using 20,000 images demonstrates the effectiveness of our approach. The algorithm results in up to 6x reduction in human computation time compared to the naïve method of querying a human annotator for the presence of every object in every image.

KW - Crowdsourcing

KW - Human computation

UR - http://www.scopus.com/inward/record.url?scp=84900456924&partnerID=8YFLogxK

UR - http://www.scopus.com/inward/citedby.url?scp=84900456924&partnerID=8YFLogxK

U2 - 10.1145/2556288.2557011

DO - 10.1145/2556288.2557011

M3 - Conference contribution

AN - SCOPUS:84900456924

SN - 9781450324731

T3 - Conference on Human Factors in Computing Systems - Proceedings

SP - 3099

EP - 3102

BT - CHI 2014

PB - Association for Computing Machinery

T2 - 32nd Annual ACM Conference on Human Factors in Computing Systems, CHI 2014

Y2 - 26 April 2014 through 1 May 2014

ER -

Scalable multi-label annotation

Abstract

Publication series

Other

All Science Journal Classification (ASJC) codes

Keywords

Access to Document

Other files and links

Fingerprint

Cite this