Crowdsourcing in computer vision

Adriana Kovashka, Olga Russakovsky, Li Fei-Fei, Kristen Grauman

Research output: Contribution to journalReview articlepeer-review

69 Scopus citations


Computer vision systems require large amounts of manually annotated data to properly learn challenging visual concepts. Crowdsourcing platforms offer an inexpensive method to capture human knowledge and understanding, for a vast number of visual perception tasks. In this survey, we describe the types of annotations computer vision researchers have collected using crowdsourcing, and how they have ensured that this data is of high quality while annotation effort is minimized. We begin by discussing data collection on both classic (e.g., object recognition) and recent (e.g., visual story-Telling) vision tasks. We then summarize key design decisions for creating effective data collection interfaces and workflows, and present strategies for intelligently selecting the most important data instances to annotate. Finally, we conclude with some thoughts on the future of crowdsourcing in computer vision.

Original languageEnglish (US)
Pages (from-to)177-243
Number of pages67
JournalFoundations and Trends in Computer Graphics and Vision
Issue number3
StatePublished - 2016
Externally publishedYes

All Science Journal Classification (ASJC) codes

  • Computer Vision and Pattern Recognition


Dive into the research topics of 'Crowdsourcing in computer vision'. Together they form a unique fingerprint.

Cite this