Leveraging the wisdom of the crowd for fine-grained recognition

Jia Deng, Jonathan Krause, Michael Stark, Li Fei-Fei

Research output: Contribution to journalArticlepeer-review

35 Scopus citations


Fine-grained recognition concerns categorization at sub-ordinate levels, where the distinction between object classes is highly local. Compared to basic level recognition, fine-grained categorization can be more challenging as there are in general less data and fewer discriminative features. This necessitates the use of a stronger prior for feature selection. In this work, we include humans in the loop to help computers select discriminative features. We introduce a novel online game called 'Bubbles' that reveals discriminative features humans use. The player's goal is to identify the category of a heavily blurred image. During the game, the player can choose to reveal full details of circular regions ('bubbles'), with a certain penalty. With proper setup the game generates discriminative bubbles with assured quality. We next propose the 'BubbleBank' representation that uses the human selected bubbles to improve machine recognition performance. Finally, we demonstrate how to extend BubbleBank to a view-invariant 3D representation. Experiments demonstrate that our approach yields large improvements over the previous state of the art on challenging benchmarks.

Original languageEnglish (US)
Article number7115172
Pages (from-to)666-676
Number of pages11
JournalIEEE Transactions on Pattern Analysis and Machine Intelligence
Issue number4
StatePublished - Apr 1 2016
Externally publishedYes

All Science Journal Classification (ASJC) codes

  • Software
  • Artificial Intelligence
  • Applied Mathematics
  • Computer Vision and Pattern Recognition
  • Computational Theory and Mathematics


  • Crowdsourcing
  • Gamification
  • Human Computation
  • Object Recognition


Dive into the research topics of 'Leveraging the wisdom of the crowd for fine-grained recognition'. Together they form a unique fingerprint.

Cite this