Multi-probe LSH: Efficient indexing for high-dimensional similarity search

Qin Lv, William Josephson, Zhe Wang, Moses Charikar, Kai Li

Research output: Chapter in Book/Report/Conference proceedingConference contribution

528 Scopus citations

Abstract

Similarity indices for high-dimensional data are very desirable for building content-based search systems for feature-rich data such as audio, images, videos, and other sensor data. Recently, locality sensitive hashing (LSH) and its variations have been proposed as indexing techniques for approximate similarity search. A significant drawback of these approaches is the requirement for a large number of hash tables in order to achieve good search quality. This paper proposes a new indexing scheme called multi-probe LSH that overcomes this drawback. Multi-probe LSH is built on the well-known LSH technique, but it intelligently probes multiple buckets that are likely to contain query results in a hash table. Our method is inspired by and improves upon recent theoretical work on entropy-based LSH designed to reduce the space requirement of the basic LSH method. We have implemented the multi-probe LSH method and evaluated the implementation with two different high-dimensional datasets. Our evaluation shows that the multi-probe LSH method substantially improves upon previously proposed methods in both space and time efficiency. To achieve the same search quality, multi-probe LSH has a similar time-efficiency as the basic LSH method while reducing the number of hash tables by an order of magnitude. In comparison with the entropy-based LSH method, to achieve the same search quality, multi-probe LSH uses less query time and 5 to 8 times fewer number of hash tables.

Original languageEnglish (US)
Title of host publication33rd International Conference on Very Large Data Bases, VLDB 2007 - Conference Proceedings
EditorsJohannes Gehrke, Christoph Koch, Minos Garofalakis, Karl Aberer, Carl-Christian Kanne, Erich J. Neuhold, Venkatesh Ganti, Wolfgang Klas, Chee-Yong Chan, Divesh Srivastava, Dana Florescu, Anand Deshpande
PublisherAssociation for Computing Machinery, Inc
Pages950-961
Number of pages12
ISBN (Electronic)9781595936493
StatePublished - Jan 1 2007
Event33rd International Conference on Very Large Data Bases, VLDB 2007 - Vienna, Austria
Duration: Sep 23 2007Sep 27 2007

Publication series

Name33rd International Conference on Very Large Data Bases, VLDB 2007 - Conference Proceedings

Other

Other33rd International Conference on Very Large Data Bases, VLDB 2007
CountryAustria
CityVienna
Period9/23/079/27/07

All Science Journal Classification (ASJC) codes

  • Hardware and Architecture
  • Information Systems and Management
  • Information Systems
  • Software

Fingerprint Dive into the research topics of 'Multi-probe LSH: Efficient indexing for high-dimensional similarity search'. Together they form a unique fingerprint.

  • Cite this

    Lv, Q., Josephson, W., Wang, Z., Charikar, M., & Li, K. (2007). Multi-probe LSH: Efficient indexing for high-dimensional similarity search. In J. Gehrke, C. Koch, M. Garofalakis, K. Aberer, C-C. Kanne, E. J. Neuhold, V. Ganti, W. Klas, C-Y. Chan, D. Srivastava, D. Florescu, & A. Deshpande (Eds.), 33rd International Conference on Very Large Data Bases, VLDB 2007 - Conference Proceedings (pp. 950-961). (33rd International Conference on Very Large Data Bases, VLDB 2007 - Conference Proceedings). Association for Computing Machinery, Inc.