Sizing sketches: A rank-based analysis for similarity search

Zhe Wang, Wei Dong, William Josephson, Qin Lv, Moses Charikar, Kai Li

Research output: Chapter in Book/Report/Conference proceedingConference contribution

22 Scopus citations

Abstract

Sketches are compact data structures that can be used to estimate properties of the original data in building large-scale search engines and data analysis systems. Recent theoretical and experimental studies have shown that sketches constructed from feature vectors using randomized projections can effectively approximate L1 distance on the feature vectors with the Hamming distance on their sketches. Furthermore, such sketches can achieve good filtering accuracy while reducing the metadata space requirement and speeding up similarity searches by an order of magnitude. However, it is not clear how to choose the size of the sketches since it depends ondata type, dataset size, and desired filtering quality. In real systems designs, it is necessary to understand how to choose sketch size without the dataset, or at least without the whole datase. This paper presents an analytical model and experimental results to help system designers make such design decisions. We present arank-based filtering model that describes the relationship between sketch size and data set size based on the dataset distance distribution. Our experimental results with several datasets including images, audio, and 3D shapes show that the model yields good, conservative predictions. We show that the parameters of the model can be set with a small sample data set and the resulting model can make good predictions for a large dataset. We illustrate how to apply the approach with a concrete example.

Original languageEnglish (US)
Title of host publicationSIGMETRICS'07 - Proceedings of the 2007 International Conference on Measurement and Modeling of Computer Systems
Pages157-168
Number of pages12
Edition1
DOIs
StatePublished - Nov 29 2007
EventSIGMETRICS'07 - 2007 International Conference on Measurement and Modeling of Computer Systems - San Diego, CA, United States
Duration: Jun 12 2007Jun 16 2007

Publication series

NamePerformance Evaluation Review
Number1
Volume35
ISSN (Print)0163-5999

Other

OtherSIGMETRICS'07 - 2007 International Conference on Measurement and Modeling of Computer Systems
CountryUnited States
CitySan Diego, CA
Period6/12/076/16/07

All Science Journal Classification (ASJC) codes

  • Software
  • Hardware and Architecture
  • Computer Networks and Communications

Keywords

  • Feature-rich data
  • Similarity search
  • Sketch

Fingerprint Dive into the research topics of 'Sizing sketches: A rank-based analysis for similarity search'. Together they form a unique fingerprint.

  • Cite this

    Wang, Z., Dong, W., Josephson, W., Lv, Q., Charikar, M., & Li, K. (2007). Sizing sketches: A rank-based analysis for similarity search. In SIGMETRICS'07 - Proceedings of the 2007 International Conference on Measurement and Modeling of Computer Systems (1 ed., pp. 157-168). (Performance Evaluation Review; Vol. 35, No. 1). https://doi.org/10.1145/1269899.1254900