An effective and efficient data cleaning technique in large databases

Ji Zhang, Han Liu

Research output: Chapter in Book/Report/Conference proceedingConference contribution

Abstract

In this paper, we will propose PC-Cleaner (PC stands for Partition Comparison), a novel technique for effective and efficient duplicate record detection in the large database collection. PC-Cleaner distinguishes itself from all of existing methods by using the notion of partition in duplicate detection. It first sorts the whole database and splits the sorted database into a number of record partitions. The Partition Comparison Graph (PCG) is then generated by performing fast partition pruning. Finally, duplicate records are effectively detected by using internal and external partition comparison based on the PCG. Four properties, used as heuristics, have been devised to achieve a remarkable efficiency improvement of the cleaner based on triangle inequity of record similarity. PC-Cleaner is insensitive to the key used to sort the database and can achieve a very good recall level that is comparable to that of the pair-wise record comparison method. PC-Cleaner is able to well solve the "Key Selection" problem and the "Low Recall" problem that the existing methods suffer.

Original languageEnglish (US)
Title of host publicationProceedings of the International Conference on Information and Knowledge Engineering, IKE'04
EditorsH.R. Arabnia
Pages501-504
Number of pages4
StatePublished - Dec 1 2004
EventProceedings of the International Conference on Information and Knowledge Engineering, IKE'04 - Las Vegas, NV, United States
Duration: Jun 21 2004Jun 24 2004

Publication series

NameProceedings of the International Conference on Information and Knowledge Engineering , IKE'04

Other

OtherProceedings of the International Conference on Information and Knowledge Engineering, IKE'04
CountryUnited States
CityLas Vegas, NV
Period6/21/046/24/04

All Science Journal Classification (ASJC) codes

  • Engineering(all)

Fingerprint Dive into the research topics of 'An effective and efficient data cleaning technique in large databases'. Together they form a unique fingerprint.

  • Cite this

    Zhang, J., & Liu, H. (2004). An effective and efficient data cleaning technique in large databases. In H. R. Arabnia (Ed.), Proceedings of the International Conference on Information and Knowledge Engineering, IKE'04 (pp. 501-504). (Proceedings of the International Conference on Information and Knowledge Engineering , IKE'04).