Divergence estimation of continuous distributions based on data-dependent partitions

Qing Wang, Sanjeev R. Kulkarni, Sergio Verdú

Research output: Contribution to journalArticlepeer-review

139 Scopus citations


We present a universal estimator of the divergence D(P ∥ Q) for two arbitrary continuous distributions P and Q satisfying certain regularity conditions. This algorithm, which observes independent and identically distributed (i.i.d.) samples from both P and Q, is based on the estimation of the Radon-Nikodym derivative dP/dQ via a data-dependent partition of the observation space. Strong convergence of this estimator is proved with an empirically equivalent segmentation of the space. This basic estimator is further improved by adaptive partitioning schemes and by bias correction. The application of the algorithms to data with memory is also investigated. In the simulations, we compare our estimators with the direct plug-in estimator and estimators based on other partitioning approaches. Experimental results show that our methods achieve the best convergence performance in most of the tested cases.

Original languageEnglish (US)
Pages (from-to)3064-3074
Number of pages11
JournalIEEE Transactions on Information Theory
Issue number9
StatePublished - Sep 2005

All Science Journal Classification (ASJC) codes

  • Information Systems
  • Computer Science Applications
  • Library and Information Sciences


  • Bias correction
  • Data-dependent partition
  • Divergence
  • Radon-Nikodym derivative
  • Stationary and ergodic data
  • Universal estimation of information measures


Dive into the research topics of 'Divergence estimation of continuous distributions based on data-dependent partitions'. Together they form a unique fingerprint.

Cite this