Entropy and inference, revisited

Ilya Nemenman; Fariel Shafee; William Bialek

Entropy and inference, revisited

Ilya Nemenman, Fariel Shafee, William Bialek

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution

Abstract

We study properties of popular near-uniform (Dirichlet) priors for learning undersampled probability distributions on discrete nonmetric spaces and show that they lead to disastrous results. However, an Occam-style phase space argument expands the priors into their infinite mixture and resolves most of the observed problems. This leads to a surprisingly good estimator of entropies of discrete distributions.

Original language	English (US)
Title of host publication	Advances in Neural Information Processing Systems 14 - Proceedings of the 2001 Conference, NIPS 2001
Publisher	Neural information processing systems foundation
ISBN (Print)	0262042088, 9780262042086
State	Published - 2002
Event	15th Annual Neural Information Processing Systems Conference, NIPS 2001 - Vancouver, BC, Canada Duration: Dec 3 2001 → Dec 8 2001

Publication series

Name	Advances in Neural Information Processing Systems
ISSN (Print)	1049-5258

Other

Other	15th Annual Neural Information Processing Systems Conference, NIPS 2001
Country/Territory	Canada
City	Vancouver, BC
Period	12/3/01 → 12/8/01

All Science Journal Classification (ASJC) codes

Computer Networks and Communications
Information Systems
Signal Processing

Cite this

@inproceedings{737cdeb7b24048839f69fe8485d362c3,

title = "Entropy and inference, revisited",

abstract = "We study properties of popular near-uniform (Dirichlet) priors for learning undersampled probability distributions on discrete nonmetric spaces and show that they lead to disastrous results. However, an Occam-style phase space argument expands the priors into their infinite mixture and resolves most of the observed problems. This leads to a surprisingly good estimator of entropies of discrete distributions.",

author = "Ilya Nemenman and Fariel Shafee and William Bialek",

year = "2002",

language = "English (US)",

isbn = "0262042088",

series = "Advances in Neural Information Processing Systems",

publisher = "Neural information processing systems foundation",

booktitle = "Advances in Neural Information Processing Systems 14 - Proceedings of the 2001 Conference, NIPS 2001",

note = "15th Annual Neural Information Processing Systems Conference, NIPS 2001 ; Conference date: 03-12-2001 Through 08-12-2001",

}

Nemenman, I, Shafee, F & Bialek, W 2002, Entropy and inference, revisited. in Advances in Neural Information Processing Systems 14 - Proceedings of the 2001 Conference, NIPS 2001. Advances in Neural Information Processing Systems, Neural information processing systems foundation, 15th Annual Neural Information Processing Systems Conference, NIPS 2001, Vancouver, BC, Canada, 12/3/01.

Entropy and inference, revisited. / Nemenman, Ilya; Shafee, Fariel; Bialek, William.
Advances in Neural Information Processing Systems 14 - Proceedings of the 2001 Conference, NIPS 2001. Neural information processing systems foundation, 2002. (Advances in Neural Information Processing Systems).

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution

TY - GEN

T1 - Entropy and inference, revisited

AU - Nemenman, Ilya

AU - Shafee, Fariel

AU - Bialek, William

PY - 2002

Y1 - 2002

N2 - We study properties of popular near-uniform (Dirichlet) priors for learning undersampled probability distributions on discrete nonmetric spaces and show that they lead to disastrous results. However, an Occam-style phase space argument expands the priors into their infinite mixture and resolves most of the observed problems. This leads to a surprisingly good estimator of entropies of discrete distributions.

AB - We study properties of popular near-uniform (Dirichlet) priors for learning undersampled probability distributions on discrete nonmetric spaces and show that they lead to disastrous results. However, an Occam-style phase space argument expands the priors into their infinite mixture and resolves most of the observed problems. This leads to a surprisingly good estimator of entropies of discrete distributions.

UR - http://www.scopus.com/inward/record.url?scp=84898931392&partnerID=8YFLogxK

UR - http://www.scopus.com/inward/citedby.url?scp=84898931392&partnerID=8YFLogxK

M3 - Conference contribution

AN - SCOPUS:84898931392

SN - 0262042088

SN - 9780262042086

T3 - Advances in Neural Information Processing Systems

BT - Advances in Neural Information Processing Systems 14 - Proceedings of the 2001 Conference, NIPS 2001

PB - Neural information processing systems foundation

T2 - 15th Annual Neural Information Processing Systems Conference, NIPS 2001

Y2 - 3 December 2001 through 8 December 2001

ER -

Entropy and inference, revisited

Abstract

Publication series

Other

All Science Journal Classification (ASJC) codes

Other files and links

Fingerprint

Cite this