TY - GEN
T1 - Maximum entropy distribution estimation with generalized regularization
AU - Dudík, Miroslav
AU - Schapire, Robert E.
PY - 2006
Y1 - 2006
N2 - We present a unified and complete account of maximum entropy distribution estimation subject to constraints represented by convex potential functions or, alternatively, by convex regularization. We provide fully general performance guarantees and an algorithm with a complete convergence proof. As special cases, we can easily derive performance guarantees for many known regularization types, including ℓ1, ℓ2, ℓ2 2 and ℓ1+ ℓ22 style regularization. Furthermore, our general approach enables us to use information about the structure of the feature space or about sample selection bias to derive entirely new regularization functions with superior guarantees. We propose an algorithm solving a large and general subclass of generalized maxent problems, including all discussed in the paper, and prove its convergence. Our approach generalizes techniques based on information geometry and Bregman divergences as well as those based more directly on compactness.
AB - We present a unified and complete account of maximum entropy distribution estimation subject to constraints represented by convex potential functions or, alternatively, by convex regularization. We provide fully general performance guarantees and an algorithm with a complete convergence proof. As special cases, we can easily derive performance guarantees for many known regularization types, including ℓ1, ℓ2, ℓ2 2 and ℓ1+ ℓ22 style regularization. Furthermore, our general approach enables us to use information about the structure of the feature space or about sample selection bias to derive entirely new regularization functions with superior guarantees. We propose an algorithm solving a large and general subclass of generalized maxent problems, including all discussed in the paper, and prove its convergence. Our approach generalizes techniques based on information geometry and Bregman divergences as well as those based more directly on compactness.
UR - http://www.scopus.com/inward/record.url?scp=33746050365&partnerID=8YFLogxK
UR - http://www.scopus.com/inward/citedby.url?scp=33746050365&partnerID=8YFLogxK
U2 - 10.1007/11776420_12
DO - 10.1007/11776420_12
M3 - Conference contribution
AN - SCOPUS:33746050365
SN - 3540352945
SN - 9783540352945
T3 - Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)
SP - 123
EP - 138
BT - Learning Theory - 19th Annual Conference on Learning Theory, COLT 2006, Proceedings
PB - Springer Verlag
T2 - 19th Annual Conference on Learning Theory, COLT 2006
Y2 - 22 June 2006 through 25 June 2006
ER -