×
Zhang, Tong

From \(\varepsilon\)-entropy to KL-entropy: analysis of minimum information complexity density estimation. (English) Zbl 1106.62005

Ann. Stat. 34, No. 5, 2180-2210 (2006).
Summary: We consider an extension of \(\varepsilon\)-entropy to a KL-divergence based complexity measures for randomized density estimation methods. Based on this extension, we develop a general information-theoretical inequality that measures the statistical complexity of some deterministic and randomized density estimators. Consequences of the new inequality will be presented.
In particular, we show that this technique can lead to improvements of some classical results concerning the convergence of minimum description length and Bayesian posterior distributions. Moreover, we are able to derive clean finite-sample convergence bounds that are not obtainable using previous approaches.

Cited in 37 Documents

MSC:

62B10 Statistical aspects of information-theoretic topics
62F15 Bayesian inference
62G07 Density estimation
62C10 Bayesian problems; characterization of Bayes procedures

Keywords:

minimum description length; Bayesian posterior distributions

Software:

ESTIMA
Cite Review PDF
Full Text: DOI arXiv

References:

[1] Barron, A. and Cover, T. (1991). Minimum complexity density estimation. IEEE Trans. Inform. Theory 37 1034–1054. · Zbl 0743.62003 · doi:10.1109/18.86996
[2] Barron, A., Schervish, M. J. and Wasserman, L. (1999). The consistency of posterior distributions in nonparametric problems. Ann. Statist. 27 536–561. · Zbl 0980.62039 · doi:10.1214/aos/1018031206
[3] Catoni, O. (2004). A PAC-Bayesian approach to adaptive classification. Available at www.proba.jussieu.fr/users/catoni/homepage/classif.pdf.
[4] Ghosal, S., Ghosh, J. K. and van der Vaart, A. W. (2000). Convergence rates of posterior distributions. Ann. Statist. 28 500–531. · Zbl 1105.62315 · doi:10.1214/aos/1016218228
[5] Le Cam, L. (1973). Convergence of estimates under dimensionality restrictions. Ann. Statist. 1 38–53. · Zbl 0255.62006 · doi:10.1214/aos/1193342380
[6] Le Cam, L. (1986). Asymptotic Methods in Statistical Decision Theory . Springer, New York. · Zbl 0605.62002
[7] Li, J. (1999). Estimation of mixture models. Ph.D. dissertation, Dept. Statistics, Yale Univ.
[8] Meir, R. and Zhang, T. (2003). Generalization error bounds for Bayesian mixture algorithms. J. Mach. Learn. Res. 4 839–860. · Zbl 1083.68096 · doi:10.1162/1532443041424300
[9] Rényi, A. (1961). On measures of entropy and information. Proc. Fourth Berkeley Symp. Math. Statist. Probab. 1 547–561. Univ. California Press, Berkeley. · Zbl 0106.33001
[10] Rissanen, J. (1989). Stochastic Complexity in Statistical Inquiry . World Scientific, Singapore. · Zbl 0800.68508
[11] Seeger, M. (2002). PAC-Bayesian generalization error bounds for Gaussian process classification. J. Mach. Learn. Res. 3 233–269. · Zbl 1088.68745 · doi:10.1162/153244303765208386
[12] Shen, X. and Wasserman, L. (2001). Rates of convergence of posterior distributions. Ann. Statist. 29 687–714. · Zbl 1041.62022 · doi:10.1214/aos/1009210686
[13] van de Geer, S. (2000). Empirical Processes in \(M\)-Estimation . Cambridge Univ. Press. · Zbl 0953.62049
[14] van der Vaart, A. W. and Wellner, J. A. (1996). Weak Convergence and Empirical Processes. With Applications to Statistics . Springer, New York. · Zbl 0862.60002
[15] Walker, S. and Hjort, N. (2001). On Bayesian consistency. J. R. Stat. Soc. Ser. B Stat. Methodol. 63 811–821. JSTOR: · Zbl 0987.62021 · doi:10.1111/1467-9868.00314
[16] Yang, Y. and Barron, A. (1999). Information-theoretic determination of minimax rates of convergence. Ann. Statist. 27 1564–1599. · Zbl 0978.62008 · doi:10.1214/aos/1017939142
[17] Zhang, T. (1999). Theoretical analysis of a class of randomized regularization methods. In Proc. Twelfth Annual Conference on Computational Learning Theory 156–163. ACM Press, New York. · doi:10.1145/307400.307433
[18] Zhang, T. (2004). Learning bounds for a generalized family of Bayesian posterior distributions. In Advances in Neural Information Processing Systems 16 (S. Thrun, L. K. Saul and B. Schölkopf, eds.) 1149–1156. MIT Press, Cambridge, MA.
This reference list is based on information provided by the publisher or from digital mathematics libraries. Its items are heuristically matched to zbMATH identifiers and may contain data conversion errors. In some cases that data have been complemented/enhanced by data from zbMATH Open. This attempts to reflect the references listed in the original paper as accurately as possible without claiming completeness or a perfect matching.