Statistical properties of the method of regularization with periodic Gaussian reproducing kernel. (English) Zbl 1045.62026

Lin, Yi; Brown, Lawrence D.

Ann. Stat. 32, No. 4, 1723-1743 (2004).

Summary: The method of regularization with Gaussian reproducing kernels is popular in the machine learning literature and successful in many practical applications. We consider the periodic version of Gaussian kernel regularization. We show, in the white noise model setting, that in function spaces of very smooth functions, such as the infinite-order Sobolev space and the space of analytic functions, the method under consideration is asymptotically minimax; in finite-order Sobolev spaces the method is rate optimal, and the efficiency in terms of constants when compared with the minimax estimator is reasonably high. The smoothing parameters in the periodic Gaussian regularization can be chosen adaptively without loss of asymptotic efficiency. The results derived in this paper give a partial explanation of the success of the Gaussian reproducing kernels in practice. Simulations are carried out to study the finite sample properties of the periodic Gaussian regularization.

MSC:

62G07	Density estimation
62G20	Asymptotic properties of nonparametric inference
62G08	Nonparametric regression and quantile regression
62C20	Minimax procedures in statistical decision theory

Keywords:

asymptotic minimax risk; Gaussian reproducing kernel; rate of convergence; Sobolev spaces; white noise model

Software:

gss

References:

[1]	Brown, L. D. and Low, M. G. (1996). Asymptotic equivalence of nonparametric regression and white noise. Ann. Statist. 24 2384–2398. · Zbl 0867.62022 · doi:10.1214/aos/1032181159
[2]	Brown, L. D. and Zhao, L. H. (2002). Direct asymptotic equivalence of nonparametric regression and the infinite dimensional location problem. Technical report. Available at http://ljsavage.wharton.upenn.edu/ lzhao/papers/.
[3]	Carter, C. K., Eagleson, G. K. and Silverman, B. W. (1992). Comparison of the Reinsch and Speckman splines. Biometrika 79 81–91. · Zbl 0753.62023 · doi:10.1093/biomet/79.1.81
[4]	Cavalier, L., Golubev, G. K., Picard, D. and Tsybakov, A. B. (2002). Oracle inequalities for inverse problems. Ann. Statist. 30 843–874. · Zbl 1029.62032 · doi:10.1214/aos/1028674843
[5]	Evgeniou, T., Pontil, M. and Poggio, T. (2000). Regularization networks and support vector machines. Adv. Comput. Math. 13 1–50. · Zbl 0939.68098 · doi:10.1023/A:1018946025316
[6]	Girosi, F., Jones, M. and Poggio, T. (1993). Priors, stabilizers and basis functions: From regularization to radial, tensor and additive splines. M.I.T. Artificial Intelligence Laboratory Memo 1430, C.B.C.I. Paper 75.
[7]	Golubev, G. and Nussbaum, M. (1998). Asymptotic equivalence of spectral density and regression estimation. Technical report, Weierstrass Institute for Applied Analysis and Stochastics, Berlin.
[8]	Grama, I. and Nussbaum, M. (1997). Asymptotic equivalence for nonparametric generalized linear models. Technical report, Weierstrass Institute for Applied Analysis and Stochastics, Berlin. · Zbl 0953.62039 · doi:10.1007/s004400050166
[9]	Gu, C. (2002). Smoothing Spline ANOVA Models . Springer, New York. · Zbl 1051.62034
[10]	Johnstone, I. M. (1998). Function estimation in Gaussian noise: Sequence models. Draft of a monograph.
[11]	Kimeldorf, G. S. and Wahba, G. (1971). Some results on Tchebycheffian spline functions. J. Math. Anal. Appl. 33 82–95. · Zbl 0201.39702 · doi:10.1016/0022-247X(71)90184-3
[12]	Kneip, A. (1994). Ordered linear smoothers. Ann. Statist. 22 835–866. JSTOR: · Zbl 0815.62022 · doi:10.1214/aos/1176325498
[13]	Li, K.-C. (1986). Asymptotic optimality of \(C_L\) and generalized cross-validation in ridge regression with application to spline smoothing. Ann. Statist. 14 1101–1112. JSTOR: · Zbl 0629.62043 · doi:10.1214/aos/1176350052
[14]	Li, K.-C. (1987). Asymptotic optimality for \(C_p, C_L\), cross-validation and generalized cross-validation: Discrete index set. Ann. Statist. 15 958–975. JSTOR: · Zbl 0653.62037 · doi:10.1214/aos/1176350486
[15]	Mallows, C. L. (1973). Some comments on \(C_p\). Technometrics 15 661–675. · Zbl 0269.62061 · doi:10.2307/1267380
[16]	Nussbaum, M. (1996). Asymptotic equivalence of density estimation and Gaussian white noise. Ann. Statist. 24 2399–2430. · Zbl 0867.62035 · doi:10.1214/aos/1032181160
[17]	Pinsker, M. (1980). Optimal filtering of square integrable signals in Gaussian white noise. Problems Inform. Transmission 16 120–133. · Zbl 0452.94003
[18]	Schölkopf, B., Herbrich, R. and Smola, A. J. (2001). A generalized representer theorem. In Proc. Fourteenth Annual Conference on Computational Learning Theory. Lecture Notes in Comput. Sci. 2111 416–426. Springer, London. · Zbl 0992.68088
[19]	Smola, A. J., Schölkopf, B. and Müller, K. (1998). The connection between regularization operators and support vector kernels. Neural Networks 11 637–649.
[20]	Wahba, G. (1990). Spline Models for Observational Data . SIAM, Philadelphia. · Zbl 0813.62001
[21]	Wahba, G. (1999). Support vector machines, reproducing kernel Hilbert spaces and randomized GACV. In Advances in Kernel Methods : Support Vector Learning (B. Schölkopf, C. J. Burges and A. J. Smola, eds.) 69–88. MIT Press, Cambridge, MA.
[22]	Williamson, R. C., Smola, A. J. and Schölkopf, B. (2001). Generalization performance of regularization networks and support vector machines via entropy numbers of compact operators. IEEE Trans. Inform. Theory 47 2516–2532. · Zbl 1008.62507 · doi:10.1109/18.945262