Hybrid regularisation and the (in)admissibility of ridge regression in infinite dimensional Hilbert spaces. (English) Zbl 1466.62275

Summary: We consider the problem of estimating the slope function in a functional regression with a scalar response and a functional covariate. This central problem of functional data analysis is well known to be ill-posed, thus requiring a regularised estimation procedure. The two most commonly used approaches are based on spectral truncation or Tikhonov regularisation of the empirical covariance operator. In principle, Tikhonov regularisation is the more canonical choice. Compared to spectral truncation, it is robust to eigenvalue ties, while it attains the optimal minimax rate of convergence in the mean squared sense, and not just in a concentration probability sense. In this paper, we show that, surprisingly, one can strictly improve upon the performance of the Tikhonov estimator in finite samples by means of a linear estimator, while retaining its stability and asymptotic properties by combining it with a form of spectral truncation. Specifically, we construct an estimator that additively decomposes the functional covariate by projecting it onto two orthogonal subspaces defined via functional PCA; it then applies Tikhonov regularisation to the one component, while leaving the other component unregularised. We prove that when the covariate is Gaussian, this hybrid estimator uniformly improves upon the MSE of the Tikhonov estimator in a non-asymptotic sense, effectively rendering it inadmissible. This domination is shown to also persist under discrete observation of the covariate function. The hybrid estimator is linear, straightforward to construct in practice, and with no computational overhead relative to the standard regularisation methods. By means of simulation, it is shown to furnish sizeable gains even for modest sample sizes.


62G05 Nonparametric estimation
60B11 Probability theory on linear topological spaces
62H25 Factor analysis and principal components; correspondence analysis
62J07 Ridge regression; shrinkage estimators (Lasso)


fda (R)
Full Text: DOI Euclid


[1] Alquier, P., Gautier, E. and Stoltz, G., eds. (2011). Inverse Problems and High-Dimensional Estimation. Lecture Notes in Statistics—Proceedings203. Heidelberg: Springer. Lecture notes from the “Stats in the Château” Summer School held in Jouy-en-Josas, August 31–September 4, 2009.
[2] Amini, A.A. and Wainwright, M.J. (2012). Sampled forms of functional PCA in reproducing kernel Hilbert spaces. Ann. Statist.40 2483–2510. · Zbl 1373.62289
[3] Bai, Z. and Saranadasa, H. (1996). Effect of high dimension: By an example of a two sample problem. Statist. Sinica6 311–329. · Zbl 0848.62030
[4] Cai, T.T. and Hall, P. (2006). Prediction in functional linear regression. Ann. Statist.34 2159–2179. · Zbl 1106.62036
[5] Cai, T.T. and Yuan, M. (2011). Optimal estimation of the mean function based on discretely sampled functional data: Phase transition. Ann. Statist.39 2330–2355. · Zbl 1231.62040
[6] Cardot, H., Ferraty, F. and Sarda, P. (2003). Spline estimators for the functional linear model. Statist. Sinica13 571–591. · Zbl 1050.62041
[7] Cardot, H. and Johannes, J. (2010). Thresholding projection estimators in functional linear models. J. Multivariate Anal.101 395–408. · Zbl 1178.62032
[8] Cardot, H., Mas, A. and Sarda, P. (2007). CLT in functional linear regression models. Probab. Theory Related Fields138 325–361. · Zbl 1113.60025
[9] Cardot, H. and Sarda, P. (2006). Linear regression models for functional data. In The Art of Semiparametrics. Contrib. Statist. 49–66. Heidelberg: Physica-Verlag/Springer. · Zbl 1271.62145
[10] Chakraborty, A. and Panaretos, V.M. (2019). Supplement to “Hybrid regularisation and the (in)admissibility of ridge regression in infinite dimensional Hilbert spaces”. DOI:10.3150/18-BEJ1041SUPP.
[11] Chen, S.X. and Qin, Y.-L. (2010). A two-sample test for high-dimensional data with applications to gene-set testing. Ann. Statist.38 808–835. · Zbl 1183.62095
[12] Comte, F. and Johannes, J. (2012). Adaptive functional linear regression. Ann. Statist.40 2765–2797. · Zbl 1373.62350
[13] Conway, J.B. (1978). Functions of One Complex Variable, 2nd ed. Graduate Texts in Mathematics11. New York: Springer.
[14] Crambes, C., Kneip, A. and Sarda, P. (2009). Smoothing splines estimators for functional linear regression. Ann. Statist.37 35–72. · Zbl 1169.62027
[15] Cuevas, A., Febrero, M. and Fraiman, R. (2002). Linear functional regression: The case of fixed design and functional response. Canad. J. Statist.30 285–300. · Zbl 1012.62039
[16] Ferraty, F. and Vieu, P. (2000). Dimension fractale et estimation de la régression dans des espaces vectoriels semi-normés. C. R. Acad. Sci. Paris Sér. I Math.330 139–142. · Zbl 0942.62045
[17] Grenander, U. (1981). Abstract Inference. Wiley Series in Probability and Mathematical Statistics. New York: Wiley. · Zbl 0505.62069
[18] Hall, P. and Horowitz, J.L. (2007). Methodology and convergence rates for functional linear regression. Ann. Statist.35 70–91. · Zbl 1114.62048
[19] Hall, P. and Hosseini-Nasab, M. (2006). On properties of functional principal components analysis. J. R. Stat. Soc. Ser. B. Stat. Methodol.68 109–126. · Zbl 1141.62048
[20] Hocking, R.R. (2003). Methods and Applications of Linear Models: Regression and the Analysis of Variance, 2nd ed. Wiley Series in Probability and Statistics. Hoboken, NJ: Wiley. · Zbl 1038.62059
[21] Hoerl, A.E. and Kennard, R.W. (1970). Ridge regression: Biased estimation for nonorthogonal problems. Technometrics12 55–67. DOI:10.2307/1267351. · Zbl 0202.17205
[22] Horváth, L. and Kokoszka, P. (2012). Inference for Functional Data with Applications. Springer Series in Statistics. New York: Springer.
[23] Hsing, T. and Eubank, R. (2015). Theoretical Foundations of Functional Data Analysis, with an Introduction to Linear Operators. Wiley Series in Probability and Statistics. Chichester: Wiley. · Zbl 1338.62009
[24] Jolliffe, I.T. (2002). Principal Component Analysis, 2nd ed. Springer Series in Statistics. New York: Springer. · Zbl 1011.62064
[25] Li, Y. and Hsing, T. (2007). On rates of convergence in functional linear regression. J. Multivariate Anal.98 1782–1804. · Zbl 1130.62035
[26] Lukas, M.A. (1993). Asymptotic optimality of generalized cross-validation for choosing the regularization parameter. Numer. Math.66 41–66. · Zbl 0791.65037
[27] Lukas, M.A. (2006). Robust generalized cross-validation for choosing the regularization parameter. Inverse Probl.22 1883–1902. · Zbl 1104.62032
[28] Marx, B.D. and Eilers, P.H. (1999). Generalized linear regression on sampled signals and curves: A P-spline approach. Technometrics41 1–13.
[29] Meister, A. (2009). Deconvolution Problems in Nonparametric Statistics. Lecture Notes in Statistics193. Berlin: Springer.
[30] Ramsay, J.O. and Dalzell, C.J. (1991). Some tools for functional data analysis. J. Roy. Statist. Soc. Ser. B53 539–572. With discussion and a reply by the authors. · Zbl 0800.62314
[31] Ramsay, J.O. and Silverman, B.W. (2005). Functional Data Analysis, 2nd ed. Springer Series in Statistics. New York: Springer. · Zbl 1079.62006
[32] Shao, P.Y.-S. and Strawderman, W.E. (1994). Improving on the James–Stein positive-part estimator. Ann. Statist.22 1517–1538. · Zbl 0820.62051
[33] Theobald, C.M. (1974). Generalizations of mean square error applied to ridge regression. J. Roy. Statist. Soc. Ser. B36 103–106. · Zbl 0282.62055
[34] Tikhonov, A.N. and Arsenin, V.Y. (1977). Solutions of Ill-Posed Problems. Scripta Series in Mathematics. Washington, DC: Wiley, New York: V. H. Winston & Sons. Translated from the Russian; preface by translation editor Fritz John.
[35] Utreras, F.I. (1987). On generalized cross-validation for multivariate smoothing spline functions. SIAM J. Sci. Statist. Comput.8 630–643. · Zbl 0622.65008
[36] Wahba, G. (1990). Spline Models for Observational Data. CBMS–NSF Regional Conference Series in Applied Mathematics59. Philadelphia, PA: Society for Industrial and Applied Mathematics (SIAM). · Zbl 0813.62001
[37] Yao, F., Müller, H.-G. and Wang, J.-L. (2005). Functional linear regression analysis for longitudinal data. Ann. Statist.33 2873–2903. · Zbl 1084.62096
[38] Yuan, M. and Cai, T.T. (2010). A reproducing kernel Hilbert space approach to functional linear regression. Ann. Statist.38 3412–3444. · Zbl 1204.62074
This reference list is based on information provided by the publisher or from digital mathematics libraries. Its items are heuristically matched to zbMATH identifiers and may contain data conversion errors. It attempts to reflect the references listed in the original paper as accurately as possible without claiming the completeness or perfect precision of the matching.