×

Smoothly adaptively centered ridge estimator. (English) Zbl 1493.62297

Summary: With a focus on linear models with smooth functional covariates, we propose a penalization framework (SACR) based on the nonzero centered ridge, where the center of the penalty is adaptively reweighted, starting from the ordinary ridge solution as the initial center. In particular, we introduce a convex formulation that jointly estimates the model’s coefficients and the weight function, with a roughness penalty on the center, and constraints on the weights in order to recover a possibly smooth and/or sparse solution. This allows for a non-iterative and continuous variable selection mechanism, as the weight function can either inflate or deflate the initial center, reducing the unwanted shrinkage on some of the coefficients. As empirical evidence of the interpretability and predictive power of our method, we provide a simulation study and two real world spectroscopy applications, with both classification and regression.

MSC:

62H12 Estimation in multivariate analysis
62F12 Asymptotic properties of parametric estimators

Software:

Scikit; Pyomo; Ipopt; ncvreg
PDFBibTeX XMLCite
Full Text: DOI arXiv

References:

[1] Ait-Saïdi, A.; Ferraty, F.; Kassa, R.; Vieu, P., Cross-validated estimations in the single-functional index model, Statistics, 42, 6, 475-494 (2008) · Zbl 1274.62519
[2] Aneiros, G.; Cao, R.; Fraiman, R.; Genest, C.; Vieu, P., Recent advances in functional data analysis and high-dimensional statistics, J. Multivariate Anal., 170, 3-9 (2019) · Zbl 1415.62043
[3] Aneiros, G.; Vieu, P., Partial linear modelling with multi-functional covariates, Comput. Statist., 30, 3, 647-671 (2015) · Zbl 1342.65016
[4] Aneiros, G.; Vieu, P., Sparse nonparametric model for regression with functional covariate, J. Nonparametr. Stat., 28, 4, 839-859 (2016) · Zbl 1348.62131
[5] Aneiros-Pérez, G.; Vieu, P., Semi-functional partial linear regression, Statist. Probab. Lett., 76, 11, 1102-1110 (2006) · Zbl 1090.62036
[6] Berrendero, J. R.; Bueno-Larraz, B.; Cuevas, A., An RKHS model for variable selection in functional linear regression, J. Multivariate Anal., 170, 25-45 (2019) · Zbl 1416.62331
[7] Bilgrau, A. E.; Peeters, C. F.; Eriksen, P. S.; Boegsted, M.; van Wieringen, W. N., Targeted fused ridge estimation of inverse covariance matrices from multiple high-dimensional data classes, J. Mach. Learn. Res., 26, 1, 1-52 (2020) · Zbl 1499.62177
[8] Breheny, P., ncvreg: Regularization paths for SCAD and MCP penalized regression models (2020), R package version 3.12.0, https://CRAN.R-project.org/package=ncvreg
[9] Breheny, P.; Huang, J., Coordinate descent algorithms for nonconvex penalized regression, with applications to biological feature selection, Ann. Statist., 5, 1, 232-253 (2011) · Zbl 1220.62095
[10] Breiman, L., Better subset regression using the nonnegative garrote, Technometrics, 37, 4, 373-384 (1995) · Zbl 0862.62059
[11] Breiman, L., Heuristics of instability and stabilization in model selection, Ann. Statist., 24, 6, 2350-2383 (1996) · Zbl 0867.62055
[12] Burba, F.; Ferraty, F.; Vieu, P., k-Nearest Neighbour method in functional nonparametric regression, J. Nonparametr. Stat., 21, 4, 453-469 (2009) · Zbl 1161.62017
[13] Cai, T. T.; Hall, P., Prediction in functional linear regression, Ann. Statist., 34, 5, 2159-2179 (2006) · Zbl 1106.62036
[14] Cardot, H.; Crambes, C.; Kneip, A.; Sarda, P., Smoothing splines estimators in functional linear regression with errors-in-variables, Comput. Statist. Data Anal., 51, 10, 4832-4848 (2007) · Zbl 1162.62333
[15] Cardot, H.; Ferraty, F.; Sarda, P., Spline estimators for the functional linear model, Statist. Sinica, 13, 3, 571-591 (2003) · Zbl 1050.62041
[16] Crambes, C.; Kneip, A.; Sarda, P., Smoothing splines estimators for functional linear regression, Ann. Statist., 37, 1, 35-72 (2009) · Zbl 1169.62027
[17] Cuevas, A., A partial overview of the theory of statistics with functional data, J. Statist. Plann. Inference, 147, 1-23 (2014) · Zbl 1278.62012
[18] Dai, L.; Chen, K.; Sun, Z.; Liu, Z.; Li, G., Broken adaptive ridge regression and its asymptotic properties, J. Multivariate Anal., 168, 1, 334-351 (2018) · Zbl 1401.62108
[19] Fan, J.; Li, R., Variable selection via nonconcave penalized likelihood and its oracle properties, J. Amer. Statist. Assoc., 96, 456, 1348-1360 (2001) · Zbl 1073.62547
[20] Ferraty, F.; Goia, A.; Salinelli, E.; Vieu, P., Functional projection pursuit regression, Test, 22, 2, 293-320 (2013) · Zbl 1367.62117
[21] Ferraty, F.; Hall, P.; Vieu, P., Most-predictive design points for functional data predictors, Biometrika, 97, 4, 807-824 (2010) · Zbl 1204.62064
[22] Ferraty, F.; Vieu, P., Nonparametric Functional Data Analysis: Theory and Practice (2006), Springer: Springer New York, NY, USA · Zbl 1119.62046
[23] Frank, I. E.; Friedman, J. H., A statistical view of some chemometrics regression tools, Technometrics, 35, 2, 109-135 (1993) · Zbl 0775.62288
[24] Fu, W.; Knight, K., Asymptotics for lasso-type estimators, Ann. Statist., 28, 5, 1356-1378 (2000) · Zbl 1105.62357
[25] Galeano, P.; Joseph, E.; Lillo, R. E., The Mahalanobis distance for functional data with applications to classification, Technometrics, 57, 2, 281-291 (2015)
[26] Goia, A.; Vieu, P., A partitioned single functional index model, Comput. Statist., 30, 3, 673-692 (2015) · Zbl 1342.65034
[27] Goia, A.; Vieu, P., An introduction to recent advances in high/infinite dimensional statistics, J. Multivariate Anal., 146, 1-6 (2016) · Zbl 1384.00073
[28] Gondzio, J., Interior point methods 25 years later, European J. Oper. Res., 218, 3, 587-601 (2012) · Zbl 1244.90007
[29] Grandvalet, Y., Least absolute shrinkage is equivalent to quadratic penalization, (Niklasson, L.; Bodén, M.; Ziemke, T., ICANN 98. Perspectives in Neural Computing (1998), Springer: Springer London), 201-206
[30] Grandvalet, Y.; Canu, S., Outcomes of the equivalence of adaptive ridge with least absolute shrinkage, (Kearns, M. J.; Solla, S. A.; Cohn, D. A., Advances in Neural Information Processing Systems 11 (1999), MIT Press), 445-451
[31] Hart, W. E.; Laird, C. D.; Watson, J.-P.; Woodruff, D. L.; Hackebeil, G. A.; Nicholson, B. L.; Siirola, J. D., Pyomo-Optimization Modeling in Python (2017), Springer Science & Business Media: Springer Science & Business Media New York, NY, USA · Zbl 1370.90003
[32] Hastie, T.; Tibshirani, R.; Wainwright, M., Statistical Learning with Sparsity: The Lasso and Generalizations (2015), CRC Press: CRC Press New York, NY, USA · Zbl 1319.68003
[33] Hebiri, M.; van de Geer, S., The smooth-lasso and other \(\ell_1+ \ell_2\)-penalized methods, Electron. J. Stat., 5, 1, 1184-1226 (2011) · Zbl 1274.62443
[34] Hoerl, A. E.; Kennard, R. W., Ridge regression: Biased estimation for nonorthogonal problems, Technometrics, 12, 1, 55-67 (1970) · Zbl 0202.17205
[35] James, G., Sparseness and functional data analysis, (Ferraty, F.; Romain, Y., The Oxford Handbook of Functional Data Analysis (2011), Oxford University Press: Oxford University Press New York), 298-323
[36] James, G. M.; Wang, J.; Zhu, J., Functional linear regression that’s interpretable, Ann. Statist., 37, 5A, 2083-2108 (2009) · Zbl 1171.62041
[37] Kara, L.-Z.; Laksaci, A.; Rachdi, M.; Vieu, P., Data-driven kNN estimation in nonparametric functional data analysis, J. Multivariate Anal., 153, 176-188 (2017) · Zbl 1351.62084
[38] Kim, H.; Kim, H., Functional logistic regression with fused lasso penalty, J. Stat. Comput. Simul., 88, 15, 2982-2999 (2018) · Zbl 07192699
[39] Kneip, A.; Poß, D.; Sarda, P., Functional linear regression with points of impact, Ann. Statist., 44, 1, 1-30 (2016) · Zbl 1331.62233
[40] Lee, E. R.; Park, B. U., Sparse estimation in functional linear regression, J. Multivariate Anal., 105, 1, 1-17 (2012) · Zbl 1236.62032
[41] Lian, H., Functional partial linear model, J. Nonparametr. Stat., 23, 1, 115-128 (2011) · Zbl 1359.62157
[42] Liebl, D.; Rameseder, S.; Rust, C., Improving estimation in functional linear regression with points of impact: Insights into google adwords, J. Comput. Graph. Statist., 29, 4, 814-826 (2020) · Zbl 07500359
[43] Lindquist, M. A.; McKeague, I. W., Logistic regression with Brownian-like predictors, J. Amer. Statist. Assoc., 104, 488, 1575-1585 (2009) · Zbl 1205.62125
[44] Ling, N.; Aneiros, G.; Vieu, P., kNN estimation in functional partial linear modeling, Statist. Papers, 61, 423-444 (2020) · Zbl 1437.62701
[45] Marx, B. D.; Eilers, P. H., Generalized linear regression on sampled signals and curves: A P-spline approach, Technometrics, 41, 1, 1-13 (1999)
[46] Matsui, H.; Konishi, S., Variable selection for functional regression models via the \(L_1\) regularization, Comput. Statist. Data Anal., 55, 12, 3304-3310 (2011) · Zbl 1271.62140
[47] McKeague, I. W.; Sen, B., Fractals with point impact in functional linear regression, Ann. Statist., 38, 4, 2559-2586 (2010) · Zbl 1196.62116
[48] Meinshausen, N., Relaxed lasso, Comput. Statist. Data Anal., 52, 1, 374-393 (2007) · Zbl 1452.62522
[49] Meinshausen, N.; Bühlmann, P., High-dimensional graphs and variable selection with the lasso, Ann. Statist., 34, 3, 1436-1462 (2006) · Zbl 1113.62082
[50] Novo, S.; Aneiros, G.; Vieu, P., Automatic and location-adaptive estimation in functional single-index regression, J. Nonparametr. Stat., 31, 2, 364-392 (2019) · Zbl 1418.62164
[51] Novo, S.; Aneiros, G.; Vieu, P., Sparse semiparametric regression when predictors are mixture of functional and high-dimensional variables, TEST (2020)
[52] Pedregosa, F.; Varoquaux, G.; Gramfort, A.; Michel, V.; Thirion, B.; Grisel, O.; Blondel, M.; Prettenhofer, P.; Weiss, R.; Dubourg, V.; Vanderplas, J.; Passos, A.; Cournapeau, D.; Brucher, M.; Perrot, M.; Duchesnay, É., Scikit-learn: Machine learning in Python, J. Mach. Learn. Res., 12, Oct, 2825-2830 (2011) · Zbl 1280.68189
[53] Poß, D.; Liebl, D.; Kneip, A.; Eisenbarth, H.; Wager, T. D.; Feldman Barrett, L., Superconsistent estimation of points of impact in non-parametric regression with functional predictors, J. R. Stat. Soc. Ser. B Stat. Methodol., 82, 4, 1115-1140 (2020) · Zbl 07554786
[54] Price, B. S.; Geyer, C. J.; Rothman, A. J., Ridge fusion in statistical learning, J. Comput. Graph. Statist., 24, 2, 439-454 (2015)
[55] Shin, H., Partial functional linear regression, J. Statist. Plann. Inference, 139, 10, 3405-3418 (2009) · Zbl 1168.62358
[56] Swindel, B. F., Good ridge estimators based on prior information, Comm. Statist. Theory Methods, 5, 11, 1065-1075 (1976) · Zbl 0342.62035
[57] Tibshirani, R., Regression shrinkage and selection via the lasso, J. R. Stat. Soc. Ser. B Stat. Methodol., 58, 1, 267-288 (1996) · Zbl 0850.62538
[58] Wächter, A.; Biegler, L. T., On the implementation of an interior-point filter line-search algorithm for large-scale nonlinear programming, Math. Program., 106, 1, 25-37 (2006) · Zbl 1134.90542
[59] Wang, G.; Feng, X.; Chen, M., Functional partial linear single-index model, Scand. J. Stat., 43, 1, 261-274 (2016) · Zbl 1371.62037
[60] van Wieringen, W. N., The generalized ridge estimator of the inverse covariance matrix, J. Comput. Graph. Statist., 28, 4, 932-942 (2019) · Zbl 07499037
[61] van Wieringen, W. N.; Stam, K. A.; Peeters, C. F.; van de Wiel, M. A., Updating of the Gaussian graphical model through targeted penalized estimation, J. Multivariate Anal., 178 (2020) · Zbl 1440.62211
[62] Yuan, M.; Cai, T. T., A reproducing kernel Hilbert space approach to functional linear regression, Ann. Statist., 38, 6, 3412-3444 (2010) · Zbl 1204.62074
[63] Yuan, M.; Lin, Y., On the non-negative garrotte estimator, J. R. Stat. Soc. Ser. B Stat. Methodol., 69, 2, 143-161 (2007) · Zbl 1120.62052
[64] Zhao, P.; Rocha, G.; Yu, B., Nearly unbiased variable selection under minimax concave penalty, Ann. Statist., 38, 2, 894-942 (2010) · Zbl 1183.62120
[65] Zhao, P.; Yu, B., On model selection consistency of lasso, J. Mach. Learn. Res., 7, Nov, 2541-2563 (2006) · Zbl 1222.62008
[66] Zhu, H.; Zhang, R.; Yu, Z.; Lian, H.; Liu, Y., Estimation and testing for partially functional linear errors-in-variables models, J. Multivariate Anal., 170, 296-314 (2019) · Zbl 1420.62186
[67] Zou, H., The adaptive lasso and its oracle properties, J. Amer. Statist. Assoc., 101, 476, 1418-1429 (2006) · Zbl 1171.62326
[68] Zou, H.; Hastie, T., Regularization and variable selection via the elastic net, J. R. Stat. Soc. Ser. B Stat. Methodol., 67, 2, 301-320 (2005) · Zbl 1069.62054
[69] Zou, H.; Li, R., One-step sparse estimates in nonconcave penalized likelihood models, Ann. Statist., 36, 4, 1509-1533 (2008) · Zbl 1142.62027
This reference list is based on information provided by the publisher or from digital mathematics libraries. Its items are heuristically matched to zbMATH identifiers and may contain data conversion errors. In some cases that data have been complemented/enhanced by data from zbMATH Open. This attempts to reflect the references listed in the original paper as accurately as possible without claiming completeness or a perfect matching.