×

zbMATH — the first resource for mathematics

Additive model selection. (English) Zbl 1359.62178
Summary: We study sparse high dimensional additive model fitting via penalization with sparsity-smoothness penalties. We review several existing algorithms that have been developed for this problem in the recent literature, highlighting the connections between them, and present some computationally efficient algorithms for fitting such models. Furthermore, using reasonable assumptions and exploiting recent results on group LASSO-like procedures, we take advantage of several oracle results which yield asymptotic optimality of estimators for high-dimensional but sparse additive models. Finally, variable selection procedures are compared with some high-dimensional testing procedures available in the literature for testing the presence of additive components.
MSC:
62H12 Estimation in multivariate analysis
62G08 Nonparametric regression and quantile regression
62G10 Nonparametric hypothesis testing
PDF BibTeX XML Cite
Full Text: DOI
References:
[1] Adams RA (1975) Sobolev spaces. Academic Press, New York · Zbl 0314.46030
[2] Amato, U; Antoniadis, A; Pensky, M, Wavelet kernel penalized estimation for nonequispaced design regression, Stat Comput, 16, 37-55, (2006)
[3] Antoniadis, A; Fan, J, Regularization of wavelets approximations (with discussion), J Am Stat Assoc, 96, 939-967, (2001) · Zbl 1072.62561
[4] Antoniadis, A; Gijbels, I; Verhasselt, A, Variable selection in additive models using P-splines, Technometrics, 54, 425-438, (2012)
[5] Avalos, M; Grandvalet, Y; Ambroise, YC, Parsimonious additive models, Comput Stat Data Anal, 51, 2851-2870, (2007) · Zbl 1161.62354
[6] Bach FR (2008) Bolasso: model consistent lasso estimation through the bootstrap. In: Proceedings of the 25th international conference on machine learning, Helsinki, Finland, pp 33-40 · Zbl 1142.62044
[7] Bazerque, JA; Mateos, G; Giannakis, GB, Group-lasso on splines for spectrum cartography, IEEE Trans Signal Process, 59, 4648-4663, (2011) · Zbl 1391.62136
[8] Belloni, A; Chernozhukov, V, Least squares after model selection in high-dimensional sparse models, Bernoulli, 19, 521-547, (2013) · Zbl 06168762
[9] Benjamini, Y; Hochberg, Y, Controlling the false discovery rate: a practical powerful approach to multiple hypothesis testing, J R Stat Soc B, 57, 289-300, (1995) · Zbl 0809.62014
[10] Benjamini, Y; Yekutieli, D, The control of false discovery rate under dependency, Ann Stat, 29, 1165-1188, (2001) · Zbl 1041.62061
[11] Bhatti, MI; Bracken, P, Solutions of differential equations in a Bernstein polynomial basis, J Appl Math Comput, 205, 272-280, (2007) · Zbl 1118.65087
[12] Bickel, PJ; Ritov, Y; Tsybakov, AB, Simultaneous analysis of lasso and Dantzig selector, Ann Stat, 37, 1705-1732, (2009) · Zbl 1173.62022
[13] Breheny, P; Huang, J, Penalized methods for bi-level variable selection, Stat Its Interface, 2, 369-380, (2009) · Zbl 1245.62034
[14] Breiman, L; Friedman, JH, Estimating optimal transformations for multiple regression and correlation, J Am Stat Assoc, 80, 580-598, (1985) · Zbl 0594.62044
[15] Breiman, L, Better subset regression using the nonnegative garrote, Technometrics, 37, 373-384, (1995) · Zbl 0862.62059
[16] Bühlmann, P, Statistical significance in high-dimensional linear models, Bernoulli, 19, 1212-1242, (2013) · Zbl 1273.62173
[17] Bühlmann P, van de Geer S (2011) Statistics for high-dimensional data: methods, theory and applications. Springer, Heidelberg · Zbl 1273.62015
[18] Bunea, F; Wegkamp, M; Auguste, A, Consistent variable selection in high dimensional regression via multiple testing, J Stat Plan Inf, 136, 4349-4364, (2006) · Zbl 1112.62062
[19] Bunea, F; Tsybakov, AB; Wegkamp, MH, Sparsity oracle inequalities for the lasso, Electron J Stat, 1, 169-194, (2007) · Zbl 1146.62028
[20] Cantoni, E; Mills Flemming, J; Ronchetti, E, Variable selection in additive models by nonnegative garrote, Stat Model, 11, 237-252, (2011)
[21] Casella, G; Moreno, E, Objective Bayesian variable selection, J Am Stat Assoc, 101, 157-167, (2006) · Zbl 1118.62313
[22] Cui, X; Peng, H; Wen, S; Zhu, L, Component selection in the additive regression model, Scand J Stat, 40, 491-510, (2013) · Zbl 1364.62091
[23] de Boor C (2001) A practical guide to splines. Springer, Applied Mathematical Sciences, vol 27, Berlin (2001) · Zbl 0987.65015
[24] Eilers, PHC; Marx, BD, Flexible smoothing with B-splines and penalties, Stat Sci, 11, 89-121, (1996) · Zbl 0955.62562
[25] Fan, J; Li, R, Variable selection via nonconcave penalized likelihood and its oracle properties, J Am Stat Assoc, 96, 1348-1360, (2001) · Zbl 1073.62547
[26] Fan, J; Jiang, J, Nonparametric inference for additive models, J Am Stat Assoc, 100, 890-907, (2005) · Zbl 1117.62328
[27] Fan, J; Feng, Y; Song, R, Nonparametric independence screening in sparse ultra-high-dimensional additive models, J Am Stat Assoc, 106, 544-557, (2011) · Zbl 1232.62064
[28] Friedman, J; Stuelze, W, Projection pursuit regression, J Am Stat Assoc, 76, 817-823, (1981)
[29] Genuer, R; Poggi, JM; Tuleau-Malot, C, Variable selection using random forests, Pattern Recogn Lett, 31, 2225-2236, (2010)
[30] Hastie TJ, Tibshirani RJ (1990) Generalized additive models. Chapman & Hall, London · Zbl 0747.62061
[31] Horowitz, J; Klemel, J; Mammen, E, Optimal estimation in additive regression models, Bernoulli, 12, 271-298, (2006) · Zbl 1098.62043
[32] Huang, J; Breheny, P; Ma, F, A selective review of group selection in high dimensional model, Stat Sci, 27, 481-499, (2012) · Zbl 1331.62347
[33] Huang, J; Horowitz, JL; Wei, F, Variable selection in nonparametric additive models, Ann Stat, 38, 2282-2313, (2010) · Zbl 1202.62051
[34] Javanmard, A; Montanari, A, Hypothesis testing in high dimensional regression under the Gaussian random design model: asymptotic theory, IEEE Trans Inf Theory, 60, 6522-6554, (2014) · Zbl 1360.62074
[35] Kim, J; Kim, Y; Kim, Y, A gradient-based optimization algorithm for LASSO, J Comput Graph Stat, 17, 994-1009, (2008) · Zbl 1294.62206
[36] Koltchinskii, V; Yuan, M, Sparsity in multiple kernel learning, Ann Stat, 38, 3660-3695, (2010) · Zbl 1204.62086
[37] Lin, Y; Zhang, HH, Component selection and smoothing in multivariate nonparametric regression, Ann Stat, 34, 2272-2297, (2006) · Zbl 1106.62041
[38] Liu H, Zhang J (2008) On the \(ℓ _1\)-\(ℓ _q\) regularized regression, Technical report, Department of Statistics, Carnegie Mellon University. Preprint arXiv:0802.1517
[39] Liu, H; Zhang, J, Estimation consistency of the group lasso and its applications, J Mach Learn, 5, 376-383, (2009)
[40] Mammen, E; Park, BU, A simple smooth backfitting method for additive models, Ann Stat, 34, 2252-2271, (2006) · Zbl 1106.62042
[41] Marra, G; Wood, SN, Practical variable selection for generalized additive models, Comput Stat Data Anal, 55, 2372-2387, (2011) · Zbl 1328.62475
[42] Meier, L; Geer, S; Bühlmann, P, High-dimensional additive modeling, Ann Stat, 37, 3779-3821, (2009) · Zbl 1360.62186
[43] Ravikumar, P; Liu, H; Lafferty, J; Wasserman, L, Spam: sparse additive models, J R Stat Soc Ser B, 71, 1009-1030, (2009)
[44] Simon, N; Tibshirani, R, Standardization and the group lasso penalty, Stat Sin, 22, 983-1001, (2012) · Zbl 1257.62080
[45] Simon, N; Friedman, J; Hastie, T; Tibshirani, R, A sparse-group lasso, J Comput Graph Stat, 22, 231-245, (2013)
[46] Stamey, T; Kabalin, J; McNeal, J; Johnston, J; Freiha, F; Redwine, E; Yang, N, Ostate specific antigen in the diagnosis and treatment of adenocarcinoma of the prostate. II. radical prostatectomy treated patients, J Urol, 16, 1076-1083, (1989)
[47] Storlie, B; Bondell, HD; Reich, BJ; Zhang, HH, Surface estimation, variable selection, and the nonparametric oracle property, Stat Sin, 21, 679-705, (2011) · Zbl 1214.62044
[48] Suzuki, T; Sugiyama, M, Fast learning rate of multiple kernel learning: trade-off between sparsity and smoothness, Ann Stat, 41, 1381-1405, (2013) · Zbl 1273.62090
[49] Tibshirani, R, Regression shrinkage and selection via the lasso, J R Stat Soc Ser B, 58, 267-288, (1996) · Zbl 0850.62538
[50] Tutz, G; Binder, H, Generalized additive modeling with implicit variable selection by likelihood based boosting, Biometrics, 62, 961-971, (2006) · Zbl 1116.62075
[51] Geer, SA, High-dimensional generalized linear models and the lasso, Ann Stat, 36, 614-645, (2008) · Zbl 1138.62323
[52] Geer, S; Bühlmann, P, On the conditions used to prove oracle results for the lasso, Electron J Stat, 3, 1360-1392, (2009) · Zbl 1327.62425
[53] Geer, S; Bühlmann, P; Ritov, Y; Dezeure, R, On asymptotically optimal confidence regions and tests for high-dimensional models, Ann Stat, 42, 1166-1202, (2014) · Zbl 1305.62259
[54] Verhasselt A (2009) Variable selection with nonnegative garrote in additive models. In: Proceedings of EYSM 2009, Bucharest, Romania, pp 23-25 · Zbl 0594.62044
[55] Wahba G (1990) Spline models for observational data. SIAM, Philadelphia · Zbl 0813.62001
[56] Wainwright, M, Sharp thresholds for high-dimensional and noisy sparsity recovery using \(l_1\)-constrained quadratic programming (lasso), IEEE Trans Inf Theory, 55, 2183-2202, (2009) · Zbl 1367.62220
[57] Wand, MP; Ormerod, JT, On semiparametric regression with O’sullivan penalised splines, Aust N Z J Stat, 50, 179-198, (2008) · Zbl 1146.62030
[58] Wang, L; Li, H; Huang, JZ, Variable selection in nonparametric varying-coefficient models for analysis of repeated measurements, J Am Stat Assoc, 103, 1556-1569, (2008) · Zbl 1286.62034
[59] Wiesenfarth, M; Krivobokova, T; Klasen, S; Sperlich, S, Direct simultaneous inference in additive models and its application to model undernutrition, J Am Stat Assoc, 107, 1286-1296, (2012) · Zbl 1258.62114
[60] Wu, S; Xue, H; Wu, Y; Wu, H, Variable selection for sparse high-dimensional nonlinear regression models by combining nonnegative garrote and sure independence screening, Stat Sin, 24, 1365-1387, (2014) · Zbl 06431835
[61] Wood, S. N., Generalized Additive Models: An Introduction with R. Chapman and Hall/CRC (2006) · Zbl 1087.62082
[62] Wood, SN, A simple test for random effects in regression models, Biometrika, 100, 1005-1010, (2013) · Zbl 1279.62149
[63] Xue, L, Consistent variable selection in additive models, Stat Sin, 19, 1281-1296, (2009) · Zbl 1166.62024
[64] Yang, L, Confidence band for additive regression model, J Data Sci, 6, 207-217, (2008)
[65] Yang, Y; Zou, H, A fast unified algorithm for computing group lasso penalized learning problems, Stat Comput, 25, 1129-1141, (2015) · Zbl 1331.62343
[66] Yuan, M; Lin, Y, Model selection and estimation in regression with grouped variables, J R Stat Soc Ser B, 68, 49-67, (2006) · Zbl 1141.62030
[67] Yuan M (2007) Nonnegative garrote component selection in functional ANOVA models. In: Proceedings of the eleventh international conference on artificial intelligence and statistics, March 21-24, 2007, San Juan, Puerto Rico 2, pp 660-666
[68] Yuan, M; Lin, Y, On the nonnegative garrote estimator, J R Stat Soc Ser B, 69, 143-161, (2007) · Zbl 1120.62052
[69] Zhang, CH; Huang, J, The sparsity and bias of the lasso selection in high-dimensional linear regression, Ann Stat, 36, 1567-1594, (2008) · Zbl 1142.62044
[70] Zhang, CH; Zhang, SS, Confidence intervals for low dimensional parameters in high dimensional linear models, J R Stat Soc Ser B, 76, 217-241, (2014)
[71] Zhao, P; Rocha, G; Yu, B, The composite absolute penalties family for grouped and hierarchical variable selection, Ann Stat, 37, 3468-3497, (2009) · Zbl 1369.62164
[72] Zheng, S, Selection of components and degrees of smoothing via lasso in high dimensional nonparametric additive models, Comput Stat Data Anal, 53, 164-175, (2008) · Zbl 05565121
This reference list is based on information provided by the publisher or from digital mathematics libraries. Its items are heuristically matched to zbMATH identifiers and may contain data conversion errors. It attempts to reflect the references listed in the original paper as accurately as possible without claiming the completeness or perfect precision of the matching.