Relaxed Lasso. (English) Zbl 1452.62522

Meinshausen, Nicolai

Comput. Stat. Data Anal. 52, No. 1, 374-393 (2007).

Summary: The Lasso is an attractive regularisation method for high-dimensional regression. It combines variable selection with an efficient computational procedure. However, the rate of convergence of the Lasso is slow for some sparse high-dimensional data, where the number of predictor variables is growing fast with the number of observations. Moreover, many noise variables are selected if the estimator is chosen by cross-validation. It is shown that the contradicting demands of an efficient computational procedure and fast convergence rates of the \(\ell _{2}\)-loss can be overcome by a two-stage procedure, termed the relaxed Lasso. For orthogonal designs, the relaxed Lasso provides a continuum of solutions that include both soft- and hard-thresholding of estimators. The relaxed Lasso solutions include all regular Lasso solutions and computation of all relaxed Lasso solutions is often identically expensive as computing all regular Lasso solutions. Theoretical and numerical results demonstrate that the relaxed Lasso produces sparser models with equal or lower prediction loss than the regular Lasso estimator for high-dimensional data.

MSC:

62J07	Ridge regression; shrinkage estimators (Lasso)
62-08	Computational methods for problems pertaining to statistics

Keywords:

high dimensionality; bridge estimation; Lasso; \(\ell_q\)-norm penalisation; dimensionality reduction

References:

[1]	Breiman, L., Better subset regression using the nonnegative garrote, Technometrics, 37, 373-384 (1995) · Zbl 0862.62059
[2]	Efron, B.; Hastie, T.; Johnstone, I.; Tibshirani, R., Least angle regression, Ann. Statist., 32, 407-451 (2004) · Zbl 1091.62054
[3]	Fan, J.; Li, R., Variable selection via penalized likelihood, J. Amer. Statist. Assoc., 96, 1348-1360 (2001) · Zbl 1073.62547
[4]	Fan, J.; Peng, H., Nonconcave penalized likelihood with a diverging number of parameters, Ann. Statist., 32, 928-961 (2004) · Zbl 1092.62031
[5]	Frank, I.; Friedman, J., A statistical view of some chemometrics regression tools (with discussion), Technometrics, 35, 109-148 (1993) · Zbl 0775.62288
[6]	Huber, P., Robust regression: asymptotics, conjectures, and Monte Carlo, Ann. Statist., 1, 799-821 (1973) · Zbl 0289.62033
[7]	Knight, K.; Fu, W., Asymptotics for lasso-type estimators, Ann. Statist., 28, 1356-1378 (2000) · Zbl 1105.62357
[8]	McCullagh, P.; Nelder, J. A., Generalized Linear Models (1989), Chapman & Hall: Chapman & Hall London · Zbl 0744.62098
[9]	Meinshausen, N.; Bühlmann, P., High dimensional graphs and variable selection with the lasso, Ann. Statist., 34, 1436-1462 (2006) · Zbl 1113.62082
[10]	Osborne, M.; Presnell, B.; Turlach, B., On the lasso and its dual, J. Comput. Graph. Statist., 9, 319-337 (2000)
[11]	Tibshirani, R., Regression shrinkage and selection via the lasso, J. Roy. Statist. Soc. Ser. B, 58, 267-288 (1996) · Zbl 0850.62538
[12]	Tsybakov, A.; van de Geer, S., Square root penalty: adaptation to the margin in classification and in edge estimation, Ann. Statist., 33, 1203-1224 (2005) · Zbl 1080.62047
[13]	van de Geer, S.; van Houwelingen, H., High dimensional data: \(p ⪢ n\) in mathematical statistics and bio-medical applications, Bernoulli, 10, 939-943 (2004)
[14]	Zhao, P., Yu, B., 2004. Boosted lasso. Technical Report 678, University of California, Berkeley.; Zhao, P., Yu, B., 2004. Boosted lasso. Technical Report 678, University of California, Berkeley.