zbMATH — the first resource for mathematics

Majorization minimization by coordinate descent for concave penalized generalized linear models. (English) Zbl 1322.62190
Summary: Recent studies have demonstrated theoretical attractiveness of a class of concave penalties in variable selection, including the smoothly clipped absolute deviation and minimax concave penalties. The computation of the concave penalized solutions in high-dimensional models, however, is a difficult task. We propose a majorization minimization by coordinate descent (MMCD) algorithm for computing the concave penalized solutions in generalized linear models. In contrast to the existing algorithms that use local quadratic or local linear approximation to the penalty function, the MMCD seeks to majorize the negative log-likelihood by a quadratic loss, but does not use any approximation to the penalty. This strategy makes it possible to avoid the computation of a scaling factor in each update of the solutions, which improves the efficiency of coordinate descent. Under certain regularity conditions, we establish theoretical convergence property of the MMCD. We implement this algorithm for a penalized logistic regression model using the SCAD and MCP penalties. Simulation studies and a data example demonstrate that the MMCD works sufficiently fast for the penalized logistic regression in high-dimensional settings where the number of covariates is much larger than the sample size.

62J07 Ridge regression; shrinkage estimators (Lasso)
62J12 Generalized linear models (logistic models)
Full Text: DOI
[1] Böhning, D.; Lindsay, B., Monotonicity of quadratic-approximation algorithms, Ann. Inst. Stat. Math., 40, 641-663, (1988) · Zbl 0723.65150
[2] Breheny, P.; Huang, J., Coordinate descent algorithms for nonconvex penalized regression, with application to biological feature selection, Ann. Appl. Stat., 5, 232-253, (2011) · Zbl 1220.62095
[3] Donoho, D.L.; Johnstone, J.M., Ideal spatial adaptation by wavelet shrinkage, Biometrika, 81, 425-455, (1994) · Zbl 0815.62019
[4] Efron, B.; Hastie, T.; Johnstone, I.; Tibshirani, R., Least angle regression, Ann. Stat., 32, 407-451, (2004) · Zbl 1091.62054
[5] Fan, J.; Li, R., Variable selection via nonconcave penalized likelihood and its oracle properties, J. Am. Stat. Assoc., 96, 1348-13608, (2001) · Zbl 1073.62547
[6] Friedman, J.; Hastie, T.; Höfling, H.; Tibshirani, R., Pathwise coordinate optimization, Ann. Appl. Stat., 1, 302-332, (2007) · Zbl 1378.90064
[7] Friedman, J.; Hastie, T.; Tibshirani, R., Regularization paths for generalized linear models via coordinate descent, J. Stat. Softw., 33, 1-22, (2010)
[8] Hunter, D.R.; Lange, K., A tutorial on MM algorithms, Am. Stat., 58, 30-37, (2004)
[9] Hunter, D.R.; Li, R., Variable selection using MM algorithms, Ann. Stat., 33, 1617-1642, (2005) · Zbl 1078.62028
[10] Jiang, D., Huang, J., Zhang, Y.: The cross-validated AUC for MCP-logistic regression with high-dimensional data. Stat. Methods Med. Res. (2011, accepted). doi:10.1177/0962280211428385 · Zbl 1267.65009
[11] Lange, K.: Optimization. Springer, New York (2004) · Zbl 1140.90004
[12] Lange, K.; Hunter, D.; Yang, I., Optimization transfer using surrogate objective functions (with discussion), J. Comput. Graph. Stat., 9, 1-59, (2000)
[13] Mazumder, R.; Friedman, J.; Hastie, T., Sparsenet: coordinate descent with non-convex penalties, J. Am. Stat. Assoc., 106, 1125-1138, (2011) · Zbl 1229.62091
[14] Ortega, J.M., Rheinbold, W.C.: Iterative Solution of Nonlinear Equations in Several Variables. Academic Press, New York (1970) · Zbl 0241.65046
[15] Osborne, M.R.; Presnell, B.; Turlach, B.A., A new approach to variable selection in least square problems, IMA J. Numer. Anal., 20, 389-403, (2000) · Zbl 0962.65036
[16] R Development Core Team: R: a language and environment for statistical computing. R Foundation for Statistical Computing, Vienna, Austria, ISBN 3-900051-07-0. http://www.R-project.org · Zbl 1378.90064
[17] Schifano, E.D.; Strawderman, R.L.; Wells, M.T., Majorization-minimization algorithms for non-smoothly penalized objective functions, Electron. J. Stat., 4, 1258-1299, (2010) · Zbl 1267.65009
[18] Tibshirani, R., Regression shrinkage and selection via the lasso, J. R. Stat. Soc. Ser. B, 58, 267-288, (1996) · Zbl 0850.62538
[19] Tseng, P., Convergence of a block coordinate descent method for non-differentiable minimization, J. Optim. Theory Appl., 109, 475-494, (2001) · Zbl 1006.65062
[20] Vijver, M.J.; He, Y.D.; van’t Veer, L.J.; etal., A gene-expression signature as a predictor of survival in breast cancer, N. Engl. J. Med., 347, 1999-2009, (2002)
[21] van’t Veer, L.J.; Dai, H.; Vijver, M.J.; etal., Gene expression profiling predicts clinical outcome of breast cancer, Nature, 415, 530-536, (2002)
[22] Warge, J., Minimizing certain convex functions, SIAM J. Appl. Math., 11, 588-593, (1963) · Zbl 0128.05801
[23] Wu, T.T.; Lange, K., Coordinate descent algorithms for lasso penalized regression, Ann. Appl. Stat., 2, 224-244, (2008) · Zbl 1137.62045
[24] Zhang, C.H., Nearly unbiased variable selection under minimax concave penalty, Ann. Stat., 38, 894-942, (2010) · Zbl 1183.62120
[25] Zou, H.; Li, R., One-step sparse estimates in nonconcave penalized likelihood models, Ann. Stat., 36, 1509-1533, (2008) · Zbl 1142.62027
This reference list is based on information provided by the publisher or from digital mathematics libraries. Its items are heuristically matched to zbMATH identifiers and may contain data conversion errors. It attempts to reflect the references listed in the original paper as accurately as possible without claiming the completeness or perfect precision of the matching.