Sparse group Lasso and high dimensional multinomial classification. (English) Zbl 1471.62200

Vincent, Martin; Hansen, Niels Richard

Comput. Stat. Data Anal. 71, 771-786 (2014).

Summary: The sparse group lasso optimization problem is solved using a coordinate gradient descent algorithm. The algorithm is applicable to a broad class of convex loss functions. Convergence of the algorithm is established, and the algorithm is used to investigate the performance of the multinomial sparse group lasso classifier. On three different real data examples the multinomial group lasso clearly outperforms multinomial lasso in terms of achieved classification error rate and in terms of including fewer features for the classification. An implementation of the multinomial sparse group lasso algorithm is available in the R package {msg1}. Its performance scales well with the problem size as illustrated by one of the examples considered – a 50 class classification problem with 10 k features, which amounts to estimating 500 k parameters.

MSC:

62-08	Computational methods for problems pertaining to statistics
62J07	Ridge regression; shrinkage estimators (Lasso)
62H30	Classification and discrimination; cluster analysis (statistical aspects)

Keywords:

sparse group Lasso; classification; high-dimensional data analysis; coordinate gradient descent; penalized loss

Software:

msgl; ElemStatLearn; glmnet; RcppArmadillo; Boost C++ Libraries; UCI-ml; Boost; glasso; Armadillo; R

References:

[1]	Bakay, M.; Wang, Z.; Melcon, G.; Schiltz, L.; Xuan, J.; Zhao, P.; Sartorelli, V.; Seo, J.; Pegoraro, E.; Angelini, C., Nuclear envelope dystrophies show a transcriptional fingerprint suggesting disruption of rb-myod pathways in muscle regeneration, Brain, 129, 4, 996, (2006)
[2]	Bertsekas, D.; Nedić, A.; Ozdaglar, A., (Convex Analysis and Optimization, Athena Scientific Optimization and Computation Series, (2003), Athena Scientific)
[3]	Boost C++ libraries, version 1.51.0, 2012. URL http://www.boost.org.
[4]	Demming, R.; Duffy, D., Introduction to the boost C++ libraries, (2010), Datasim Education
[5]	Eddelbuettel, D.; Sanderson, C., Rcpparmadillo : accelerating \(r\) with high-performance C++ linear algebra, Computational Statistics & Data Analysis, (2013), URL http://eprints.qut.edu.au/57210/ · Zbl 1471.62055
[6]	Frank, A., Asuncion, A., 2010. UCI machine learning repository. URL http://archive.ics.uci.edu/ml.
[7]	Friedman, J., Hastie, T., Tibshirani, R., A note on the group lasso and a sparse group lasso, Tech. Rep., Comments: 8 pages, 3 figs (Jan 2010a), arXiv:1001.0736.
[8]	Friedman, J.; Hastie, T.; Tibshirani, R., Regularization paths for generalized linear models via coordinate descent, Journal of Statistical Software, 33, 1, 1-22, (2010), URL http://www.jstatsoft.org/v33/i01/
[9]	Hastie, T.; Tibshirani, R.; Friedman, J. H., The elements of statistical learning: data mining, inference, and prediction: with 200 full-color illustrations, (2001), Springer-Verlag New York
[10]	Kim, Y.; Kwon, S.; Heun Song, S., Multiclass sparse logistic regression for classification of multiple cancer types using gene expression data, Computational Statistics & Data Analysis, 51, 3, 1643-1655, (2006), URL http://dx.doi.org/10.1016/j.csda.2006.06.007 · Zbl 1157.62535
[11]	Liu, S.; Liu, Z.; Sun, J.; Liu, L., Application of synergetic neural network in online writeprint identification, International Journal of Digital Content Technology and its Applications, 5, 3, 126-135, (2011)
[12]	Lu, J.; Getz, G.; Miska, E.; Alvarez-Saavedra, E.; Lamb, J.; Peck, D.; Sweet-Cordero, A.; Ebert, B.; Mak, R.; Ferrando, A., Microrna expression profiles classify human cancers, Nature, 435, 7043, 834-838, (2005)
[13]	Meier, L.; Geer, S. V.D.; Bhlmann, P.; Zrich, E. T.H., The group lasso for logistic regression, Journal of the Royal Statistical Society, Series B, (2008)
[14]	The openmp api specification for parallel programming, version 3.1, 2011. URL http://www.openmp.org.
[15]	Sanderson, C., Armadillo: an open source C++ linear algebra library for fast prototyping and computationally intensive experiments, Tech. rep., NICTA (October 2010).
[16]	Simil, T.; Selection, J. T.I.; In, S.; Simil, T.; Tikka, J., Input selection and shrinkage in multiresponse linear regression, Computational Statistics and Data Analysis, 52, 406-422, (2007) · Zbl 1452.62513
[17]	Simon, N.; Friedman, J.; Hastie, T., A blockwise descent algorithm for group-penalized multiresponse and multinomial regression, (2013)
[18]	Simon, N.; Friedman, J.; Hastie, T.; Tibshirani, R., A sparse group lasso, Journal of Computational and Graphical Statistics, (2013)
[19]	Tibshirani, R., Regression shrinkage and selection via the lasso, Journal of the Royal Statistical Society, Series B, 58, 267-288, (1994) · Zbl 0850.62538
[20]	Tseng, P., Convergence of a block coordinate descent method for nondifferentiable minimization, Journal of optimization theory and applications, 109, 3, 475-494, (2001) · Zbl 1006.65062
[21]	Tseng, P.; Yun, S., A coordinate gradient descent method for nonsmooth separable minimization, Mathematical Programming, 117, 1, 387-423, (2009) · Zbl 1166.90016