zbMATH — the first resource for mathematics

An improved GLMNET for L1-regularized logistic regression. (English) Zbl 1432.68404
Summary: Recently, G.-X. Yuan et al. [J. Mach. Learn. Res. 11, 3183–3234 (2010; Zbl 1242.62065)] conducted a comprehensive comparison on software for L1-regularized classification. They concluded that a carefully designed coordinate descent implementation CDN is the fastest among state-of-the-art solvers. In this paper, we point out that CDN is less competitive on loss functions that are expensive to compute. In particular, CDN for logistic regression is much slower than CDN for SVM because the logistic loss involves expensive exp/ operations. In optimization, Newton methods are known to have fewer iterations although each iteration costs more. Because solving the Newton sub-problem is independent of the loss calculation, this type of methods may surpass CDN under some circumstances. In L1-regularized classification, GLMNET by Friedman et al. is already a Newton-type method, but experiments in [loc. cit.] indicated that the existing GLMNET implementation may face difficulties for some large-scale problems. In this paper, we propose an improved GLMNET to address some theoretical and implementation issues. In particular, as a Newton-type method, GLMNET achieves fast local convergence, but may fail to quickly obtain a useful solution. By a careful design to adjust the effort for each iteration, our method is efficient for both loosely or strictly solving the optimization problem. Experiments demonstrate that our improved GLMNET is more efficient than CDN for L1-regularized logistic regression.

68T05 Learning and adaptive systems in artificial intelligence
62J07 Ridge regression; shrinkage estimators (Lasso)
62J12 Generalized linear models (logistic models)
Full Text: Link