zbMATH — the first resource for mathematics

Boosting with the \(L_ 2\) loss: Regression and classification. (English) Zbl 1041.62029
Summary: This article investigates a computationally simple variant of boosting, \(L_2\)Boost, which is constructed from a functional gradient descent algorithm with the \(L_2\)-loss function. Like other boosting algorithms, \(L_2\)Boost uses many times in an iterative fashion a prechosen fitting method, called the learner. Based on the explicit expression of refitting of residuals of \(L_2\)Boost, the case with (symmetric) linear learners is studied in detail in both regression and classification.
In particular, with the boosting iteration \(m\) working as the smoothing or regularization parameter, a new exponential bias-variance trade-off is found with the variance (complexity) term increasing very slowly as \(m\) tends to infinity. When the learner is a smoothing spline, an optimal rate of convergence result holds for both regression and classification and the boosted smoothing spline even adapts to higher-order, unknown smoothness. Moreover, a simple expansion of a (smoothed) 0-1 loss function is derived to reveal the importance of the decision boundary, bias reduction, and impossibility of an additive bias-variance decomposition in classification.
Finally, simulation and real dataset results are obtained to demonstrate the attractiveness of \(L_2\)Boost. In particular, we demonstrate that \(L_2\)Boosting with a novel component-wise cubic smoothing spline is both practical and effective in the presence of high-dimensional predictors.

62G08 Nonparametric regression and quantile regression
62H30 Classification and discrimination; cluster analysis (statistical aspects)
Full Text: DOI