×

zbMATH — the first resource for mathematics

Stochastic gradient boosting. (English) Zbl 1072.65502
Summary: Gradient boosting constructs additive regression models by sequentially fitting a simple parameterized function (base learner) to current “pseudo”-residuals by least squares at each iteration. The pseudo-residuals are the gradient of the loss functional being minimized, with respect to the model values at each training data point evaluated at the current step. It is shown that both the approximation accuracy and execution speed of gradient boosting can be substantially improved by incorporating randomization into the procedure. Specifically, at each iteration a subsample of the training data is drawn at random (without replacement) from the full training data set. This randomly selected subsample is then used in place of the full sample to fit the base learner and compute the model update for the current iteration. This randomized approach also increases robustness against overcapacity of the base learner.

MSC:
65C60 Computational problems in statistics (MSC2010)
62J99 Linear inference, regression
PDF BibTeX XML Cite
Full Text: DOI
References:
[1] Breiman, L., Bagging predictors, Mach. learning, 26, 123-140, (1996) · Zbl 0858.68080
[2] Breiman, L., 1999. Using adaptive bagging to debias regressions. Technical Report, Department of Statistics, University of California, Berkeley. · Zbl 1052.68109
[3] Freund, Y., Schapire, R., 1996. Experiments with a new boosting algorithm. Machine Learning: Proceedings of the 13th International Conference, pp. 148-156.
[4] Friedman, J.H., 1999. Greedy function approximation: a gradient boosting machine. Technical Report, Department of Statistics, Stanford University. · Zbl 1043.62034
[5] Friedman, J.H., Hastie, T., Tibshirani, R., 1998. Additive logistic regression: a statistical view of boosting. Technical Report, Department of Statistics, Stanford University. · Zbl 1106.62323
This reference list is based on information provided by the publisher or from digital mathematics libraries. Its items are heuristically matched to zbMATH identifiers and may contain data conversion errors. It attempts to reflect the references listed in the original paper as accurately as possible without claiming the completeness or perfect precision of the matching.