×

zbMATH — the first resource for mathematics

On boosting kernel regression. (English) Zbl 1182.62091
Summary: We propose a simple multistep regression smoother which is constructed in an iterative manner, by learning the Nadaraya-Watson estimator with \(L_{2}\) boosting. We find, in both theoretical analysis and simulation experiments, that the bias converges exponentially fast, and the variance diverges exponentially slow. The first boosting step is analysed in more detail, giving asymptotic expressions as functions of the smoothing parameter, and relationships with previous work are explored. Practical performance is illustrated by both simulated and real data.

MSC:
62G08 Nonparametric regression and quantile regression
62J02 General nonlinear regression
65C60 Computational problems in statistics (MSC2010)
62G05 Nonparametric estimation
PDF BibTeX XML Cite
Full Text: DOI
References:
[1] Breiman, L., 1997. Arcing the edge. Technical Report 486, Department of Statistics, University of California, Berkeley.
[2] Breiman, L.; Friedman, J., Estimating optimal transformations for multiple regression and correlation, J. amer. statist. assoc., 80, 580-598, (1985) · Zbl 0594.62044
[3] Bühlmann, P., Boosting for high-dimensional linear models, Ann. statist., 34, 559-583, (2006) · Zbl 1095.62077
[4] Bühlmann, P.; Yu, B., Boosting with the \(L_2\) loss: regression and classification, J. amer. statist. assoc., 98, 324-339, (2003) · Zbl 1041.62029
[5] Chaudhuri, P.; Doksum, K.; Samarov, A., On average derivative quantile regression, Ann. statist., 25, 715-744, (1997) · Zbl 0898.62082
[6] Di Marzio, M., Taylor, C.C., 2004. Boosting kernel density estimates: a bias reduction technique? Biometrika 91, 226-233. · Zbl 1132.62318
[7] Di Marzio, M.; Taylor, C.C., Kernel density classification and boosting: an \(L_2\) analysis, Statist. comput., 15, 113-123, (2005)
[8] Doksum, K.; Samarov, A., Nonparametric estimation of global functionals and a measure of explanatory power of covariates in regression, Ann. statist., 23, 1443-1473, (1995) · Zbl 0843.62045
[9] Fan, J.; Gijbels, I., Local polynomial modelling and its applications, (1996), Chapman & Hall London · Zbl 0873.62037
[10] Freund, Y., Boosting a weak learning algorithm by majority, Inform. comput., 121, 256-285, (1995) · Zbl 0833.68109
[11] Freund, Y., Schapire, R., 1996. Experiments with a new boosting algorithm. In: Machine Learning: Proceedings of the 13th International Conference, Morgan Kauffman, San Francisco, pp. 148-156.
[12] Friedman, J., Greedy function approximation: a gradient boosting machine, Ann. statist., 29, 1189-1232, (2001) · Zbl 1043.62034
[13] Friedman, J.; Hastie, T.; Tibshirani, R., Additive logistic regression: a statistical view of boosting, Ann. statist., 28, 337-407, (2000) · Zbl 1106.62323
[14] Härdle, W., Applied nonparametric regression, (1990), Cambridge University Press Cambridge · Zbl 0714.62030
[15] Harrison, D.; Rubinfeld, D., Hedonic prices and the demand for Clean air, J. environ. econom. management, 5, 81-102, (1978) · Zbl 0375.90023
[16] Hastie, T.; Loader, C., Local regression: automatic kernel carpentry, Statist. sci., 8, 120-143, (1993)
[17] Jiang, W., Process consistency for adaboost, Ann. statist., 32, 13-29, (2004) · Zbl 1105.62316
[18] Jones, M.C., Simple boundary correction for kernel density estimation, Statist. comput., 3, 135-146, (1993)
[19] Lax, P.D., Functional analysis, (2002), Wiley New York · Zbl 1009.47001
[20] Lugosi, G.; Vayatis, N., On the Bayes-risk consistency of regularized boosting methods, Ann. statist., 32, 30-55, (2004) · Zbl 1105.62319
[21] Müller, H.G., Comment on “local regression: automatic kernel carpentry” by T. hastie and C. loader, Statist. sci., 8, 120-143, (1993)
[22] Rice, J.A., Boundary modifications for kernel regression, Comm. statist. theory methods, 13, 893-900, (1984) · Zbl 0552.62022
[23] Schapire, R., The strength of weak learnability, Mach. learning, 5, 197-227, (1990)
[24] Stuetzle, W.; Mittal, Y., Some comments on the asymptotic behavior of robust smoothers, (), 191-195
[25] Tukey, J.W., Exploratory data analysis, (1977), Addison-Wesley Philippines · Zbl 0409.62003
[26] Zhang, T., Statistical behaviour and consistency of classification methods based on convex risk minimization, Ann. statist., 32, 56-85, (2004) · Zbl 1105.62323
[27] Zhang, T.; Yu, B., Boosting with early stopping: convergence and consistency, Ann. statist., 33, 1538-1579, (2005) · Zbl 1078.62038
This reference list is based on information provided by the publisher or from digital mathematics libraries. Its items are heuristically matched to zbMATH identifiers and may contain data conversion errors. It attempts to reflect the references listed in the original paper as accurately as possible without claiming the completeness or perfect precision of the matching.