Double-bagging: Combining classifiers by bootstrap aggregation. (English) Zbl 1028.68144

Summary: The combination of classifiers leads to substantial reduction of misclassification error in a wide range of applications and benchmark problems. We suggest using an out-of-bag sample for combining different classifiers. In our setup, a linear discriminant analysis is performed using the observations in the out-of-bag sample, and the corresponding discriminant variables computed for the observations in the bootstrap sample are used as additional predictors for a classification tree. Two classifiers are combined and therefore method and variable selection bias is no problem for the corresponding estimate of misclassification error, the need of an additional test sample disappears. Moreover, the procedure performs comparable to the best classifiers used in a number of artificial examples and applications.


68T10 Pattern recognition, speech recognition


ipred; rpart; R
Full Text: DOI


[1] Efron, B.; Tibshirani, R., Improvements on cross-validationthe 0.632+ bootstrap method, J. am. stat. assoc., 92, 438, 548-560, (1997) · Zbl 0887.62044
[2] Friedman, J.H., On bias, variance, 0/1-loss, and the course-of-dimensionality, Data mining knowledge discovery, 1, 1, 55-77, (1997)
[3] T. Hothorn, B. Lausen, Bagging tree classifiers for laser scanning images: data and simulation based strategy, Universität Erlangen-Nürnberg, preprint, submitted for publication. · Zbl 1429.62246
[4] Lausen, B.; Schumacher, M., Evaluating the effect of optimized cutoff values in the assessment of prognostic factors, Comput. stat. data anal., 21, 3, 307-326, (1996) · Zbl 0875.62567
[5] Breiman, L.; Friedman, J.H.; Olshen, R.A.; Stone, C.J., Classification and regression trees, (1984), Wadsworth California · Zbl 0541.62042
[6] Breiman, L., Bagging predictors, Mach. learning, 24, 2, 123-140, (1996) · Zbl 0858.68080
[7] Breiman, L., Arcing classifiers, Ann. stat., 26, 3, 801-824, (1998) · Zbl 0934.62064
[8] Schapire, R.E.; Freund, Y.; Bartlett, P.; Lee, W.S., Boosting the margina new explanation for the effectiveness of voting methods, Ann. stat., 26, 5, 1651-1686, (1998) · Zbl 0929.62069
[9] Kuncheva, L.I.; Bezdek, J.C.; Duin, R.P.W., Decision templates for multiple classifier fusionan experimental comparison, Pattern recognition, 34, 2, 299-314, (2001) · Zbl 0991.68064
[10] Saranli, A.; Demirekler, M., A statistical unified framework for rank-based multiple classifier decision combination, Pattern recognition, 34, 4, 865-884, (2001) · Zbl 0970.68759
[11] Giacinto, G.; Roli, F., Dynamic classifier selection based on multiple classifier behaviour, Pattern recognition, 34, 9, 1879-1881, (2001) · Zbl 0995.68100
[12] Du, Q.; Chang, C.-I., A linear constrained distance-based discriminant analysis for hyperspectral image classification, Pattern recognition, 34, 2, 361-373, (2001) · Zbl 0991.68067
[13] Yu, H.; Yang, J., A direct LDA algorithm for high-dimensional data—with application to face recognition, Pattern recognition, 34, 10, 2067-2070, (2001) · Zbl 0993.68091
[14] Skurichina, M.; Duin, R.P.W., Bagging for linear classifiers, Pattern recognition, 31, 7, 909-930, (1998)
[15] Skurichina, M.; Duin, R.P.W., The role of combining rules in bagging and boosting, Lecture notes in computer science, adv. pattern recognition, 1876, 631-640, (2000) · Zbl 0996.68773
[16] Lausen, B.; Sauerbrei, W.; Schumacher, M., Classification and regression trees (CART) used for the exploration of prognostic factors measured on different scales, (), 483-496
[17] L. Breiman, Out-of-bag estimation, Technical Report, Statistics Department, University of California Berkeley, Berkeley CA 94708, 1996.
[18] Ripley, B.D., Pattern recognition and neural networks, (1996), Cambridge University Press Cambridge, UK · Zbl 0853.62046
[19] Heidelberg Engineering, Heidelberg Retina Tomograph: Bedienungsanleitung Software version 2.01, Heidelberg Engineering GmbH, Heidelberg, 1997.
[20] Swindale, N.V.; Stjepanovic, G.; Chin, A.; Mikelberg, F.S., Automated analysis of normal and glaucomatous optic nerve head topography images, Invest. ophthalmol. visual sci., 41, 7, 1730-1742, (2000)
[21] Peters, A.; Hothorn, T.; Lausen, B., Ipredimproved predictors, R news, 2, 2, 33-36, (2002)
[22] Ihaka, R.; Gentlemen, R., Ra language for data analysis and graphics, J. comput. graphical stat., 5, 299-314, (1996)
[23] T. Therneau, E. Atkinson, An Introduction to recursive partitioning using the rpart routine, Technical Report, Vol. 61, Section of Biostatistics, Mayo Clinic, Rochester, 1997.
[24] LeBlanc, M.; Tibshirani, R., Combing estimates in regression and classification, J. amer. stat. assoc., 91, 436, 1641-1650, (1996) · Zbl 0881.62046
[25] Webb, G.I., Multiboostinga technique for combining boosting and wagging, Mach. learning, 40, 2, 159-196, (2000)
[26] Merz, C.J., Using correspondence analysis to combine classifiers, Mach. learning, 36, 1-2, 33-58, (1999)
[27] J.S. Rao, R. Tibshirani, The out-of-boostrap method for model averaging and selection, Technical Report, Cleveland Clinic and University of Toronto, 1997.
[28] Wolpert, D.H.; Macready, W.G., An efficient method to estimate Bagging’s generalization, Mach. learning, 35, 1, 41-51, (1999) · Zbl 0936.68090
This reference list is based on information provided by the publisher or from digital mathematics libraries. Its items are heuristically matched to zbMATH identifiers and may contain data conversion errors. It attempts to reflect the references listed in the original paper as accurately as possible without claiming the completeness or perfect precision of the matching.