×

A weight-adjusted voting algorithm for ensembles of classifiers. (English) Zbl 1296.62131

Summary: We present a new weighted voting classification ensemble method, called WAVE, that uses two weight vectors: a weight vector of classifiers and a weight vector of instances. The instance weight vector assigns higher weights to observations that are hard to classify. The weight vector of classifiers puts larger weights on classifiers that perform better on hard-to-classify instances. One weight vector is designed to be calculated in conjunction with the other through an iterative procedure. That is, the instances of higher weights play a more important role in determining the weights of classifiers, and vice versa. We proved that the iterated weight vectors converge to the optimal weights which can be directly calculated from the performance matrix of classifiers in an ensemble. The final prediction of the ensemble is obtained by voting using the optimal weight vector of classifiers. To compare the performance between a simple majority voting and the proposed weighted voting, we applied both of the voting methods to bootstrap aggregation and investigated the performance on 28 datasets. The result shows that the proposed weighted voting performs significantly better than the simple majority voting in general.

MSC:

62H30 Classification and discrimination; cluster analysis (statistical aspects)
68T10 Pattern recognition, speech recognition
91B12 Voting theory
PDFBibTeX XMLCite
Full Text: DOI

References:

[1] Asuncion, A., & Newman, D. J. (2007). UCI machine learning repository. University of California. Irvine. School of Information and Computer Science. http://www.ics.uci.edu/ mlearn/MLRepository.html; Asuncion, A., & Newman, D. J. (2007). UCI machine learning repository. University of California. Irvine. School of Information and Computer Science. http://www.ics.uci.edu/ mlearn/MLRepository.html
[2] Bauer, E.; Kohavi, R., An empirical comparison of voting classification algorithms: bagging, boosting, and variants, Machine Learning, 36, 105-139 (1999)
[3] Breiman, L., Bagging predictors, Machine Learning, 24, 123-140 (1996) · Zbl 0858.68080
[4] Breiman, L., Arcing classifiers, The Annals of Statistics, 26, 801-824 (1998) · Zbl 0934.62064
[5] Breiman, L., Random forests, Machine Learning, 45, 5-32 (2001) · Zbl 1007.68152
[6] Breiman, L.; Friedman, J. H.; Olshen, R. A.; Stone, C. J., Classification and regression trees (1984), Chapman & Hall: Chapman & Hall New York · Zbl 0541.62042
[7] Chandra, A.; Yao, X., Evolving hybrid ensembles of learning machines for better generalisation, Neurocomputing, 69, 686-700 (2006)
[8] Dietterich, T. G., Ensemble methods in machine learning (2000), Springer: Springer Berlin · Zbl 0963.68085
[9] Freund, Y.; Schapire, R., Experiments with a new boosting algorithm, (Proceedings of the thirteenth international conference on machine learning (1996), Morgan Kaufmann), 148-156
[10] Friedman, J.; Hastie, T.; Tibshirani, R., Additive logistic regression: a statistical view of boosting, The Annals of Statistics, 28, 337-407 (2000) · Zbl 1106.62323
[11] Geman, S.; Bienenstock, E.; Doursat, R., Neural networks and the bias/variance dilemma, Neural Computation, 4, 1-48 (1992)
[12] Hansen, L. K.; Salamon, P., Neural network ensembles, IEEE Transactions on Pattern Analysis and Machine Intelligence, 12, 993-1001 (1990)
[13] Hastie, T.; Tibshirani, R.; Friedman, J., The elements of statistical learning: data mining, inference, and prediction (2001), Springer-Verlag: Springer-Verlag New York · Zbl 0973.62007
[14] Heinz, G.; Peterson, L. J.; Johnson, R. W.; Kerk, C. J., Exploring relationships in body dimensions, Journal of Statistics Education, 11 (2003)
[15] Ho, T. K., The random subspace method for constructingdecision forests, IEEE Transactions of Pattern Analysis and Machine Intelligence, 20, 832-844 (1998)
[16] Hothorn, T.; Lausen, B., Double-bagging: combining classifiers by bootstrap aggregation, Pattern Recognition, 36, 1303-1309 (2003) · Zbl 1028.68144
[17] Hothorn, T.; Lausen, B., Bundling classifiers by bagging trees, Computational Statistics & Data Analysis, 49, 1068-1078 (2005) · Zbl 1429.62246
[18] Kim, H.; Loh, W.-Y., Classification trees with unbiased multiway splits, Journal of the American Statistical Association, 96, 589-604 (2001)
[19] Kim, H.; Loh, W.-Y., Classification trees with bivariate linear discriminant node models, Journal of Computational and Graphical Statistics, 12, 512-530 (2003)
[20] Kohavi, R.; Wolpert, D. H., Bias plus variance decomposition for zero-one loss functions, (Proceedings of the thirteenth international conference on machine learning (1996), Morgan Kaufmann), 275-283
[21] Kong, E. B.; Dietterich, T. G., Error-correcting output coding corrects bias and variance, (Proceedings of the twelfthth international conference on machine learning (1995), Morgan Kaufmann), 313-321
[22] Kuncheva, L., Diversity in multiple classifier systems, Information Fusion, 6, 3-4 (2005)
[23] Lam, L.; Suen, C. Y., Application of majority voting to pattern recognition: an analysis of its behavior and performancee, IEEE Transactions on Pattern Analysis and Machine Intelligence, 27, 553-568 (1997)
[24] Loh, W.-Y., Improving the precision of classification trees, The Annals of Applied Statistics, 3, 1710-1737 (2009) · Zbl 1184.62109
[25] Maslov, I. V.; Gertner, I., Multi-sensor fusion: an evolutionary algorithm approach, Information Fusion, 7, 304-330 (2006)
[26] Masnadi-Shirazi, H.; Vasconcelos, N., Cost-sensitive boosting, IEEE Transactions on Pattern Analysis and Machine Intelligence, 33, 294-309 (2011)
[27] Rahman, A. F.R.; Alam, H.; Fairhurst, M. C., Multiple classifier combination for character recognition: revisiting the majority voting system and its variations, (Document analysis systems V. Document analysis systems V, Lecture notes in computer science, Vol. 2423 (2002)), 167-178 · Zbl 1021.68776
[28] Rodriguez, J. J.; Kuncheva, L. I.; Alonso, C. J., Rotation forest: a new classifier ensemble method, IEEE Transactions on Pattern Analysis and Machine Intelligence, 28, 1619-1630 (2006)
[29] Rokach, L., Taxonomy for characterizing ensemble methods in classification tasks: a review and annotated bibliography, Computational Statistics and Data Analysis, 53, 4046-4072 (2009) · Zbl 1453.62185
[30] Schapire, R. E., The strength of weak learnability, Machine Learning, 5, 197-227 (1990)
[31] Statlib. 2010. Datasets archive. Carnegie Mellon University. Department of Statistics. http://lib.stat.cmu.edu; Statlib. 2010. Datasets archive. Carnegie Mellon University. Department of Statistics. http://lib.stat.cmu.edu
[32] Terhune, J. M., Geographical variation of harp seal underwater vocalisations, Canadian Journal of Zoology, 72, 892-897 (1994)
[33] Webb, G. I., Multiboosting: a technique for combining boosting and wagging, Machine Learning, 40, 159-196 (2000)
[34] Zhang, C. X.; Zhang, J. S., A local boosting algorithm for solving classification problems, Computational Statistics and Data Analysis, 52, 1928-1941 (2008) · Zbl 1452.62478
[35] Zhu, J.; Zou, H.; Rosset, S.; Hastie, T., Multi-class adaboost, Statistics and its Interface, 2, 349-360 (2009) · Zbl 1245.62080
This reference list is based on information provided by the publisher or from digital mathematics libraries. Its items are heuristically matched to zbMATH identifiers and may contain data conversion errors. In some cases that data have been complemented/enhanced by data from zbMATH Open. This attempts to reflect the references listed in the original paper as accurately as possible without claiming completeness or a perfect matching.