×

Ensemble learning based on approximate reducts and bootstrap sampling. (English) Zbl 1475.68275

Summary: Ensemble learning is an effective approach for improving the generalization ability of base classifiers. To generate a set of accurate and diverse base classifiers, different data perturbation schemes have been proposed. For instance, Bagging perturbs the training data via bootstrap sampling. However, when a stable learning algorithm (e.g., KNN, Naive Bayes) is used to train base classifiers, the sole perturbation on the training data may not produce diverse base classifiers. In this paper, by using the attribute reduction technology in rough sets, a multi-modal perturbation-based algorithm (called ‘E_EARBS’) is proposed for the ensemble of base classifiers. E\(\_\)EARBS simultaneously perturbs the feature space, training data and learning parameters, where the relative decision entropy(RDE)-based approximate reducts are used to perturb the feature space, and bootstrap sampling is used to perturb the training data. Experimental results show that E_EARBS can provide competitive solutions for ensemble learning.

MSC:

68T05 Learning and adaptive systems in artificial intelligence
62H30 Classification and discrimination; cluster analysis (statistical aspects)

Software:

UCI-ml; FSMRDE
PDFBibTeX XMLCite
Full Text: DOI

References:

[1] Acharjya, D. P.; Tripathy, B. K., Rough sets on fuzzy approximation spaces and applications to distributed knowledge systems, Int. J. Artif. Intell. Soft Comput., 1, 1, 1-14 (2008)
[2] Acharjya, D. P.; Tripathy, B. K., Rough sets on intuitionistic fuzzy approximation spaces and knowledge representation, Int. J. Artif. Intell. Comput. Res., 1, 1, 29-36 (2009)
[3] Acharjya, D. P., Rough computing based information retrieval in knowledge discovery databases, (Roy, A. K., Inf. Knowl. Manag. (2013)), 123-153
[4] Ahmed, N. S.S.; Acharjya, D. P., Detection of denial of service attack in wireless network using dominance based rough set, Int. J. Adv. Comput. Sci. Appl., 6, 12, 267-278 (2015)
[5] K. Bache, M. Lichman, UCI machine learning repository, 2013. http://archive. ics.uci.edu/ml.
[6] Banfield, R. E.; Hall, L. O.; Bowyer, K. W.; Kegelmeyer, W. P., A comparison of decision tree ensemble creation techniques, IEEE Trans. Pattern Anal. Mach. Intell., 29, 1, 173-180 (2007)
[7] Benítez-Caballero, M. J.; Medina, J.; Ramírez-Poussa, E.; Śle¸zak, D., Bireducts with tolerance relations, Inf. Sci., 435, 26-39 (2018) · Zbl 1440.68283
[8] Breiman, L., Bagging predictors, Mach. Learn., 24, 2, 123-140 (1996) · Zbl 0858.68080
[9] J.H. Dai, H.F. Han, Q.H. Hu, M.F. Liu, Discrete particle swarm optimization approach for cost sensitive attribute reduction, Knowl.-Based Syst. 102 (C) (2016) 116-126.
[10] Dai, J. H.; Wei, B. J.; Zhang, X. H.; Zhang, Q. L., Uncertainty measurement for incomplete interval-valued information systems based on α-weak similarity, Knowl.-Based Syst., 136, 159-171 (2017)
[11] Dai, J. H.; Hu, H.; Wu, W. Z.; Qian, Y. H.; Huang, D. B., Maximal-discernibility-pair-based approach to attribute reduction in fuzzy rough sets, IEEE Trans. Fuzzy Syst., 26, 4, 2174-2187 (2018)
[12] Dai, J. H.; Hu, H.; Hu, Q. H.; Huang, W.; Zheng, N. G.; Liu, L., Locally linear approximation approach for incomplete data, IEEE Trans. Cybern., 48, 6, 1720-1732 (2018)
[13] Demšar, J., Statistical comparisons of classifiers over multiple data sets, J. Mach. Learn. Res., 7, 1-30 (2006) · Zbl 1222.68184
[14] Dietterich, T. G., Ensemble learning, (The Handbook of Brain Theory and Neural Networks (2002), MIT Press: MIT Press Cambridge, MA)
[15] Díez-Pastor, J. F.; Rodríguez, J. J.; García-Osorio, C. I.; Kuncheva, L. I., Diversity techniques improve the performance of the best imbalance learning ensembles, Inf. Sci., 325, 98-117 (2015)
[16] Dubois, D.; Prade, H., Rough fuzzy sets and fuzzy rough sets, Int. J. Gen Syst, 17, 2-3, 191-209 (1990) · Zbl 0715.04006
[17] U.M. Fayyad, K.B. Irani, Multi-interval discretization of continuous-valued attributes for classification learning, in: Proc. of the 13th Int. Joint Conf. on Artificial Intelligence (IJCAI-93), Chambery, France, 1993, pp. 1022-1027.
[18] Fernández-Delgado, M.; Cernadas, E.; Barro, S.; Amorim, D., Do we need hundreds of classifiers to solve real world classification problems?, J. Mach. Learn. Res., 15, 3133-3181 (2014) · Zbl 1319.62005
[19] Gao, C.; Pedrycz, W.; Miao, D. Q., Rough subspace-based clustering ensemble for categorical data, Soft Comput., 17, 9, 1643-1658 (2013) · Zbl 1325.68187
[20] García-Pedrajas, N.; Ortiz-Boyer, D., Boosting random subspace method, Neural Networks, 21, 1344-1362 (2008) · Zbl 1254.68202
[21] Grzegorowski, M.; Śle¸zak, D., On resilient feature selection: Computational foundations of r-\( C\)-reducts, Inf. Sci., 499, 25-44 (2019) · Zbl 1451.68224
[22] Ho, T. K., The random subspace method for constructing decision forests, IEEE Trans. Pattern Anal. Mach. Intell., 20, 8, 832-844 (1998)
[23] Hu, Q. H.; Yu, D. R.; Xie, Z. X.; Li, X. D., EROS: ensemble rough subspaces, Pattern Recognit., 40, 12, 3728-3739 (2007) · Zbl 1129.68074
[24] Hu, Q. H.; Pedrycz, W.; Yu, D. R.; Lang, J., Selecting discrete and continuous features based on neighborhood decision error minimization, IEEE Trans. SMC, Part B, 40, 1, 137-150 (2010)
[25] A. Janusz, D. Śle¸zak, Computation of approximate reducts with dynamically adjusted approximation threshold, in: Proc. of the 22nd Int. Symposium on Methodologies for Intelligent Systems (ISMIS 2015), Lyon, France, 2015, pp. 19-28.
[26] Jiang, F.; Sui, Y. F.; Zhou, L., A relative decision entropy-based feature selection approach, Pattern Recognit., 48, 7, 2151-2163 (2015) · Zbl 1374.68391
[27] P. Latinne, O. Debeir, Ch. Decaestecker, Different ways of weakening decision trees and their impact on classification accuracy of DT combination, in: Proc. of the First Int. Workshop of Multiple Classifier Systems (MCS 2000), Cagliari, Italy, 2000, pp. 200-209.
[28] Li, F.; Miao, D. Q.; Pedrycz, W., Granular multi-label feature selection based on mutual information, Pattern Recognit., 67, 410-423 (2017)
[29] Marqués, A. I.; García, V.; Sánchez, J. S., Two-level classifier ensembles for credit risk assessment, Exp. Sys. Appl., 39, 12, 10916-10922 (2012)
[30] Nguyen, H. S.; Nguyen, S. H., Discretization methods in data mining, (Polkowski, L.; Skowron, A., Rough Sets in Knowledge Discovery (1998), Physica-Verlag), 451-482 · Zbl 0940.68139
[31] H.S. Nguyen, D. Śle¸zak, Approximate reducts and association rules — correspondence and complexity results, in: Proc. of the 7th Int. Workshops on New Directions in Rough Sets, Data Mining, and Granular-Soft Computing, Yamaguchi, Japan, 1999, pp. 137-145. · Zbl 0954.68129
[32] Pawlak, Z., Rough Sets: Theoretical Aspects of Reasoning About Data (1991), Kluwer Academic Publishers: Kluwer Academic Publishers Dordrecht · Zbl 0758.68054
[33] Pedrycz, W.; Vukovich, G., Feature analysis through information granulation and fuzzy sets, Pattern Recognit., 35, 4, 825-834 (2002) · Zbl 0997.68114
[34] Pietruczuk, L.; Rutkowski, L.; Jaworski, M.; Duda, P., How to adjust an ensemble size in stream data mining?, Inf. Sci., 381, 46-54 (2017) · Zbl 1429.68237
[35] Rodríguez, J. J.; Kuncheva, L. I.; Alonso, C. J., Rotation forest: A new classifier ensemble method, IEEE Trans. Pattern Anal. Mach. Intell., 28, 10, 1619-1630 (2006)
[36] Schapire, R. E., The strength of weak learnability, Mach. Learn., 5, 2, 197-227 (1990)
[37] Shannon, C. E., The mathematical theory of communication, Bell Syst. Tech. J., 27, 3-4, 373-423 (1948) · Zbl 1154.94303
[38] D. Śle¸zak, Approximation reducts in decision tables, in: Proc. of the 6th Int. Conf. on Information Processing and Management of Uncertainty in Knowledge-Based Systems (IPMU 1996), vol. 3, 1996, pp. 1159-1164.
[39] Śle¸zak, D., Approximate entropy reducts, Fundam. Inform., 53, 3-4, 365-390 (2002) · Zbl 1092.68676
[40] D. Śle¸zak, S. Widz, Evolutionary inspired optimization of feature subset ensembles, in: Proc. of the 2nd World Congress on Nature and Biologically Inspired Computing (NaBIC 2010), Fukuoka, Japan, 2010, pp. 444-449.
[41] D. Śle¸zak, A. Janusz, Ensembles of bireducts: towards robust classification and simple representation, in: Proc. of the 3rd Int. Conf. on Future Generation Information Technology (FGIT 2011), Jeju Island, Korea, 2011, pp. 64-77.
[42] D. Śle¸zak, On generalized decision functions: reducts, networks and ensembles, in: Proc. of the 15th Int. Conf. on Rough Sets, Fuzzy Sets, Data Mining, and Granular Computing (RSFDGrC 2015), Tianjin, China, 2015, pp. 13-23. · Zbl 1444.68245
[43] S. Stawicki, S. Widz, Decision bireducts and approximate decision reducts: Comparison of two approaches to attribute subset ensemble construction, in: Proc. of the 2012 Federated Conf. on Computer Science and Information Systems (FedCSIS 2012), Wroclaw, Poland, 2012, pp. 331-338.
[44] Stawicki, S.; Śle¸zak, D.; Janusz, A.; Widz, S., Decision bireducts and decision reducts — a comparison, Int. J. Approx. Reason., 84, 75-109 (2017) · Zbl 1419.68178
[45] J. Stefanowski, Combining answers of sub-classifiers in the bagging-feature ensembles, in: Proc. of the Int. Conf. on Rough Sets and Intelligent Systems Paradigms (RSEISP 2007), Warsaw, Poland, 2007, pp. 574-583.
[46] Tsymbal, A.; Pechenizkiy, M.; Cunningham, P., Diversity in search strategies for ensemble feature selection, Inf. Fusion, 6, 1, 83-98 (2005)
[47] Witten, I. H.; Frank, E.; Hall, M. A., Data Mining: Practical Machine Learning Tools and Techniques (2011), Morgan Kaufmann Publishers: Morgan Kaufmann Publishers San Francisco, CA
[48] Yang, X. B.; Yang, J. Y.; Wu, C.; Yu, D. J., Dominance-based rough set approach and knowledge reductions in incomplete ordered information system, Inf. Sci., 178, 4, 1219-1234 (2008) · Zbl 1134.68057
[49] Y.Q. Zhang, F. Jiang, J.W. Du, G.Z. Liu, A novel selective ensemble algorithm based on approximate reducts and optimal sampling, in: Proc. of the 2016 IEEE Int. Conf. on Computing, Communication and Automation(ICCCA 2016), Greater Noida, India, 2016, pp. 263-268.
[50] Zhou, Z. H.; Yu, Y., Ensembling local learners through multimodal perturbation, IEEE Trans. SMC, Part B, 35, 4, 725-735 (2005)
This reference list is based on information provided by the publisher or from digital mathematics libraries. Its items are heuristically matched to zbMATH identifiers and may contain data conversion errors. In some cases that data have been complemented/enhanced by data from zbMATH Open. This attempts to reflect the references listed in the original paper as accurately as possible without claiming completeness or a perfect matching.