×

Clustering-based ensembles for one-class classification. (English) Zbl 1335.68205

Summary: This paper presents a novel multi-class classifier based on weighted one-class support vector machines (OCSVM) operating in the clustered feature space. We show that splitting the target class into atomic subsets and using these as input for one-class classifiers leads to an efficient and stable recognition algorithm. The proposed system extends our previous works on combining OCSVM classifiers to solve both one-class and multi-class classification tasks. The main contribution of this work is the novel architecture for class decomposition and combination of classifier outputs. Based on the results of a large number of computational experiments we show that the proposed method outperforms both the OCSVM for a single class, as well as the multi-class SVM for multi-class classification problems. Other advantages are the highly parallel structure of the proposed solution, which facilitates parallel training and execution stages, and the relatively small number of control parameters.

MSC:

68T05 Learning and adaptive systems in artificial intelligence
68T10 Pattern recognition, speech recognition

Software:

Kernlab; LIBSVM; R; e1071; SHOGUN
PDFBibTeX XMLCite
Full Text: DOI

References:

[1] Alpaydin, Ethem, Combined 5×2 cv f test for comparing supervised classification learning algorithms, Neural Comput., 11, 8, 1885-1892 (1999)
[2] Bartkowiak, A.; Zimroz, R., Outliers analysis and one class classification approach for planetary gearbox diagnosis, J. Phys.: Conf. Ser., 305, 1 (2011)
[3] Bezdek, J., Pattern Recognition With Fuzzy Objective Function Algorithms (1981), Plenum Press: Plenum Press New York · Zbl 0503.68069
[4] Bezdek, J. C.; Ehrlich, R.; Full, W., Fcm: the fuzzy c-means clustering algorithm, Comput. Geosci., 10, 2-3, 191-203 (1984)
[5] Bi, Y., The impact of diversity on the accuracy of evidential classifier ensembles, Int. J. Approx. Reason., 53, 4, 584-607 (2012)
[6] Bicego, M.; Figueiredo, M. A.T., Soft clustering using weighted one-class support vector machines, Pattern Recogn., 42, 1, 27-32 (2009) · Zbl 1173.68623
[7] Chan, P. P.K.; Yeung, D. S.; Ng, W. W.Y.; Lin, C. M.; Liu, J. N.K., Dynamic fusion method using localized generalization error model, Inform. Sci., 217, 1-20 (2012)
[8] Chang, Chih-Chung; Lin, Chih-Jen, LIBSVM: a library for support vector machines, ACM Trans. Intell. Syst. Technol., 2, 27:1-27:27 (2011), <http://www.csie.ntu.edu.tw/cjlin/libsvm>
[9] Chen, B.; Feng, A.; Chen, S.; Li, B., One-cluster clustering based data description, Jisuanji Xuebao/Chinese J. Comput., 30, 8, 1325-1332 (2007)
[10] Chen, Y.; Zhou, X. S.; Huang, T. S., One-class svm for learning in image retrieval, IEEE Int. Conf. Image Process., 1, 34-37 (2001)
[11] Cheung, K. F., Fuzzy one-mean algorithm: Formulation, convergence analysis, and applications, J. Intell. Fuzzy Syst., 5, 4, 323-332 (1997)
[12] Cohen, G.; Sax, H.; Geissbuhler, A., Novelty detection using one-class parzen density estimator: an application to surveillance of nosocomial infections, Stud. Health Technol. Inform., 136, 21-26 (2008)
[14] Cyganek, B., One-class support vector ensembles for image segmentation and classification, J. Math. Imag. Vis., 42, 2-3, 103-117 (2012) · Zbl 1255.68223
[15] Dai, Q., A competitive ensemble pruning approach based on cross-validation technique, Knowl.-Based Syst., 37, 394-414 (2013)
[16] Dietterich, Thomas G.; Bakiri, Ghulum, Solving multiclass learning problems via error-correcting output codes, J. Artif. Int. Res., 2, 263-286 (January 1995) · Zbl 0900.68358
[19] Frahling, Gereon; Sohler, Christian, A fast k-means implementation using coresets, Int. J. Comput. Geometry Appl., 18, 6, 605-625 (2008) · Zbl 1182.65034
[20] Galar, M.; Fernndez, A.; Barrenechea, E.; Bustince, H.; Herrera, F., An overview of ensemble methods for binary classifiers in multi-class problems: experimental study on one-vs-one and one-vs-all schemes, Pattern Recogn., 44, 8, 1761-1776 (2011)
[21] Garcia-Pedrajas, N.; Maudes-Raedo, J.; Garca-Osorio, C.; Rodriguez-Diez, J. J., Supervised subspace projections for constructing ensembles of classifiers, Inform. Sci., 193, 1-21 (2012)
[22] Gardner, A. B.; Krieger, A. M.; Vachtsevanos, G.; Litt, B., One-class novelty detection for seizure analysis from intracranial eeg, J. Machine Learn. Res., 7, 1025-1044 (2006) · Zbl 1222.92052
[23] Giacinto, Giorgio; Perdisci, Roberto; Del Rio, Mauro; Roli, Fabio, Intrusion detection in computer networks by a modular ensemble of one-class classifiers, Inf. Fusion, 9, 69-82 (January 2008)
[24] Ho, Tin Kam, The random subspace method for constructing decision forests, IEEE Trans. Pattern Anal. Mach. Intell., 20, August, 832-844 (1998)
[26] Jain, A. K.; Duin, R. P.W.; Mao, Jianchang, Statistical pattern recognition: a review, IEEE Trans. Pattern Anal. Machine Intell., 22, 1, 4-37 (2000)
[27] Jiang, H.; Liu, G.; Xiao, X.; Mei, C.; Ding, Y.; Yu, S., Monitoring of solid-state fermentation of wheat straw in a pilot scale using ft-nir spectroscopy and support vector data description, Microchem. J., 102 (2012)
[29] Juszczak, P.; Tax, D. M.J.; Pekalska, E.; Duin, R. P.W., Minimum spanning tree based one-class classifier, Neurocomputing, 72, 7-9, 1859-1869 (2009)
[30] Karatzoglou, A.; Hornik, K.; Smola, A.; Zeileis, A., kernlab - an s4 package for kernel methods in r, J. Stat. Softw., 11, 1-20 (2004)
[31] Koch, M. W.; Moya, M. M.; Hostetler, L. D.; Fogler, R. J., Cueing, feature discovery, and one-class learning for synthetic aperture radar automatic target recognition, Neural Netw., 8, 7-8, 1081-1102 (1995)
[32] Krawczyk, B., Diversity in ensembles for one-class classification, (Pechenizkiy, Mykola; Wojciechowski, Marek, New Trends in Databases and Information Systems. New Trends in Databases and Information Systems, Advances in Intelligent Systems and Computing, vol. 185 (2012), Springer: Springer Berlin Heidelberg), 119-129
[33] Krawczyk, B.; Woźniak, M., Combining diverse one-class classifiers, (Corchado, Emilio; Snasel, Vaclav; Abraham, Ajith; Wozniak, Michal; Grana, Manuel; Cho, Sung-Bae, Hybrid Artificial Intelligent Systems. Hybrid Artificial Intelligent Systems, Lecture Notes in Computer Science, vol. 7209 (2012), Springer: Springer Berlin/Heidelberg), 590-601
[35] Krawczyk, B.; Woźniak, M., Diversity measures for one-class classifier ensembles, Neurocomputing (2013)
[36] Kuncheva, L. I.; Whitaker, C. J.; Shipp, C. A.; Duin, R. P.W., Limits on the majority vote accuracy in classifier fusion, Pattern Anal. Appl., 6, 22-31 (2003) · Zbl 1035.68101
[39] MacQueen, J. B., Some methods for classification and analysis of multivariate observations, (Le Cam, L. M.; Neyman, J., Proc. of the fifth Berkeley Symposium on Mathematical Statistics and Probability, vol. 1 (1967), University of California Press), 281-297 · Zbl 0214.46201
[40] Manevitz, L.; Yousef, M., One-class document classification via neural networks, Neurocomputing, 70, 7-9, 1466-1481 (2007)
[42] Ratsch, G.; Mika, S.; Scholkopf, B.; Muller, K., Constructing boosting algorithms from svms: an application to one-class classification, IEEE Trans. Pattern Anal. Machine Intell., 24, 9, 1184-1199 (2002)
[43] Hitendra Sarma, T.; Viswanath, P.; Eswara Reddy, B., A hybrid approach to speed-up the k-means clustering method, Int. J. Machine Learn. Cybernet., 4, 2, 107-117 (2013)
[44] Schölkopf, B.; Smola, A. J., Learning with Kernels: Support Vector Machines, Regularization, Optimization, and Beyond (2002), MIT Press: MIT Press Adaptive computation and machine learning
[47] Sonnenburg, S.; Rtsch, G.; Schfer, C.; Schlkopf, B., Large scale multiple kernel learning, J. Machine Learn. Res., 7, 1531-1565 (2006) · Zbl 1222.90072
[48] Szczurek, A.; Krawczyk, B.; Maciejewska, M., Vocs classification based on the committee of classifiers coupled with single sensor signals, Chemometr. Intell. Lab. Syst., 125, 1-10 (2013)
[49] Tadeusiewicz, R.; Ogiela, M. R.; Szczepaniak, P. S., Notes on a linguistic description as the basis for automatic image understanding, Int. J. Appl. Math. Comput. Sci., 19, 1, 143-150 (2009) · Zbl 1176.68219
[50] Tax, D. M.J.; Duin, R. P.W., Support vector data description, Machine Learn., 54, 1, 45-66 (2004) · Zbl 1078.68728
[51] Tax, David M. J.; Duin, Robert P. W., Combining one-class classifiers, (Proceedings of the Second International Workshop on Multiple Classifier Systems, MCS ’01 (2001), Springer-Verlag: Springer-Verlag London, UK), 299-308 · Zbl 0980.68622
[52] Tax, David M. J.; Juszczak, Piotr; Pekalska, Elzbieta; Duin, Robert P. W., Outlier detection using ball descriptions with adjustable metric, (Proceedings of the 2006 Joint IAPR International Conference on Structural, Syntactic, and Statistical Pattern Recognition, SSPR’06/SPR’06 (2006), Springer-Verlag: Springer-Verlag Berlin, Heidelberg), 587-595
[54] Taylor, O.; MacIntyre, J., Adaptive local fusion systems for novelty detection and diagnostics in condition monitoring, Proc. SPIE - Int. Soc. Opt. Eng., 3376, 210-218 (1998)
[57] Wang, Xizhao; Wang, Yadong; Wang, Lijuan, Improving fuzzy c-means clustering based on feature-weight learning, Pattern Recogn. Lett., 25, 10, 1123-1132 (2004)
[58] Wilk, T.; Woźniak, M., Soft computing methods applied to combination of one-class classifiers, Neurocomputing, 75, 185-193 (January 2012)
[59] Windeatt, T.; Ghaderi, R., Coding and decoding strategies for multi-class learning problems, Inform. Fusion, 4, 1, 11-21 (2003)
[60] Woźniak, M.; Grana, M.; Corchado, E., A survey of multiple classifier systems as hybrid systems, Inform. Fusion, 16, 3-17 (2014)
[61] Xu, L.; Krzyzak, A.; Suen, C. Y., Methods of combining multiple classifiers and their applications to handwriting recognition, IEEE Trans. Syst, Man Cybernet., 22, 3, 418-435 (1992)
[62] Yeh, C.; Lee, Z.; Lee, S., Boosting one-class support vector machines for multi-class classification, Appl. Artif. Intell., 23, 4, 297-315 (2009)
[63] Yu, H.; Man, J.; Chang, K. C., Pebl: Web page classification without negative examples, IEEE Trans. Knowl. Data Eng., 16, 1, 70-81 (2004)
[64] Zhang, L.; Zhou, W.; Jiao, L., Kernel clustering algorithm, Jisuanji Xuebao/Chinese J. Comput., 25, 6, 587-590 (2002)
This reference list is based on information provided by the publisher or from digital mathematics libraries. Its items are heuristically matched to zbMATH identifiers and may contain data conversion errors. In some cases that data have been complemented/enhanced by data from zbMATH Open. This attempts to reflect the references listed in the original paper as accurately as possible without claiming completeness or a perfect matching.