×

Parsimonious kernel extreme learning machine in primal via Cholesky factorization. (English) Zbl 1414.68091

Summary: Recently, extreme learning machine (ELM) has become a popular topic in machine learning community. By replacing the so-called ELM feature mappings with the nonlinear mappings induced by kernel functions, two kernel ELMs, i.e., P-KELM and D-KELM, are obtained from primal and dual perspectives, respectively. Unfortunately, both P-KELM and D-KELM possess the dense solutions in direct proportion to the number of training data. To this end, a constructive algorithm for P-KELM (CCP-KELM) is first proposed by virtue of Cholesky factorization, in which the training data incurring the largest reductions on the objective function are recruited as significant vectors. To reduce its training cost further, PCCP-KELM is then obtained with the application of a probabilistic speedup scheme into CCP-KELM. Corresponding to CCP-KELM, a destructive P-KELM (CDP-KELM) is presented using a partial Cholesky factorization strategy, where the training data incurring the smallest reductions on the objective function after their removals are pruned from the current set of significant vectors. Finally, to verify the efficacy and feasibility of the proposed algorithms in this paper, experiments on both small and large benchmark data sets are investigated.

MSC:

68T05 Learning and adaptive systems in artificial intelligence

Software:

RSVM; OP-ELM
PDFBibTeX XMLCite
Full Text: DOI

References:

[2] An, S.; Liu, W.; Venkatesh, S., Fast cross-validation algorithms for least squares support vector machine and kernel ridge regression, Pattern Recognition, 40, 8, 2154-2162 (2007), [Online]. Available: URL: http://dx.doi.org/10.1016/j.patcog.2006.12.015 · Zbl 1115.68125
[3] Bi, X.; Zhao, X.; Wang, G.; Zhang, P.; Wang, C., Distributed extreme learning machine with kernels based on mapreduce, Neurocomputing, 149, Part A, 456-463 (2015), [Online]. Available: URL: http://www.sciencedirect.com/science/article/pii/S0925231214011473
[4] Bottou, L.; Chapelle, O.; Decoste, D.; Weston, J., Training a support vector machine in the primal, Neural Computation, 19, 5, 1155-1178 (2007) · Zbl 1123.68101
[5] Boyd, S.; Vandenberghe, L., Convex optimization (2004), Cambridge university press · Zbl 1058.90049
[6] Castano, A.; Fernandez-Navarro, F.; Hervas-Martinez, C., Pca-elm: a robust and pruned extreme learning machine approach based on principal component analysis, Neural Processing Letters, 37, 3, 377-392 (2013), [Online]. Available: URL: http://dx.doi.org/10.1007/s11063-012-9253-x
[7] Cortes, C.; Vapnik, V., Support-vector networks, Machine Learning, 20, 3, 273-297 (1995), [Online]. Available: URL: http://dx.doi.org/10.1023/A:1022627411411 · Zbl 0831.68098
[9] Deng, W.-Y.; Zheng, Q.-H.; Wang, Z.-M., Cross-person activity recognition using reduced kernel extreme learning machine, Neural Networks, 53, 1-7 (2014), [Online]. Available: URL: http://dx.doi.org/10.1016/j.neunet.2014.01.008
[10] Feng, G.; Huang, G.-B.; Lin, Q.; Gay, R., Error minimized extreme learning machine with growth of hidden nodes and incremental learning, IEEE Transactions on Neural Networks, 20, 8, 1352-1357 (2009), [Online]. Available: URL: http://dx.doi.org/10.1109/TNN.2009.2024147
[11] Fu, H.; Vong, C.-M.; Wong, P.-K.; Yang, Z., Fast detection of impact location using kernel extreme learning machine, Neural Computing and Applications (2015), [Online]. Available: URL: http://dx.doi.org/10.1007/s00521-014-1568-2
[12] Huang, G.-B.; Chen, L., Convex incremental extreme learning machine, Neurocomputing, 70, 16-18, 3056-3062 (2007), [Online]. Available: URL: http://dx.doi.org/10.1016/j.neucom.2007.02.009
[13] Huang, G.-B.; Chen, L., Enhanced random search based incremental extreme learning machine, Neurocomputing, 71, 16-18, 3460-3468 (2008), [Online]. Available: URL: http://dx.doi.org/10.1016/j.neucom.2007.10.008
[14] Huang, G.-B.; Chen, L.; Siew, C.-K., Universal approximation using incremental constructive feedforward networks with random hidden nodes, IEEE Transactions on Neural Networks, 17, 4, 879-892 (2006), [Online]. Available: URL: http://dx.doi.org/10.1109/TNN.2006.875977
[15] Huang, G.-B.; Ding, X.; Zhou, H., Optimization method based extreme learning machine for classification, Neurocomputing, 74, 1-3, 155-163 (2010), [Online]. Available: URL: http://dx.doi.org/10.1016/j.neucom.2010.02.019
[16] Huang, G.; Huang, G.-B.; Song, S.; You, K., Trends in extreme learning machines: a review, Neural Networks, 61, 32-48 (2015), [Online]. Available: URL: http://dx.doi.org/10.1016/j.neunet.2014.10.001 · Zbl 1325.68190
[17] Huang, G.-B.; Wang, D. H.; Lan, Y., Extreme learning machines: a survey, International Journal of Machine Learning and Cybernetics, 2, 2, 107-122 (2011), [Online]. Available: URL: http://dx.doi.org/10.1007/s13042-011-0019-y
[18] Huang, G.-B.; Zhou, H.; Ding, X.; Zhang, R., Extreme learning machine for regression and multiclass classification, IEEE Transactions on Systems, Man, and Cybernetics, Part B: Cybernetics, 42, 2, 513-529 (2012), [Online]. Available: URL: http://dx.doi.org/10.1109/TSMCB.2011.2168604
[20] Huang, G.-B.; Zhu, Q.-Y.; Siew, C.-K., Extreme learning machine: theory and applications, Neurocomputing, 70, 1-3, 489-501 (2006), [Online]. Available: URL: http://dx.doi.org/10.1016/j.neucom.2005.12.126
[21] Iosifidis, A.; Gabbouj, M., On the kernel extreme learning machine speedup, Pattern Recognition Letters, 68, Part 1, 205-210 (2015), [Online]. Available: URL: http://www.sciencedirect.com/science/article/pii/S0167865515003256
[22] Iosifidis, A.; Tefas, A.; Pitas, I., On the kernel extreme learning machine classifier, Pattern Recognition Letters, 54, 11-17 (2015), [Online]. Available: URL: http://dx.doi.org/10.1016/j.patrec.2014.12.003
[24] Li, X.; Mao, W.; Jiang, W., Fast sparse approximation of extreme learning machine, Neurocomputing, 128, 96-103 (2014), [Online]. Available: URL: http://www.sciencedirect.com/science/article/pii/S0925231213010047
[26] Liu, X.; Wang, L.; Huang, G.-B.; Zhang, J.; Yin, J., Multiple kernel extreme learning machine, Neurocomputing, 149, Part A, 253-264 (2015), [Online]. Available: URL: http://www.sciencedirect.com/science/article/pii/S0925231214011199
[27] Miche, Y.; Sorjamaa, A.; Bas, P.; Simula, O.; Jutten, C.; Lendasse, A., Op-elm: optimally pruned extreme learning machine, IEEE Transactions on Neural Networks, 21, 1, 158-162 (2010), [Online]. Available: URL: http://dx.doi.org/10.1109/TNN.2009.2036259
[29] Nair, P. B.; Choudhury, A.; Keane, A. J., Some greedy learning algorithms for sparse regression and classification with mercer kernels, Journal of Machine Learning Research, 3, 4-5, 781-801 (2003) · Zbl 1089.68602
[30] Rong, H.-J.; Ong, Y.-S.; Tan, A.-H.; Zhu, Z., A fast pruned-extreme learning machine for classification problem, Neurocomputing, 72, 1-3, 359-366 (2008), [Online]. Available: URL: http://dx.doi.org/10.1016/j.neucom.2008.01.005
[31] Rumelhart, D. E.; Hinton, G. E.; Williams, R. J., Learning representation by back-propagating errors, Nature, 323, 533-536 (1986) · Zbl 1369.68284
[32] Scardapane, S.; Comminiello, D.; Scarpiniti, M.; Uncini, A., Online sequential extreme learning machine with kernels, IEEE Transactions on Neural Networks and Learning Systems, 26, 9, 2214-2220 (2015), [Online]. Available: URL: http://dx.doi.org/10.1109/TNNLS.2014.2382094
[34] Schölkopf, B.; Smola, A. J., Learning with kernels: support vector machines, regularization, optimization, and beyond (2002), MIT Press
[35] Shamshirband, S.; Mohammadi, K.; Chen, H.-L.; Narayana Samy, G.; Petkovi, D.; Ma, C., Daily global solar radiation prediction from air temperatures using kernel extreme learning machine: a case study for iran, Journal of Atmospheric and Solar-Terrestrial Physics, 134, 109-117 (2015), [Online]. Available: URL: http://dx.doi.org/10.1016/j.jastp.2015.09.014
[37] Suykens, J.; Vandewalle, J., Least squares support vector machine classifiers, Neural Processing Letters, 9, 3, 293-300 (1999)
[38] Vapnik, V. N., The nature of statistical learning theory (1995), Springer-Verlag New York, Inc.: Springer-Verlag New York, Inc. New York, NY, USA · Zbl 0833.62008
[39] Vapnik, V., An overview of statistical learning theory, IEEE Transactions on Neural Networks, 10, 5, 988-999 (1999)
[40] Wang, X.; Han, M., Online sequential extreme learning machine with kernels for nonstationary time series prediction, Neurocomputing, 145, 90-97 (2014), [Online]. Available: URL: http://dx.doi.org/10.1016/j.neucom.2014.05.068
[41] Wang, H.-Q.; Sun, F.-C.; Cai, Y.-N.; Ding, L.-G.; Chen, N., An unbiased lssvm model for classification and regression, Soft Computing, 14, 2, 171-180 (2010), [Online]. Available: URL: http://dx.doi.org/10.1007/s00500-009-0435-z · Zbl 1191.68604
[42] Wei, Y.; Xiao, G.; Deng, H.; Chen, H.; Tong, M.; Zhao, G.; Liu, Q., Hyperspectral image classification using FPCA-based kernel extreme learning machine, Optik—International Journal for Light and Electron Optics, 126, 23, 3942-3948 (2015), [Online]. Available: URL: http://www.sciencedirect.com/science/article/pii/S0030402615007378
[43] Wong, P. K.; Wong, K. I.; Vong, C. M.; Cheung, C. S., Modeling and optimization of biodiesel engine performance using kernel-based extreme learning machine and cuckoo search, Renewable Energy, 74, 640-647 (2015), [Online]. Available: URL: http://www.sciencedirect.com/science/article/pii/S096014811400545X
[45] Yang, Y.; Wang, Y.; Yuan, X., Bidirectional extreme learning machine for regression problem and its learning effectiveness, IEEE Transactions on Neural Networks and Learning Systems, 23, 9, 1498-1505 (2012), [Online]. Available: URL: http://dx.doi.org/10.1109/TNNLS.2012.2202289
[46] Ye, Y.; Qin, Y., Qr factorization based incremental extreme learning machine with growth of hidden nodes, Pattern Recognition Letters, 65, 177-183 (2015), [Online]. Available: URL: http://dx.doi.org/10.1016/j.patrec.2015.07.031
[47] Zhang, X., Matrix analysis and applications (2004), Tsinghua university press
[48] Zhao, Y.-P.; Huerta, R., Improvements on parsimonious extreme learning machine using recursive orthogonal least squares, Neurocomputing, 191, 82-94 (2016), [Online]. Available: URL: http://www.sciencedirect.com/science/article/pii/S0925231216000527
[49] Zhao, Y.-P.; Li, B.; Li, Y.-B., An accelerating scheme for destructive parsimonious extreme learning machine, Neurocomputing, 167, 671-687 (2015), [Online]. Available: URL: http://dx.doi.org/10.1016/j.neucom.2015.04.002
[50] Zhao, Y.; Sun, J., Robust support vector regression in the primal, Neural Networks, 21, 10, 1548-1555 (2008), [Online]. Available: URL: http://dx.doi.org/10.1016/j.neunet.2008.09.001 · Zbl 1254.68236
[51] Zhao, Y.; Sun, J., Recursive reduced least squares support vector regression, Pattern Recognition, 42, 5, 837-842 (2009), [Online]. Available: URL: http://dx.doi.org/10.1016/j.patcog.2008.09.028 · Zbl 1162.68645
[52] Zhao, Y.-P.; Wang, K.-K.; Li, Y.-B., Parsimonious regularized extreme learning machine based on orthogonal transformation, Neurocomputing, 156, 280-296 (2015), [Online]. Available: URL: http://dx.doi.org/10.1016/j.neucom.2014.12.046
[53] Zhou, S., Sparse LSSVM in primal using cholesky factorization for large-scale problems, IEEE Transactions on Neural Networks and Learning Systems, 27, 4, 783-795 (2016)
[54] Zhou, X.-R.; Wang, C.-S., Cholesky factorization based online regularized and kernelized extreme learning machines with forgetting mechanism, Neurocomputing, 174, Part B, 1147-1155 (2016), [Online]. Available: URL: http://www.sciencedirect.com/science/article/pii/S0925231215014927
This reference list is based on information provided by the publisher or from digital mathematics libraries. Its items are heuristically matched to zbMATH identifiers and may contain data conversion errors. In some cases that data have been complemented/enhanced by data from zbMATH Open. This attempts to reflect the references listed in the original paper as accurately as possible without claiming completeness or a perfect matching.