Multi-target support vector regression via correlation regressor chains. (English) Zbl 1443.62160

Summary: Multi-target regression is a challenging task that consists of creating predictive models for problems with multiple continuous target outputs. Despite the increasing attention on multi-label classification, there are fewer studies concerning multi-target (MT) regression. The current leading MT models are based on ensembles of regressor chains, where random, differently ordered chains of the target variables are created and used to build separate regression models, using the previous target predictions in the chain. The challenges of building MT models stem from trying to capture and exploit possible correlations among the target variables during training. This paper presents three multi-target support vector regression models. The first involves building independent, single-target Support Vector Regression (SVR) models for each output variable. The second builds an ensemble of random chains using the first method as a base model. The third calculates the targets’ correlations and forms a maximum correlation chain, which is used to build a single chained support vector regression model, improving the models’ prediction performance while reducing the computational complexity. The experimental study evaluates and compares the performance of the three approaches with seven other state-of-the-art multi-target regressors on 24 multi-target datasets. The experimental results are then analyzed using non-parametric statistical tests. The results show that the maximum correlation SVR approach improves the performance of using ensembles of random chains.


62H12 Estimation in multivariate analysis
62J02 General nonlinear regression
62G10 Nonparametric hypothesis testing


Full Text: DOI


[1] Aho, T.; Zenko, B.; Dzeroski, S.; Elomaa, T., Multi-target regression with rule ensembles, J. Mach. Learn. Res., 13, 2267-2407 (2012)
[2] Appice, A.; Dzeroski, S., Stepwise induction of multi-target model trees, European Conference of Machine Learning. European Conference of Machine Learning, Lecture Notes on Artificial Intelligence, vol. 4701, 502-509 (2007)
[3] Baxter, J., A Bayesian/information theoretic model of learning to learn via multiple task sampling, Mach. Learn., 28, 7-39 (1997)
[4] Ben-David, S.; Schuller, R., Exploiting task relatedness for multiple task learning, Proceedings of the Sixteenth Annual Conference on Learning Theory, 567-580 (2003)
[5] Borchani, H.; Varando, G.; Bielza, C.; Larrañaga, P., A survey on multi-output regression, WIREs Data Min. Knowl. Discov., 5, 216-233 (2015)
[6] Boyd, S.; Vandenberghe, L., Convex Optimization (2004), Cambridge University Press
[7] Breiman, L., Bagging predictors, Mach. Learn., 24, 123-140 (1996)
[8] Cano, A.; Luna, J.; Gibaja, E.; Ventura, S., LAIM discretization for multi-label data, Inf. Sci., 330, 370-384 (2016)
[9] Caruana, R., Multitask learning, Mach. Learn., 28, 41-75 (1997)
[10] Chang, C. C.; Lin, C. J., LIBSVM: a library for support vector machines, ACM Trans. Intell. Syst. Technol., 2, 27:1-27:27 (2011)
[11] Charte, F.; Rivera, A.; Jesus, M. D.; Herrera, F., LI-MLC: a label inference methodology for addressing high dimensionality in the label space for multilabel classification, IEEE Trans. Neural Netw. Learn Syst., 25, 1842-1854 (2014)
[12] Chen, J.; Pedrycz, W.; Ha, M.; Ma, L., Set-valued samples based support vector regression and its applications, Expert Syst. Appl., 42, 2502-2509 (2015)
[13] Cortes, C.; Vapnik, V., The Nature of Statistical Learning Theory (1995), Springer: Springer New York
[14] Derrac, J.; García, S.; Molina, D.; Herrera, F., A practical tutorial on the use of nonparametric statistical tests as a methodology for comparing evolutionary and swarm intelligence algorithms, Swarm Evol. Comput., 1, 3-18 (2011)
[15] Ding, Y.; Cheng, L.; Pedrycz, W.; Hao, K., Global nonlinear kernel prediction for large data set with a particle swarm-optimized interval support vector regression, IEEE Trans. Neural Netw. Learn. Syst., 26, 2521-2534 (2015)
[16] Drucker, H.; Burges, C.; Kaufman, L.; Smola, A.; Vapnik, V., Support vector regression machines, Proceedings of the Advances in Neural Information Processing Systems, 155-161 (1997)
[17] Dunn, O., Multiple comparisons among means, J. Am. Stat. Assoc., 56, 52-64 (1961)
[18] Friedman, J.; Hastie, T.; Tibshirani, R., Regularization paths for generalized linear models via coordinate descent, J. Stat. Softw., 33, 1-22 (2010)
[19] García, S.; Fernández, A.; Luengo, J.; Herrera, F., Advanced nonparametric tests for multiple comparisons in the design of experiments in computational intelligence and data mining: experimental analysis of power, Inf. Sci., 180, 2044-2064 (2010)
[20] García, S.; Herrera, F., An extension on statistical comparisons of classifiers over multiple data sets for all pairwise comparisons, J. Mach. Learn. Res., 9, 2677-2694 (2008)
[21] García, S.; Molina, D.; Lozano, M.; Herrera, F., A study on the use of non-parametric tests for analyzing the evolutionary algorithms’ behaviour - a case study on the CEC2005 special session on real parameter optimization, Heuristics, 15, 617-644 (2008)
[22] Hadavandi, E.; Shahrabi, J.; Shamishirband, S., A novel boosted-neural network ensemble for modeling multi-target regression problems, Eng. Appl. Artif. Intell., 45, 204-219 (2015)
[23] Hadavandi, E.; Shahrabi, K.; Hayashi, Y., SPMoe: a novel subspace-projected mixture of experts model for multi-target regression problems, Soft Comput., 20, 2047-2065 (2016)
[24] He, J.; Gu, H.; Wang, Z., Multi-instance multi-label learning based on gaussian process with application to visual mobile robot navigation, Inf. Sci., 190, 162-177 (2011)
[25] Jeong, M.; Lee, G., Multi-domain spoken language understanding with transfer learning, Speech Commun., 51, 412-424 (2009)
[26] Kecman, V., Learning and Soft Computing: Support Vector Machines, Neural Networks, and Fuzzy Logic Models (2001), MIT Press
[27] Kocev, D.; Ceci, M., Ensembles of Extremely Randomized Trees for Multi-target Regression, Lecture Notes in Computer Science, vol. 9356, 86-100 (2015)
[28] Kocev, D.; Dzeroski, S.; White, M.; Newell, G.; Griffioen, P., Using single- and multi-target regression trees and ensembles to model a compound index of vegetation condition, Ecol. Modell., 20, 1159-1168 (2009)
[29] Kocev, D.; Vens, C.; Struyf, J.; Dzeroski, S., Ensembles of Multi-Objective Decision Trees, pp.86-100 (2007), Springer: Springer Heidelberg
[30] Kocev, D.; Vens, C.; Struyf, J.; Dzeroski, S., Tree ensembles for predicting structured outputs, Pattern Recognit., 43, 817-833 (2013)
[31] Lee, J.; Kim, D.-W., Memetic feature selection algorithm for multi-label classification, Inf. Sci., 293, 80-96 (2015)
[32] Li, H.; Li, D.; Zhai, Y.; Wang, S.; Zhang, J., A novel attribute reduction approach for multi-label data based on rough set theory, Inf. Sci., 367-368, 827-847 (2016)
[34] Liu, Q.; Xu, Q.; Zheng, V.; Xue, H.; Cao, Z.; Yang, Q., Multi-task learning for cross-platform sirna efficacy prediction: an in-silico study, BMC Bioinf., 11, 181-196 (2010)
[35] Melki, G.; Kecman, V., Speeding up online training of l1 support vector machines, IEEE SoutheastConf (2016)
[36] Pérez, F.; Camps, G.; Soria, E.; Pérez, J.; Figueiras, A.; Artés, A., Multi-dimensional function approximation and regression estimation, Artif. Neural Netw., 757-762 (2002)
[37] Qian, B.; Wang, X.; Ye, J.; Davidson, I., A reconstruction error based framework for multi-label and multi-view learning, IEEE Trans. Knowl. Data Eng., 27, 594-607 (2015)
[38] Read, J.; Bielza, C.; Larranaga, P., Multi-dimensional classification with super-classes, IEEE Trans. Knowl. Data Eng., 26, 1720-1733 (2014)
[39] Spyromitros-Xioufis, E.; Tsoumakas, G.; Groves, W.; Vlahavas, I., Multi-Label Classification Methods for Multi-Target Regression, Cornell University Library (2014)
[40] Spyromitros-Xioufis, E.; Tsoumakas, G.; Groves, W.; Vlahavas, I., Multi-target regression via input space expansion: treating targets as inputs, Mach. Learn., 104, 55-98 (2016)
[41] Tsoumakas, G.; Spyromitros-Xioufis, E.; Vilcek, J.; Vlahavas, I., Mulan: a java library for multi-label learning, J. Mach. Learn. Res., 12, 2411-2414 (2011)
[42] Tsoumakas, G.; Spyromitros-Xioufis, E.; Vrekou, A.; Vlahavas, I., Multi-target regression via random linear target combinations, Mach. Learn. Knowl. Discov. Databases, 8726, 225-240 (2014)
[43] Wilcoxon, F., Individual comparisons by ranking methods, Biometr. Bull., 1, 80-83 (1945)
[44] Witten, I.; Frank, E.; Hall, M., Data Mining: Practical Machine Learning Tools and Techniques (2011), Morgan Kaufmann
[45] Wu, Q.; Ye, Y.; Zhang, H.; Chow, T.; Ho, S.-S., ML-TREE: a tree-structure-based approach to multilabel learning, IEEE Trans. Neural Netw. Learn. Syst., 26, 430-443 (2015)
[46] Xiong, T.; Bao, Y.; Hu, Z., Multiple-output support vector regression with a firefly algorithm for interval-valued stock price index forecasting, Knowl. Based Syst., 55, 87-100 (2014)
[47] Xu, S.; An, X.; Qiao, X.; Zhu, L.; Li, L., Multi-output least-squares support vector regression machines, Pattern Recognit., 34, 1078-1084 (2013)
[48] Zhang, M.-L.; Zhou, Z.-H., A review on multi-label learning algorithms, IEEE Trans. Knowl. Data Eng., 26, 1819-1837 (2014)
[49] Zhang, W.; Liu, X.; Shi, D., Multi-output LS-SVR machine in extended feature space, Proceedings of the 2012 IEEE International Conference on Computational Intelligence for Measurement Systems and Applications, 130-134 (2012)
[50] Zhao, Y.-P.; Wang, K.-K.; Li, F., A pruning method of refining recursive reduced least squares support vector regression, Inf. Sci., 296, 160-174 (2015)
[51] Zhu, F.; Yang, J.; Gao, J.; Xu, C.; Xu, S.; Gao, C., Finding the samples near the decision plane for support vector learning, Inf. Sci., 382-383, 292-307 (2017)
This reference list is based on information provided by the publisher or from digital mathematics libraries. Its items are heuristically matched to zbMATH identifiers and may contain data conversion errors. It attempts to reflect the references listed in the original paper as accurately as possible without claiming the completeness or perfect precision of the matching.