×

zbMATH — the first resource for mathematics

Joint ranking SVM and binary relevance with robust low-rank learning for multi-label classification. (English) Zbl 1434.68474
Summary: Multi-label classification studies the task where each example belongs to multiple labels simultaneously. As a representative method, Ranking Support Vector Machine (Rank-SVM) aims to minimize the Ranking Loss and can also mitigate the negative influence of the class-imbalance issue. However, due to its stacking-style way for thresholding, it may suffer error accumulation and thus reduces the final classification performance. Binary Relevance (BR) is another typical method, which aims to minimize the Hamming Loss and only needs one-step learning. Nevertheless, it might have the class-imbalance issue and does not take into account label correlations. To address the above issues, we propose a novel multi-label classification model, which joints Ranking support vector machine and Binary Relevance with robust Low-rank learning (RBRL). RBRL inherits the ranking loss minimization advantages of Rank-SVM, and thus overcomes the disadvantages of BR suffering the class-imbalance issue and ignoring the label correlations. Meanwhile, it utilizes the hamming loss minimization and one-step learning advantages of BR, and thus tackles the disadvantages of Rank-SVM including another thresholding learning step. Besides, a low-rank constraint is utilized to further exploit high-order label correlations under the assumption of low dimensional label space. Furthermore, to achieve nonlinear multi-label classifiers, we derive the kernelization RBRL. Two accelerated proximal gradient methods (APG) are used to solve the optimization problems efficiently. Extensive comparative experiments with several state-of-the-art methods illustrate a highly competitive or superior performance of our method RBRL.
MSC:
68T05 Learning and adaptive systems in artificial intelligence
62H30 Classification and discrimination; cluster analysis (statistical aspects)
PDF BibTeX XML Cite
Full Text: DOI
References:
[1] Argyriou, A.; Evgeniou, T.; Pontil, M., Convex multi-task feature learning, Machine Learning, 73, 3, 243-272 (2008)
[2] Boutell, M. R.; Luo, J.; Shen, X.; Brown, C. M., Learning multi-label scene classification, Pattern Recognition, 37, 9, 1757-1771 (2004)
[3] Cai, J.-F.; Candès, E. J.; Shen, Z., A singular value thresholding algorithm for matrix completion, SIAM Journal on Optimization, 20, 4, 1956-1982 (2010) · Zbl 1201.90155
[4] Chang, C.-C.; Lin, C.-J., LIBSVM: A library for support vector machines, ACM Transactions on Intelligent Systems and Technology (TIST), 2, 27:1-27:27 (2011)
[5] Clare, A.; King, R. D., Knowledge discovery in multi-label phenotype data, (European conference on principles of data mining and knowledge discovery (2001), Springer), 42-53 · Zbl 1009.68730
[6] Cortes, C.; Mohri, M., AUC optimization vs. error rate minimization, (Advances in neural information processing systems (vol. 16) (2004)), 313-320
[7] Demšar, J., Statistical comparisons of classifiers over multiple data sets, Journal of Machine Learning Research (JMLR), 7, 1-30 (2006) · Zbl 1222.68184
[8] Dinuzzo, F., The representer theorem for Hilbert spaces: A necessary and sufficient condition, (Advances in neural information processing systems (vol. 25) (2012)), 189-196
[9] Elisseeff, A.; Weston, J., A kernel method for multi-labelled classification, (Advances in neural information processing systems (vol. 14) (2001)), 681-687
[10] Fan, R.-E.; Chang, K.-W.; Hsieh, C.-J.; Wang, X.-R.; Lin, C.-J., LIBLINEAR: A library for large linear classification, Journal of Machine Learning Research (JMLR), 9, 1871-1874 (2008) · Zbl 1225.68175
[11] Frank, M.; Wolfe, P., An algorithm for quadratic programming, Naval Research Logistics Quarterly, 3, 1-2, 95-110 (1956)
[12] Friedman, M., The use of ranks to avoid the assumption of normality implicit in the analysis of variance, Journal of the American Statistical Association, 32, 200, 675-701 (1937) · JFM 63.1098.02
[13] Fürnkranz, J.; Hüllermeier, E.; Loza Mencía, E.; Brinker, K., Multilabel classification via calibrated label ranking, Machine Learning, 73, 2, 133-153 (2008)
[14] Ghamrawi, N., & McCallum, A. (2005). Collective multi-label classification. In Proceedings of the 14th ACM international conference on information and knowledge management (pp. 195-200).
[15] Golub, G. H.; Van Loan, C. F., Matrix computations (1996), Johns Hopkins University, Press: Johns Hopkins University, Press Baltimore, MD, USA · Zbl 0865.65009
[16] Gopal, S.; Yang, Y., Multilabel classification with meta-level features, (Proceedings of the 33rd international ACM SIGIR conference on research and development in information retrieval (2010), ACM), 315-322
[17] Hou, P.; Geng, X.; Zhang, M.-L., Multi-label manifold learning, (Proceedings of the 30th AAAI conference on artificial intelligence (2016), AAAI Press), 1680-1686
[18] Hsu, C. -W., Chang, C. -C., & Lin, C. -J., et al. (2003). A practical guide to support vector classification. Taipei, Taiwan.
[19] Huang, J., Li, G., Huang, Q., & Wu, X. (2015). Learning label specific features for multi-label classification. In 2015 IEEE international conference on data mining (pp. 181-190).
[20] Huang, J.; Li, G.; Huang, Q.; Wu, X., Joint feature selection and classification for multilabel learning, IEEE Transactions on Cybernetics, 48, 3, 876-889 (2018)
[21] Jaggi, M. (2013). Revisiting Frank-Wolfe: Projection-free sparse convex optimization. In Proceedings of the 30th international conference on machine learning (pp. 427-435).
[22] Ji, S., & Ye, J. (2009). An accelerated gradient method for trace norm minimization. In Proceedings of the 26th international conference on machine learning (pp. 457-464).
[23] Jiang, A., Wang, C., & Zhu, Y. (2008). Calibrated rank-SVM for multi-label image categorization. In IEEE international joint conference on neural networks(pp. 1450-1455).
[24] Jing, L., Yang, L., Yu, J., & Ng, M. K. (2015). Semi-supervised low-rank mapping learning for multi-label classification. In Proceedings of the IEEE conference on computer vision and pattern recognition (pp. 1483-1491).
[25] Kivinen, J.; Smola, A.; Williamson, R., Learning with kernels, 2165-2176 (2002), MIT Press · Zbl 1369.68281
[26] Li, H.; Chen, N.; Li, L., Error analysis for matrix elastic-net regularization algorithms, IEEE Transactions on Neural Networks and Learning Systems, 23, 5, 737-748 (2012)
[27] McCallum, A. (1999). Multi-label text classification with a mixture model trained by EM. In AAAI’99 workshop on text learning (pp. 1-7).
[28] Nesterov, Y., Smooth minimization of non-smooth functions, Mathematical Programming, 103, 1, 127-152 (2005) · Zbl 1079.90102
[29] Qi, G. -J., Hua, X. -S., Rui, Y., Tang, J., Mei, T., & Zhang, H. -J. (2007). Correlative multi-label video annotation. In Proceedings of the 15th international conference on multimedia (pp. 17-26).
[30] Read, J.; Pfahringer, B.; Holmes, G.; Frank, E., Classifier chains for multi-label classification, Machine Learning, 85, 3, 333-359 (2011)
[31] Schapire, R. E.; Singer, Y., BoosTexter: A boosting-based system for text categorization, Machine Learning, 39, 2-3, 135-168 (2000) · Zbl 0951.68561
[32] Schölkopf, B.; Smola, A. J., Learning with kernels: support vector machines, regularization, optimization, and beyond (2002), MIT Press
[33] Shalev-Shwartz, S.; Ben-David, S., Understanding machine learning: From theory to algorithms (2014), Cambridge University Press · Zbl 1305.68005
[34] Trohidis, K., Tsoumakas, G., Kalliris, G., & Vlahavas, I. P. (2008). Multi-label classification of music into emotions. In ISMIR 2008, 9th international conference on music information retrieval (vol. 8) (pp. 325-330).
[35] Tsoumakas, G.; Katakis, I., Multi-label classification: An overview, International Journal of Data Warehousing and Mining, 3, 3 (2006)
[36] Tsoumakas, G.; Katakis, I.; Vlahavas, I., Mining multi-label data, (Data mining and knowledge discovery handbook (2009), Springer), 667-685
[37] Tsoumakas, G.; Katakis, I.; Vlahavas, I., Random k-labelsets for multilabel classification, IEEE Transactions on Knowledge and Data Engineering, 23, 7, 1079-1089 (2011)
[38] Tsoumakas, G.; Spyromitros-Xioufis, E.; Vilcek, J.; Vlahavas, I., Mulan: A java library for multi-label learning, Journal of Machine Learning Research (JMLR), 12, 2411-2414 (2011) · Zbl 1280.68207
[39] Ueda, N.; Saito, K., Parametric mixture models for multi-labeled text, (Advances in neural information processing systems (vol. 15) (2002)), 721-728
[40] Wang, H., Huang, H., & Ding, C. (2009). Image annotation using multi-label correlated Green’s function. In IEEE 12th international conference on computer vision (pp. 2029-2034).
[41] Wang, J., Yang, Y., Mao, J., Huang, Z., Huang, C., & Xu, W. (2016). Cnn-rnn: A unified framework for multi-label image classification. In Proceedings of the IEEE conference on computer vision and pattern recognition (pp. 2285-2294).
[42] Wu, Q.; Tan, M.; Song, H.; Chen, J.; Ng, M. K., ML-FOREST: A multi-label tree ensemble method for multi-label classification, IEEE Transactions on Knowledge and Data Engineering, 28, 10, 2665-2680 (2016)
[43] Wu, G.; Tian, Y.; Liu, D., Cost-sensitive multi-label learning with positive and negative label pairwise correlations, Neural Networks, 108, 411-423 (2018)
[44] Wu, G.; Tian, Y.; Zhang, C., A unified framework implementing linear binary relevance for multi-label learning, Neurocomputing, 289, 86-100 (2018)
[45] Wu, X. -Z., & Zhou, Z. -H. (2017). A unified view of multi-label performance measures. In Proceedings of the 34th international conference on machine learning (pp. 3780-3788).
[46] Xing, Y.; Yu, G.; Domeniconi, C.; Wang, J.; Zhang, Z., Multi-label co-training, (Proceedings of the 27th international joint conference on artificial intelligence (2018), AAAI Press), 2882-2888
[47] Xu, J., An efficient multi-label support vector machine with a zero label, Expert Systems with Applications, 39, 5, 4796-4804 (2012)
[48] Xu, J., A weighted linear discriminant analysis framework for multi-label feature extraction, Neurocomputing, 275, 107-120 (2018)
[49] Xu, C.; Liu, T.; Tao, D.; Xu, C., Local rademacher complexity for multi-label learning, IEEE Transactions on Image Processing, 25, 3, 1495-1507 (2016) · Zbl 1408.94734
[50] Xu, L., Wang, Z., Shen, Z., Wang, Y., & Chen, E. (2014). Learning low-rank label correlations for multi-label classification with missing labels. In 2014 IEEE international conference on data mining (pp. 1067-1072).
[51] Yu, H. -F., Jain, P., Kar, P., & Dhillon, I. (2014). Large-scale multi-label learning with missing labels. In Proceedings of the 31th international conference on machine learning (pp. 593-601).
[52] Yu, K.; Yu, S.; Tresp, V., Multi-label informed latent semantic indexing, (Proceedings of the 28th annual international ACM SIGIR conference on research and development in information retrieval (2005), ACM), 258-265
[53] Zhang, M. -L., Li, Y. -K., & Liu, X. -Y. (2015). Towards class-imbalance aware multi-label learning. In Proceedings of the 24th international joint conference on artificial intelligence (pp. 4041-4047).
[54] Zhang, M.-L.; Wu, L., LIFT: Multi-label learning with label-specific features, IEEE Transactions on Pattern Analysis and Machine Intelligence, 37, 1, 107-120 (2015)
[55] Zhang, Q.; Zhong, Y.; Zhang, M., Feature-induced labeling information enrichment for multi-label learning, (Proceedings of the 32nd AAAI conference on artificial intelligence (2018), AAAI Press), 4446-4453
[56] Zhang, M.-L.; Zhou, Z.-H., Multilabel neural networks with applications to functional genomics and text categorization, IEEE Transactions on Knowledge and Data Engineering, 18, 10, 1338-1351 (2006)
[57] Zhang, M.-L.; Zhou, Z.-H., ML-KNN: A lazy learning approach to multi-label learning, Pattern Recognition, 40, 7, 2038-2048 (2007) · Zbl 1111.68629
[58] Zhang, M.-L.; Zhou, Z.-H., A review on multi-label learning algorithms, IEEE Transactions on Knowledge and Data Engineering, 26, 8, 1819-1837 (2014)
[59] Zhen, X.; Yu, M.; He, X.; Li, S., Multi-target regression via robust low-rank learning, IEEE Transactions on Pattern Analysis and Machine Intelligence, 40, 2, 497-504 (2018)
[60] Zhou, Z.-H., Ensemble methods: Foundations and algorithms (2012), CRC Press
[61] Zhu, S.; Ji, X.; Xu, W.; Gong, Y., Multi-labelled classification using maximum entropy method, (Proceedings of the 28th annual international ACM SIGIR conference on research and development in information retrieval (2005), ACM), 274-281
[62] Zhu, Y.; Kwok, J. T.; Zhou, Z.-H., Multi-label learning with global and local label correlation, IEEE Transactions on Knowledge and Data Engineering, 30, 6, 1081-1094 (2018)
This reference list is based on information provided by the publisher or from digital mathematics libraries. Its items are heuristically matched to zbMATH identifiers and may contain data conversion errors. It attempts to reflect the references listed in the original paper as accurately as possible without claiming the completeness or perfect precision of the matching.