×

zbMATH — the first resource for mathematics

Distributed online semi-supervised support vector machine. (English) Zbl 1441.68210
Summary: Recently, the research on semi-supervised support vector machine (S\(^3\)VM) has received much attention, and many S\(^3\)VM algorithms have been proposed. Existing studies have shown that S\(^3\)VM is effective especially in the situations where labeled data is scarce. Nevertheless, most of existing S\(^3\)VM algorithms belong to centralized learning, that is, all the data is stored and processed at a fusion center. In many real-world applications, data may be horizontally or vertically distributed over multiple nodes (parties). Besides, from the concerns of privacy and security, each node would not like to share its original data with the others. On the other hand, considering that the data is usually sequentially generated, online processing is preferred. In this paper, we propose two online distributed S\(^3\)VM (dS\(^3\)VM) algorithms, which are respectively used for horizontally and vertically partitioned data classification. In these two algorithms, to get a fully decentralized implementation, we propose a new form of manifold regularization defined on some anchor points that are adaptively selected by an online strategy. Besides, we use the sparse random feature map to approximate the kernel feature map. In this manner, the model parameters can be collaboratively estimated without transmitting the original data between neighbors. The convergence performances of the proposed algorithms are analyzed. Simulations on several data sets are performed. Results show that the proposed dS\(^3\)VM algorithms achieve good classification performance even when there is only a small portion of labeled data.
MSC:
68T05 Learning and adaptive systems in artificial intelligence
62H30 Classification and discrimination; cluster analysis (statistical aspects)
68W27 Online algorithms; streaming algorithms
Software:
Pegasos
PDF BibTeX XML Cite
Full Text: DOI
References:
[1] Belkin, M.; Niyogi, P.; Sindhwani, V., Manifold regularization: A geometric framework for learning from labeled and unlabeled examples, J. Mach. Learn. Res., 7, 11, 2399-2434 (2006) · Zbl 1222.68144
[3] Cattivelli, F. S.; Sayed, A. H., Diffusion LMS strategies for distributed estimation, IEEE Trans. Signal Process., 58, 3, 1035-1048 (2010) · Zbl 1392.94123
[4] Chang, T. H.; Hong, M.; Wang, X., Multi-agent distributed optimization via inexact consensus ADMM, IEEE Trans. Signal Process., 63, 2, 482-497 (2015) · Zbl 1393.90124
[5] Dong, A.; Chung, F.l.; Deng, Z.; Wang, S., Semi-supervised svm with extended hidden features, IEEE Trans. Cyber., 46, 12, 2924-2937 (2016)
[6] Fierimonte, R.; Scardapane, S.; Uncini, A.; Panella, M., Fully decentralized semi-supervised learning via privacy-preserving matrix completion, IEEE Trans. Neural Netw. Learn. Syst., 28, 11, 2699-2711 (2016)
[7] Forero, P. A.; Cano, A.; Giannakis, G. B., Consensus-based distributed support vector machines, J. Mach. Learn. Res., 11, 3, 1663-1707 (2010) · Zbl 1242.68222
[8] Forestier, G.; Wemmert, C., Semi-supervised learning using multiple clusterings with limited labeled data, Inf. Sci., 361-362, 20, 48-65 (2016)
[9] Geary, A.; Bertsekas, D. P., Incremental subgradient methods for nondifferentiable optimization, Proceedings of IEEE Conference on Decision and Control, 907-912 (1999)
[10] Huang, G.; Song, S.; Gupta, J. N.; Wu, C., Semi-supervised and unsupervised extreme learning machines, IEEE Trans. Cyber., 44, 12, 2405-2417 (2014)
[11] Li, P.; Hastie, T. J.; Church, K. W., Very sparse random projections, Proceedings of 12th ACM SIGKDD, 287-296 (2006)
[12] Lin, K. P.; Chen, M. S., On the design and analysis of the privacy-preserving SVM classifier, IEEE Trans. Knowl. Data Eng., 23, 11, 1704-1717 (2011)
[13] Luo, P.; Xiong, H.; Lü, K.; Shi, Z., Distributed classification in peer-to-peer networks, Proceedings of 13th ACM SIGKDD, 968-976 (2007)
[16] Ouyang, H.; He, N.; Long, Q. T.; Gray, A., Stochastic alternating direction method of multipliers, Proceedings of International Conference on Machine Learning, 80-88 (2013)
[19] Shalev-Shwartz, S.; Singer, Y.; Srebro, N., Pegasos:primal estimated sub-gradient solver for SVM, Proceedings of International Conference on Machine Learning, 807-814 (2007)
[20] Shen, P.; Li, C., Distributed information theoretic clustering, IEEE Trans. Signal Process., 62, 13, 3442-3453 (2014) · Zbl 1394.94805
[21] Shi, W.; Ling, Q.; Yuan, K.; Wu, G., On the linear convergence of the ADMM in decentralized consensus optimization, IEEE Trans. Signal Process., 62, 7, 1750-1761 (2014) · Zbl 1394.94532
[22] Sreekanth, V.; Vedaldi, A.; Zisserman, A.; Jawahar, C., Generalized RBF feature maps for efficient detection, Proceedings of Conference on British Machine Vision, 1-11 (2010)
[23] Sun, L.; Mu, W. S.; Qi, B.; Zhou, Z. J., A new privacy-preserving proximal support vector machine for classification of vertically partitioned data, Int. J. Mach. Learn. Cyber., 6, 1, 109-118 (2015)
[24] Tu, E.; Zhang, Y.; Zhu, L.; Yang, J.; Kasabov, N., A graph-based semi-supervised k nearest-neighbor method for nonlinear manifold distributed data classification, Inf. Sci., 367-368, 1, 673-688 (2016) · Zbl 1428.68259
[25] Vedaldi, A.; Zisserman, A., Efficient additive kernels via explicit feature maps, IEEE Trans. Pattern Anal. Mach. Intell., 34, 3, 480-492 (2012)
[26] Wang, H.; Banerjee, A., Online alternating direction method (longer version), Mathematics, 45, 2, 288-290 (2013)
[27] Yu, H.; Jiang, X.; Vaidya, J., Privacy-preserving SVM using nonlinear kernels on horizontally partitioned data, Proceedings of ACM Symposium on Applied Computing, 603-610 (2006)
[28] Yu, H.; Vaidya, J.; Jiang, X., Privacy-Preserving SVM Classification on Vertically Partitioned Data (2006), Springer Berlin Heidelberg Press
[29] Yuan, X. T.; Wang, Z.; Deng, J.; Liu, Q., Efficient \(χ^2\) kernel linearization via random feature maps, IEEE Trans. Neural Netw. Learn. Syst., 574, 2, 108-110 (2015)
[30] Yoo, J.; Kim, H. J., Online estimation using semi-supervised least square SVR, Proceedings of International Conference on Systems, Man and Cybernetics, San Diego, CA, 1624-1629 (2014)
[31] Zheng, H.; Kulkarni, S. R.; Poor, H. V., Attribute-distributed learning: models, limits, and algorithms, IEEE Trans. Signal process., 59, 1, 386-398 (2011) · Zbl 1392.94572
[32] Zhang, T., Solving large scale linear prediction problems using stochastic gradient descent algorithms, Proceedings of ACM International Conference on Machine Learning, 919-926 (2004)
This reference list is based on information provided by the publisher or from digital mathematics libraries. Its items are heuristically matched to zbMATH identifiers and may contain data conversion errors. It attempts to reflect the references listed in the original paper as accurately as possible without claiming the completeness or perfect precision of the matching.