×

Semi-supervised partial least squares. (English) Zbl 1455.68168

Summary: Traditional supervised dimensionality reduction methods can establish a better model often under the premise of a large number of samples. However, in real-world applications where labeled data are scarce, traditional methods tend to perform poorly because of overfitting. In such cases, unlabeled samples could be useful in improving the performance. In this paper, we propose a semi-supervised dimensionality reduction method by using partial least squares (PLS) which we call semi-supervised partial least squares (S2PLS). To combine the labeled and unlabeled samples into a S2PLS model, we first apply the PLS algorithm to unsupervised dimensionality reduction. Then, the final S2PLS model is established by ensembling the supervised PLS model and the unsupervised PLS model which using the basic idea of principal model analysis (PMA) method. Compared with unsupervised or supervised dimensionality reduction algorithms, S2PLS not only can improve the prediction accuracy of the samples but also enhance the generalization ability of the model. Meanwhile, it can obtain better results even there are only a few or no labeled samples. Experimental results on five UCI data sets also confirmed the above properties of S2PLS algorithm.

MSC:

68T05 Learning and adaptive systems in artificial intelligence
62H30 Classification and discrimination; cluster analysis (statistical aspects)
68T10 Pattern recognition, speech recognition
PDFBibTeX XMLCite
Full Text: DOI

References:

[1] Barker, M. and Rayens, W., Partial least squares for discrimination, J. Chem. A J. Chem. Soc.17(3) (2003) 166-173.
[2] Bayne, C. K., Latent variable path modeling with partial least squares, Technometrics34 (1992) 110-111.
[3] S. Ben-David, Lu, T. and Pál, D., Does unlabeled data provably help? worst-case analysis of the sample complexity of semi-supervised learning, in 21st Annual Conf. on Learning Theory (COLT), Finland: Helsinki (2008), pp. 33-44.
[4] Boulesteix, A.-L., Pls dimension reduction for classification with microarray data, Stat. Appl. Gen. Mol. Biol.3(1) (2004) 1-30. · Zbl 1140.62050
[5] Cai, D., He, X. and Han, J., Semi-supervised discriminant analysis, in 11th Int. Conf. Computer Vision, IEEE, 2007, pp. 1-7.
[6] M. A. Carreira-Perpinán, A review of dimension reduction techniques, Department of Computer Science. University of Sheffield. Tech. Rep. CS-96-09, 9 (1997) 1-69.
[7] Cerquides, J. and De Mántaras, R. L., Robust bayesian linear classifier ensembles, in European Conf. Machine Learning (Springer, 2005), pp. 72-83.
[8] Chapelle, O., Scholkopf, B. and Zien, A., Semi-supervised learning (O. Chapelle et al., eds.; 2006)[book reviews], IEEE Trans. Neural Netw.20(3) (2009) 542-542.
[9] Chatpatanasiri, R. and Kijsirikul, B., A unified semi-supervised dimensionality reduction framework for manifold learning, Neurocomput.73(10-12) (2010) 1631-1640.
[10] Chen, W.-S., Dai, X., Pan, B. and Tang, Y. Y., Semi-supervised discriminant analysis method for face recognition, Int. J. Wavelets Multiresolution Inf. Process.13(6) (2015) 2979-2984. · Zbl 1351.68245
[11] Chin, W. W., Marcolin, B. L. and Newsted, P. R., A partial least squares latent variable modeling approach for measuring interaction effects: Results from a monte carlo simulation study and an electronic-mail emotion/adoption study, Inf. Syst. Res.14(2) (2003) 189-217.
[12] Deng, H., Ma, C., Shen, L. and Yang, C., Semi-supervised learning using autodidactic interpolation on sparse representation-based multiple one-dimensional embedding, Int. J. Wavelets Multiresolution Inf. Process.17(3) (2019) 1950013. · Zbl 1410.68314
[13] Fu, Y., Peng, J. and Dong, X., Partial least squares with a regularized weight, J. Math. Chem.54(2) (2016) 403-415. · Zbl 1349.78043
[14] Goodhue, D. L., Lewis, W. and Thompson, R., Does pls have advantages for small sample size or non-normal data?, Mis Q.36(3) (2012) 981-1001.
[15] He, X., Yan, S., Hu, Y., Niyogi, P. and Zhang, H.-J., Face recognition using laplacianfaces, IEEE Trans. Pattern Anal. Mach. Intell.27(3) (2005) 328-340.
[16] John, G. H. and Langley, P., Estimating continuous distributions in bayesian classifiers, in Proc. 11th Conf. Uncertainty in Artificial Intelligence (Morgan Kaufmann Publishers Inc., 1995), pp. 338-345.
[17] Jolliffe, I., Principal component analysis, in Int. Encyclopedia of Statistical Science (Springer, 2011), pp. 1094-1096.
[18] Keim, D. A., Information visualization and visual data mining, IEEE Trans. Visua. Comput. Graph.8(1) (2002) 1-8.
[19] Lee, C. and Landgrebe, D. A., Analyzing high-dimensional multispectral data, IEEE Trans. Geosci. Remote Sens.31(4) (1993) 792-800.
[20] Li, W., Duan, Z. and Zhao, W., Drones air-materials demands predictions method based on partial least squares model, Syst. Eng. Theory Practice38(5) (2018) 1354-1360.
[21] Li, L., Yang, C. and Xie, Q., 1D embedding multi-category classification methods, Int. J. Wavelets Multiresolution Inf. Process.14(2) (2016) 1640006. · Zbl 1333.68231
[22] Liu, C., Li, Y., Zhao, Q. and Liu, C., Reference vector-based multi-objective clustering for high-dimensional data, Appl. Soft Comput.78 (2019) 614-629.
[23] Liu, Y. and Rayens, W., Pls and dimension reduction for classification, Comput. Stat.22(2) (2007) 189-208. · Zbl 1196.62085
[24] Liu, C., Zhao, Q., Yan, B., Elsayed, S. and Sarker, R., Transfer learning-assisted multi-objective evolutionary clustering framework with decomposition for high-dimensional data, Inf. Sci.505 (2019) 440-456. · Zbl 1456.68156
[25] Luo, H., Tang, Y. Y., Wang, Y., Wang, J., Biuk-Aghai, R. P., Pan, J., Liu, R. and Yang, L., Hyperspectral image classification using metric learning in one-dimensional embedding framework, IEEE J. Sel. Top. Appl. Earth Obs. Remote Sens.10(5) (2017) 1987-2001.
[26] Luo, H., Tang, Y. Y., Wang, Y., Wang, J., Yang, L., Li, C. and Hu, T., Hyperspectral image classification based on spectral-spatial one-dimensional manifold embedding, IEEE Trans. Geosci. Remote Sens.54(9) (2016) 5319-5340.
[27] Peng, J. and Tao, L., Sparse matrix transform-based linear discriminant analysis for hyperspectral image classification, Signal Image Video Process.10(4) (2016) 761-768.
[28] J. Shlens, A tutorial on principal component analysis, preprint (2014), arXiv:1404.1100.
[29] Sugiyama, M., Idé, T., Nakajima, S. and Sese, J., Semi-supervised local fisher discriminant analysis for dimensionality reduction, Mach. Learn.78 (2010) 35-61. · Zbl 1470.68180
[30] Verleysen, M.et al., Learning high-dimensional data, Nato Sci. Ser. Sub Ser. III Comput. Syst. Sci.186 (2003) 141-162.
[31] Wang, Y., Tang, Y. Y., Li, L. and Wang, J., Face recognition via collaborative representation based multiple one-dimensional embedding, Int. J. Wavelets Multiresolution Inf. Process.14(2) (2016) 1640003. · Zbl 1334.68197
[32] Wang, Y., Tang, Y. Y., Li, L. and Wang, P., Information-theoretic atomic representation for robust pattern classification, in 23rd Int. Conf. Pattern Recognition (ICPR), IEEE, 2016, pp. 3685-3690.
[33] M. Welling, Fisher linear discriminant analysis, Department of Computer Science, University of Toronto (2005).
[34] Wold, H., Partial least squares, Encyclopedia of Statistical Sciences, Vol. 9 (Wiley Online Library, 2004).
[35] Wold, S., Sjöström, M. and Eriksson, L., Pls-regression: A basic tool of chemometrics, Chemometr. Intell. Lab. Syst.58(2) (2001) 109-130.
[36] Xia, J., Chanussot, J., Du, P. and He, X., Semi-supervised dimensionality reduction for hyperspectral remote sensing image classification, in 4th Workshop on Hyperspectral Image and Signal Processing: Evolution in Remote Sensing (WHISPERS), IEEE, 2012, pp. 1-4.
[37] Q. Xie, L. Tang, W. Li, V. John and Y. Hu, Principal model analysis based on partial least squares, preprint (2019), arXiv:1902.02422.
[38] Yan, A., Huang, X. and Shao, H., On the sparseness and generalization capability of least squares support vector machines, J. Syst. Sci. Inf.3(3) (2015) 279-288.
[39] Ye, Z., Li, H., Song, Y., Benediktsson, J. A. and Yuan, Y. T., Hyperspectral image classification using principal components-based smooth ordering and multiple 1-d interpolation, IEEE Trans. Geosci. Remote Sens.PP(99) (2017) 1-11.
[40] Ye, Z., Li, H., Song, Y., Wang, J. and Benediktsson, J. A., A novel semi-supervised learning framework for hyperspectral image classification, Int. J. Wavelets Multiresolution Inf. Process.14(2) (2016) 1640005. · Zbl 1334.68288
[41] Yu, S., Yu, K., Tresp, V., Kriegel, H.-P. and Wu, M., Supervised probabilistic principal component analysis, in Proc. 12th ACM SIGKDD Int. Conf. Knowledge Discovery and Data Mining, ACM, 2006, pp. 464-473.
[42] Zhang, X., Li, A. and Ran, P., Construction and practice of the optimal smooth semi-supervised support vector machine, J. Syst. Sci. Inf.3(5) (2015) 398-410.
This reference list is based on information provided by the publisher or from digital mathematics libraries. Its items are heuristically matched to zbMATH identifiers and may contain data conversion errors. In some cases that data have been complemented/enhanced by data from zbMATH Open. This attempts to reflect the references listed in the original paper as accurately as possible without claiming completeness or a perfect matching.