×

A general soft label based linear discriminant analysis for semi-supervised dimensionality reduction. (English) Zbl 1308.68105

Summary: Dealing with high-dimensional data has always been a major problem in research of pattern recognition and machine learning, and Linear Discriminant Analysis (LDA) is one of the most popular methods for dimension reduction. However, it only uses labeled samples while neglecting unlabeled samples, which are abundant and can be easily obtained in the real world. In this paper, we propose a new dimension reduction method, called “SL-LDA”, by using unlabeled samples to enhance the performance of LDA. The new method first propagates label information from the labeled set to the unlabeled set via a label propagation process, where the predicted labels of unlabeled samples, called “soft labels”, can be obtained. It then incorporates the soft labels into the construction of scatter matrixes to find a transformed matrix for dimension reduction. In this way, the proposed method can preserve more discriminative information, which is preferable when solving the classification problem. We further propose an efficient approach for solving SL-LDA under a least squares framework, and a flexible method of SL-LDA (FSL-LDA) to better cope with datasets sampled from a nonlinear manifold. Extensive simulations are carried out on several datasets, and the results show the effectiveness of the proposed method.

MSC:

68T10 Pattern recognition, speech recognition
68T05 Learning and adaptive systems in artificial intelligence
PDFBibTeX XMLCite
Full Text: DOI

References:

[1] Belhumeur, P. N.; Hespanha, J. P.; Kriegman, D. J., Eigenfaces vs. fisherfaces: recognition using class specific linear projection, IEEE Transactions on Pattern Analysis and Machine Intelligence, 19, 7, 711-720 (1997)
[2] Belkin, M.; Niyogi, P.; Sindhwani, V., Manifold regularization: a geometric framework for learning from labeled and unlabeled examples, Journal of Machine Learning Research, 7, 2399-2434 (2006) · Zbl 1222.68144
[3] Cai, D.; He, X.; Han, J., SRDA: an efficient algorithm for large scale discriminant analysis, IEEE Transactions on Knowledge and Data Engineering, 20, 1, 1-12 (2008)
[4] Cai, D.; He, X.; Han, J., Semi-supervised discriminant analysis, (Proc. of ICCV (2007))
[5] Cai, D.; He, X.; Han, J., Spectral regression: a unified subspace learning framework for content-based image retrieval, (Proc. of ACM MM (2007))
[6] Chen, L.; Liao, H.; Ko, M.; Lin, J.; Yu, G., A new LDA-based face recognition system which can solve the small sample size problem, Pattern Recognition, 33, 10, 1713-1726 (2000)
[7] Chen, J.; Ye, J.; Li, Q., Integrating global and local structures: a least square framework for dimension reduction, (Proc. of CVPR (2007))
[8] Chow, T. W.S.; Rahman, M. K.M., Content-based image retrieval by using tree-structured features and multi-layer self-organizing map, Pattern Analysis and Application, 9, 1-20 (2006)
[9] Cortes, C.; Vapnik, V. N., Support-vector networks, Machine Learning, 20, 3, 273-297 (1995) · Zbl 0831.68098
[10] Cun, Y. L.; Bottou, L.; Bengio, Y.; Haffner, P., Gradient-based learning applied to document recognition, Proceedings of IEEE, 86, 11, 2278-2324 (1998)
[11] Friedman, J. H., Regularized discriminant analysis, Journal of American Statistical Association, 84, 405, 165-175 (1989)
[12] Fukunaga, K., Introduction to statistical pattern classification (1990), Academic Press · Zbl 0711.62052
[13] Graham, D. B.; Allinson, N. M., Characterizing virtual eigensignatures for general purpose face recognition in face recognition: from theory to application, NATO ASI Series F, Computer and Systems Sciences, 163, 446-456 (1998)
[14] Hastie, T.; Tibshirani, R.; Friedman, J., The elements of statistical learning: data mining, inference and prediction (2001), Springer · Zbl 0973.62007
[15] He, X.; Cai, C.; Han, J., Learning a maximium margin subspace for image retrieval, IEEE Transactions on Knowledge and Data Engineering, 20, 2, 189-201 (2008)
[16] He, X.; Yan, S.; Hu, Y.; Niyogi, P.; Zhang, H., Face recognition using Laplacianfaces, IEEE Transactions on Patter Analysis and Machine Intelligence, 27, 3, 328-340 (2005)
[17] Huang, Y.; Xu, D.; Nie, F., Semi-supervised dimension reduction using trace ratio criterion, IEEE Transactions on Neural Networks and Learning Systems, 23, 3, 519-526 (2012)
[18] Hull, J., A database for handwritten text recognition research, IEEE Transactions on Pattern Analysis and Machine Intelligence, 16, 5, 550-554 (1994)
[19] Jia, Y.; Nie, F.; Zhang, C., Trace ratio problem revisited, IEEE Transactions on Neural Networks and Learning Systems, 20, 4, 729-735 (2009)
[20] Leibe, B.; Schiele, B., Analyzing appearance and contour based methods for object categorization, (Proc. of CVPR (2003))
[21] Li, J.; Allinson, A.; Tao, D.; Li, X., Multi-training support vector machine for image retrieval, IEEE Transactions on Image Processing, 11, 5, 3597-3601 (2006)
[22] Nene, S. A.; Nayar, S. K.; Murase, H., Columbia object image library (COIL-100). Technical report CUCS-005-96 (1996), Columbia University
[23] Nie, F.; Xiang, S.; Jia, Y.; Zhang, C., Semi-supervised orthogonal discriminant analysis via label propagation, Pattern Recognition, 42, 11, 2615-2627 (2009) · Zbl 1175.68338
[24] Nie, F.; Xiang, S.; Liu, Y.; Zhang, C., A general graph-based semi-supervised learning with novel class discovery, Neural Computation and Application, 19, 4, 549-555 (2010)
[25] Nie, F.; Xu, D.; Tsang, I. W.H.; Zhang, C., Flexible manifold embedding: a framework for semi-supervised and unsupervised dimension reduction, IEEE Transactions on Image Processing, 19, 7, 1921-1932 (2010) · Zbl 1371.94276
[26] Paige, C. C.; Saunders, M. A., LSQR: sparse linear equations and least squares problems, ACM Transaction on Mathematical Software, 8, 2, 195-209 (1982)
[27] Roweis, S.; Saul, L., Nonlinear dimension reduction by locally linear embedding, Science, 290, 2323-2326 (2000)
[28] Scholkopf, B.; Smola, A. J.; Muller, K. R., Nonlinear component analysis as a kernel eigenvalue problem, Neural Computation, 10, 5, 1299-1319 (1998)
[29] Sim, T.; Baker, S., The CMU posse, illumination and expression database, IEEE Transactions on Pattern Analysis and Machine Intelligence, 23, 6, 643-660 (2001)
[30] Sun, L.; Ceran, B.; Ye, J., A scalable two-stage approach for a class of dimension reduction techniques, (Proc. of KDD (2010))
[31] Tenenbaum, J. B.; Silva, V.de; Langford, J. C., A global geometric framework for nonlinear dimension reduction, Science, 290, 2319-2323 (2000)
[32] Turk, M.; Pentland, A., Face recognition using Eigenfaces, (Proc. of CVPR (1991))
[33] Vapnik, V. N., The nature of statistical learning theory (1995), Springer-Verlag: Springer-Verlag New York · Zbl 0833.62008
[34] Vapnik, V. N., Statistical learning theory (1998), Wiley: Wiley New York · Zbl 0935.62007
[35] Wang, M.; Hua, X. S.; Mei, T.; Hong, R.; Qi, G. J.; Song, Y.; Dai, L. R., Semi-supervised kernel density estimation for video annotation, Computer Vision and Image Understanding, 113, 3, 384-396 (2009)
[36] Wang, J. Z.; Li, J.; Wiederhold, G., Simplicity: semantics sentitive integrated matching for picture libraries, IEEE Transactions on Pattern Recognition and Machine Intelligence, 23, 9, 947-963 (2001)
[37] Wang, J.; Wang, F.; Zhang, C.; Shen, H. C.; Quan, L., Linear neighborhood propagation and its applications, IEEE Transactions on Pattern Analysis and Machine Intelligence, 31, 9, 1600-1615 (2009)
[38] Wang, F.; Zhang, C., Label propagation through linear neighborhoods, IEEE Transactions on Knowledge and Data Engineering, 20, 1, 55-67 (2008)
[39] Yang, J.; Frangi, A. F.; Yang, J.; Zhang, D.; Jin, Z., KPCA plus LDA: a complete kernel Fisher discriminant framework for feature extraction and recognition, IEEE Transactions on Pattern Analysis and Machine Intelligence, 27, 2, 230-244 (2005)
[40] Yan, S.; Xu, D.; Zhang, B.; Zhang, H.; Yang, Q.; Lin, S., Graph embedding and extensions: a general framework for dimension reduction, IEEE Transactions on Pattern Analysis and Machine Intelligence, 29, 1, 40-51 (2007)
[41] Ye, J., Least square linear discriminant analysis, (Proc. of ICML (2007))
[42] Ye, J.; Janardan, R.; Li, Q.; Park, H., Feature reduction via generalized uncorrelated linear discriminant analysis, IEEE Transactions on Knowledge and Data Engineering, 18, 10, 1312-1322 (2006)
[43] Yu, H.; Li, M.; Zhang, H. J.; Feng, J., Color texture moments for content-based image retrieval, (Proc. ICIP (2002))
[44] Zhang, Z.; Chow, T. W.S.; Zhao, M., Trace ratio optimization-based semi-supervised nonlinear dimension reduction for marginal manifold visualization, IEEE Transactions on Knowledge and Data Engineering, 25, 5, 1148-1161 (2013)
[45] Zhang, Z.; Dai, G.; Xu, C.; Jordan, M. I., Regularized discriminant analysis, ridge regression and beyond, Journal of Machine Learning Research, 11, 2199-2228 (2010) · Zbl 1242.62067
[46] Zhang, L.; Lin, F.; Zhang, B., Support vector machine learning for image retrieval, (Proc. ICIP (2001))
[47] Zhang, C.; Nie, F.; Xiang, S., A general kernelization framework for learning algorithms based on kernel PCA, Neurocomputing, 73, 4-6, 959-967 (2010)
[48] Zhang, T.; Tao, D.; Li, X.; Yang, J., Patch alignment for dimension reduction, IEEE Transactions on Knowledge and Data Engineering, 21, 9, 1299-1313 (2009)
[49] Zhang, Z.; Yan, S.; Zhao, M., Similarity preserving low-rank representation for enhanced data representation and effective subspace learning, Neural Networks, 53, 81-94 (2014) · Zbl 1308.68102
[50] Zhang, Z.; Zhao, M.; Chow, T. W.S., Marginal semi-supervised sub-manifold projections with informative constraints for dimension reduction and recognition, Neural Networks, 36, 97-111 (2012) · Zbl 1258.68132
[51] Zhang, Z.; Zhao, M.; Chow, T. W.S., Graph based constrained semi-supervised learning framework via label propagation over adaptive neighborhood, IEEE Transactions on Knowledge and Data Engineering (2013)
[52] Zhao, M.; Zhang, Z.; Chow, T. W.S., Trace ratio criterion based generalized discriminative learning for semi-supervised dimension reduction, Pattern Recognition, 45, 4, 1482-1499 (2012) · Zbl 1231.68226
[53] Zhao, M.; Zhang, Z.; Chow, T. W.S.; Li, B., Soft label based linear discriminant analysis for image recognition and retrieval, Computer Vision and Image Understanding, 121, 86-99 (2014)
[54] Zhou, D.; Bousquet, O.; Lal, T. N.; Weston, J.; Scholkopf, B., Learning with local and global consistency, (Proc. of NIPS (2004))
[55] Zhu, X.; Ghahramani, Z.; Lafferty, J. D., Semi-supervised learning using Gaussian fields and harmonic functions, (Proc. of ICML. (2003))
This reference list is based on information provided by the publisher or from digital mathematics libraries. Its items are heuristically matched to zbMATH identifiers and may contain data conversion errors. In some cases that data have been complemented/enhanced by data from zbMATH Open. This attempts to reflect the references listed in the original paper as accurately as possible without claiming completeness or a perfect matching.