×

Deep random walk of unitary invariance for large-scale data representation. (English) Zbl 1484.68199

Summary: Data representation aims at learning an efficient low-dimensional representation, which is always a challenging task in machine learning and computer vision. It can largely improve the performance of specific learning tasks. Unsupervised methods are extensively applied to data representation, which considers the internal connection among data. Most of existing unsupervised models usually use a specific norm to favor certain distributions of the input data, leading to an unsustainable encouraging performance for given learning tasks. In this paper, we propose an efficient data representation method to address large-scale feature representation problems, where the deep random walk of unitary invariance is exploited for learning discriminative features. First, the data representation is formulated as deep random walk problems, where unitarily invariant norms are employed to capture diverse beneficial perspectives hidden in the data. It is embedded into a state transition matrix model, where an arbitrary number of transition steps is available for an accurate affinity evaluation. Second, data representation problems are then transformed as high-order matrix factorization tasks with unitary invariance. Third, a closed-form solution is proved for the formulated data representation problem, which may provide a new perspective for solving high-order matrix factorization problems. Finally, extensive comparative experiments are conducted in publicly available real-world data sets. In addition, experimental results demonstrate that the proposed method achieves better performance than other compared state-of-the-art approaches in terms of data clustering.

MSC:

68T05 Learning and adaptive systems in artificial intelligence
15A23 Factorization of matrices
60G50 Sums of independent random variables; random walks
68T09 Computational aspects of data analysis and big data
PDFBibTeX XMLCite
Full Text: DOI

References:

[1] Ayas, S.; Ekinci, M., Single image super resolution using dictionary learning and sparse coding with multi-scale and multi-directional gabor feature representation, Inf. Sci., 512, 1264-1278 (2020)
[2] Belkin, M.; Niyogi, P., Laplacian eigenmaps and spectral techniques for embedding and clustering, (Advances in Neural Information Processing Systems (2002)), 585-591
[3] Chen, C.; Chen, Z.; Jiang, B.; Jin, X., Joint domain alignment and discriminative feature learning for unsupervised deep domain adaptation, (Proceedings of the Thirty-Third AAAI Conference on Artificial Intelligence (2019)), 3296-3303
[4] Chen, P.-H.; Yang, H.-C.; Chen, K.-W.; Chen, Y.-S., Mvsnet++: learning depth-based attention pyramid features for multi-view stereo, IEEE Trans. Image Process., 29, 7261-7273 (2020) · Zbl 07586398
[5] Dai, X.; Su, X.; Zhang, W.; Xue, F.; Li, H., Robust manhattan non-negative matrix factorization for image recovery and representation, Inf. Sci., 527, 70-87 (2020) · Zbl 1462.62399
[6] Duersch, J. A.; Gu, M., Randomized projection for rank-revealing matrix factorizations and low-rank approximations, SIAM Rev., 62, 661-682 (2020) · Zbl 07279897
[7] Federici, M.; Dutta, A.; Forré, P.; Kushman, N.; Akata, Z., Learning robust representations via multi-view information bottleneck, (Proceedings of the International Conference on Learning Representations (2020)), 1-26
[8] Feige, I., Invariant-equivariant representation learning for multi-class data, (Proceedings of the Thirty-Sixth International Conference on Machine Learning (2019)), 1882-1891
[9] Feng, X.; Wu, S., Robust sparse coding via self-paced learning for data representation, Inf. Sci., 546, 448-468 (2020) · Zbl 1475.68121
[10] Gong, C.; Shi, H.; Yang, J.; Yang, J., Multi-manifold positive and unlabeled learning for visual analysis, IEEE Trans. Circuits Syst. Video Technol., 30, 1396-1409 (2020)
[11] Guo, L.; Cai, X.; Qin, H.; Guo, Y.; Li, F.; Tian, G., Citation recommendation with a content-sensitive deepwalk based approach, (Proceedings of the Nineteenth IEEE International Conference on Data Mining Workshop (2019)), 538-543
[12] Hardoon, D. R.; Szedmak, S.; Shawe-Taylor, J., Canonical correlation analysis: An overview with application to learning methods, Neural Comput., 16, 2639-2664 (2004) · Zbl 1062.68134
[13] He, X.; Cai, D.; Yan, S.; Zhang, H.-J., Neighborhood preserving embedding, (IEEE International Conference on Computer Vision (2005)), 1208-1213
[14] X. He, P. Niyogi, Locality preserving projections. In Advances in Neural Information Processing Systems, 2004. pp. 153-160.
[15] Hinton, G. E.; Salakhutdinov, R. R., Reducing the dimensionality of data with neural networks, Science, 313, 504-507 (2006) · Zbl 1226.68083
[16] Hsu, C.-C.; Lin, C.-W., Cnn-based joint clustering and representation learning with feature drift compensation for large-scale image data, IEEE Trans. Multimedia, 20, 421-429 (2018)
[17] Huynh-The, T.; Hua, C.-H.; Ngo, T.-T.; Kim, D.-S., Image representation of pose-transition feature for 3d skeleton-based action recognition, Inf. Sci., 513, 112-126 (2020)
[18] Jean, N.; Wang, S.; Samar, A.; Azzari, G.; Lobell, D.; Ermon, S., Tile2vec: Unsupervised representation learning for spatially distributed data, (Proceedings of the Thirty-Third AAAI Conference on Artificial Intelligence, vol. 33 (2019)), 3967-3974
[19] Jin, S.; Wiseman, S.; Stratos, K.; Livescu, K., Discrete latent variable representations for low-resource text classification, (Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics (2020)), 4831-4842
[20] Jing, P.; Su, Y.; Li, Z.; Nie, L., Learning robust affinity graph representation for multi-view clustering, Inf. Sci., 544, 155-167 (2020) · Zbl 1475.68276
[21] Lee, D. D.; Seung, H. S., Learning the parts of objects by non-negative matrix factorization, Nature, 401, 788-791 (1999) · Zbl 1369.68285
[22] Li, Z.; Liu, J.; Tang, J.; Lu, H., Robust structured subspace learning for data representation, IEEE Trans. Pattern Anal. Mach. Intell., 37, 2085-2098 (2015)
[23] Li, Z.; Liu, J.; Yang, Y.; Zhou, X.; Lu, H., Clustering-guided sparse structural learning for unsupervised feature selection, IEEE Trans. Knowl. Data Eng., 26, 2138-2150 (2014)
[24] Li, Z.; Tang, J.; Mei, T., Deep collaborative embedding for social image understanding, IEEE Trans. Pattern Anal. Mach. Intell., 41, 2070-2083 (2019)
[25] Lu, C.; Feng, J.; Chen, Y.; Liu, W.; Lin, Z.; Yan, S., Tensor robust principal component analysis with a new tensor nuclear norm, IEEE Trans. Pattern Anal. Mach. Intell., 42, 925-938 (2020)
[26] Lu, Y.; Yuan, C.; Lai, Z.; Li, X.; Zhang, D.; Wong, W. K., Horizontal and vertical nuclear norm-based 2dlda for image representation, IEEE Trans. Circuits Syst. Video Technol., 29, 941-955 (2019)
[27] Meng, Y.; Shang, R.; Shang, F.; Jiao, L.; Yang, S.; Stolkin, R., Semi-supervised graph regularized deep nmf with bi-orthogonal constraints for data representation, IEEE Trans. Neural Networks Learn. Syst., 31, 3245-3258 (2020)
[28] Mikolov, T.; Sutskever, I.; Chen, K., Distributed representations of words and phrases and their compositionality, (Advances in Neural Information Processing Systems (2013)), 3111-3119
[29] Nie, F.; Yuan, J.; Huang, H., Optimal mean robust principal component analysis, (Proceedings of the 31st International Conference on Machine Learning (2014)), 1062-1070
[30] Peng, H.; Long, F.; Ding, C., Feature selection based on mutual information criteria of max-dependency, max-relevance, and min-redundancy, IEEE Trans. Pattern Anal. Mach. Intell., 27, 1226-1238 (2005)
[31] Perera, P.; Morariu, V. I.; Jain, R.; Manjunatha, V.; Wigington, C.; Ordonez, V.; Patel, V. M., Generative-discriminative feature representations for open-set recognition, (Proceedings of the International Conference on Computer Vision and Pattern Recognition (2020)), 11811-11820
[32] Perozzi, B.; Al-Rfou, R.; Skiena, S., Max-margin deepwalk: Discriminative learning of network representation, (Proceedings of the Twentieth ACM SIGKDD International Conference on Knowledge Discovery and Data Mining (2014)), 701-710
[33] Roweis, S. T.; Saul, L. K., Nonlinear dimensionality reduction by locally linear embedding, Science, 290, 2323-2326 (2002)
[34] Shu, Z.; Wu, X.-J.; You, C.-Z.; Liu, Z.; Li, P.; Fan, H.; Ye, F., Rank-constrained nonnegative matrix factorization for data representation, Inf. Sci., 528, 133-146 (2020) · Zbl 1459.68187
[35] Sutton, R. S.; Barto, A. G., Reinforcement learning: An introduction, IEEE Trans. Neural Networks, 9 (1998), 1054-1054
[36] Tenenbaum, J. B.; de Silva, V.; Langford, J. C., A global geometric framework for nonlinear dimensionality reduction, Science, 290, 2319-2323 (2000)
[37] Tu, C.; Zhang, W.; Liu, Z.; Sun, M., Max-margin deepwalk: Discriminative learning of network representation, (Proceedings of the Twenty-Fifth International Joint Conference on Artificial Intelligence (2016)), 3889-3895
[38] Wang, J. X.; Kurth-Nelson, Z.; Kumaran, D.; Tirumala, D., Prefrontal cortex as a meta-reinforcement learning system, Nat. Neurosci., 21, 860-868 (2018)
[39] Wang, Q.; Lai, J.; Claesen, L.; Yang, Z.; Lei, L.; Liu, W., A novel feature representation: Aggregating convolution kernels for image retrieval, Neural Networks, 130, 1-10 (2020)
[40] Wang, S.; Ding, Z.; Fu, Y., Discerning feature supported encoder for image representation, IEEE Trans. Image Process., 28, 3728-3738 (2019) · Zbl 07122937
[41] Wang, S.; Guo, W., Sparse multigraph embedding for multimodal feature representation, IEEE Trans. Multimedia, 19, 1454-1466 (2017)
[42] Wang, S.; Pedrycz, W.; Zhu, Q.; Zhu, W., Subspace learning for unsupervised feature selection via matrix factorization, Pattern Recogn., 48, 10-19 (2015) · Zbl 1373.68338
[43] Wang, S.; Zhu, W., Sparse graph embedding unsupervised feature selection, IEEE Trans. Syst., Man, Cybernetics: Syst., 48, 329-341 (2018)
[44] Wang, X.; Liu, Y.; Nie, F.; Huang, H., Discriminative unsupervised dimensionality reduction, (Proceedings of the Twenty-Fourth International Joint Conference on Artificial Intelligence (2015)), 3925-3931
[45] Xiao, Y.; Li, R.; Lu, X.; Liu, Y., Link prediction based on feature representation and fusion, Inf. Sci., 548, 1-17 (2020) · Zbl 1479.91286
[46] Xu, M.; Zhu, Z.; Zhang, X.; Zhao, Y.; Li, X., Canonical correlation analysis with \(\ell_{2 , 1}\)-norm for multiview data representation, IEEE Trans. Cybern, 1-11 (2019)
[47] Yan, W.; Sun, Q.; Sun, H.; Li, Y., Joint dimensionality reduction and metric learning for image set classification, Inf. Sci., 516, 109-124 (2020) · Zbl 1457.68243
[48] Yang, C.; Liu, Z.; Zhao, D.; Sun, M.; Chang, E. Y., Network representation learning with rich text information, (Proceedings of the Twenty-Fourth International Joint Conference on Artificial Intelligence (2015)), 2111-2117
[49] Yang, J.; Frangi, A. F.; Yang, J.-Y.; Zhang, D.; Jin, Z., Kpca plus lda: a complete kernel fisher discriminant framework for feature extraction and recognition, IEEE Trans. Pattern Anal. Mach. Intell., 27, 230-244 (2005)
[50] Yang, Z.; Li, Q.; Wenyin, L.; Lv, J., Shared multi-view data representation for multi-domain event detection, IEEE Trans. Pattern Anal. Mach. Intell., 42, 1243-1256 (2020)
[51] Yu, Y.-L.; Schuurmans, D., Rank/norm regularization with closed-form solutions: Application to subspace clustering, (Proceedings of International Joint Conference on Artificial Intelligence (2011)), 778-785
[52] Yuan, X.; Huang, B.; Wang, Y.; Yang, C.; Gui, W., Deep learning-based feature representation and its application for soft sensor modeling with variable-wise weighted sae, IEEE Trans. Industr. Inf., 14, 3235-3243 (2018)
[53] Zhan, X., Matrix theory (2013), American Mathematical Society: American Mathematical Society Providence · Zbl 1275.15001
[54] Zhu, Q.; Xu, N.; Huang, S.; Qian, J.; Zhang, D., Adaptive feature weighting for robust \(\ell_p\)-norm sparse representation with application to biometric image classification, Int. J. Mach. Learn. Cybern., 11, 463-474 (2020)
This reference list is based on information provided by the publisher or from digital mathematics libraries. Its items are heuristically matched to zbMATH identifiers and may contain data conversion errors. In some cases that data have been complemented/enhanced by data from zbMATH Open. This attempts to reflect the references listed in the original paper as accurately as possible without claiming completeness or a perfect matching.