Orthogonal canonical correlation analysis and applications. (English) Zbl 1455.65059

Summary: Canonical correlation analysis (CCA) is a cornerstone of linear dimensionality reduction techniques that jointly maps two datasets to achieve maximal correlation. CCA has been widely used in applications for capturing data features of interest. In this paper, we establish a range constrained orthogonal CCA (OCCA) model and its variant and apply them for three data analysis tasks of datasets in real-life applications, namely unsupervised feature fusion, multi-target regression and multi-label classification. Numerical experiments show that the OCCA and its variant produce superior accuracy compared to the traditional CCA.


65F15 Numerical computation of eigenvalues and eigenvectors of matrices
15A18 Eigenvalues, singular values, and eigenvectors
65K05 Numerical mathematical programming methods
90C90 Applications of mathematical programming


Full Text: DOI


[1] Absil, P.-A.; Mahony, R.; Sepulchre, R., Optimization Algorithms On Matrix Manifolds (2008), Princeton University Press: Princeton University Press, Princeton, NJ · Zbl 1147.65043
[2] Bai, Z.; Demmel, J.; Dongarra, J.; Ruhe, A.; van der Vorst, H., Templates for the Solution of Algebraic Eigenvalue Problems: A Practical Guide (2000), SIAM: SIAM, Philadelphia, PA · Zbl 0965.65058
[3] Chu, D.; Liao, L.; Ng, M. K.; Zhang, X., Sparse canonical correlation analysis: new formulation and algorithm, IEEE Trans. Pattern Anal. Mach. Intell., 35, 12, 3050-3065 (2013)
[4] Cortes, C. and Mohri, M., AUC optimization vs. error rate minimization, in Advances in Neural Information Processing Systems 16, S. Thrun, L.K. Saul, and B. Scholkopf, eds., 2004, pp. 313-320.
[5] Cunningham, J. P.; Ghahramani, Z., Linear dimensionality reduction: survey, insights, and generalizations, J. Mach. Learning Res., 16, 2859-2900 (2015) · Zbl 1351.62123
[6] Dembszynski, K., Waegeman, W., Cheng, W., and Hüllermeier, E., On label dependence in multilabel classification, LastCFP: ICML Workshop on Learning from Multi-label data, Ghent University, KERMIT, Department of Applied Mathematics, Biometrics and Process Control, 2010.
[7] Džeroski, S.; Demšar, D.; Grbović, J., Predicting chemical parameters of river water quality from bioindicator data, Appl. Intell., 13, 1, 7-17 (2000)
[8] Edelman, A.; Arias, T. A.; Smith, S. T., The geometry of algorithms with orthogonality constraints, SIAM J. Matrix Anal. Appl., 20, 2, 303-353 (1999) · Zbl 0928.65050
[9] Fan, R.-E.; Chang, K.-W.; Hsieh, C.-J.; Wang, X.-R.; Lin, C.-J., Liblinear: A library for large linear classification, J. Mach. Learn. Res., 9, Aug, 1871-1874 (2008) · Zbl 1225.68175
[10] Gao, B.; Liu, X.; Chen, X. J.; Yuan, Y., A new first-order algorithmic framework for optimization problems with orthogonality constraints, SIAM J. Optim., 28, 1, 302-332 (2018) · Zbl 1382.65171
[11] Gao, B.; Liu, X.; Yuan, Y., First-order algorithms for optimization problems with orthogonality constraints, Oper. Res. Trans., 21, 4, 57-68 (2017) · Zbl 1399.65130
[12] Golub, G. H.; Van Loan, C. F., Matrix Computations (2013), Johns Hopkins University Press: Johns Hopkins University Press, Baltimore, MD · Zbl 1268.65037
[13] Hardoon, D. R.; Szedmak, S.; Shawe-Taylor, J., Canonical correlation analysis: an overview with application to learning methods, Neural Comput., 16, 12, 2639-2664 (2004) · Zbl 1062.68134
[14] Hariharan, B., Zelnik-Manor, L., Varma, M., and Vishwanathan, S., Large scale max-marginmultilabel classification with priors, in Proceedings of the 27th International Conference on Machine Learning, J. Fürnkranz and T. Joachims, eds., Omnipress, Madison, WI, 2010, pp. 423-430.
[15] Hatzikos, E.; Tsoumakas, G.; Tzanisand, G.; Bassiliades, N.; Vlahavas, I., An empirical study on sea water quality prediction, Knowl. Based Syst., 21, 6, 471-478 (2008)
[16] Hotelling, H., Relations between two sets of variates, Biometrika, 28, 3-4, 321-377 (1936) · Zbl 0015.40705
[17] Hu, J.; Milzarek, A.; Wen, Z.; Yuan, Y., Adaptive quadratically regularized Newton method for Riemannian optimization, SIAM J. Matrix Anal. Appl., 39, 1181-1207 (2018) · Zbl 1415.65139
[18] Jiang, B.; Dai, Y.-H., A framework of constraint preserving update schemes for optimization on stiefel manifold, Math. Program., 153, 535-575 (2015) · Zbl 1325.49037
[19] Karalič, A.; Bratko, I., First order regression, Mach. Learn., 26, 2-3, 147-176 (1997) · Zbl 0866.68089
[20] Mackey, L., Deflation methods for sparse PCA, in Advances in Neural Information Processing Systems 21, D. Koller, D. Schuurmans, Y. Bengio and L. Bottou, eds., NIPS, 2008, pp. 1017-1024.
[21] Muirhead, R. J., Aspects of Multivariate Statistical Theory (2005), Wiley: Wiley, New York, NY
[22] Nocedal, J.; Wright, S., Numerical Optimization (2006), Springer: Springer, Berlin · Zbl 1104.65059
[23] Rong, G., Jin, C., Kakade, S., and Sidford, P.N.A., Efficient algorithms for large-scale generalized eigenvector computation and canonical correlation analysis, in Proceedings of the 33rd International Conference on Machine Learning, M. Balcan and K. Weinberger, eds., JMLR, 2016, pp. 2741-2750.
[24] Shen, X., Sun, Q., and Yuan, Y., Orthogonal canonical correlation analysis and its application in feature fusion, in Proceedings of the 16th International Conference on Information Fusion, IEEE, Istanbul, 2013, pp. 151-157.
[25] Spyromitros-Xioufis, E.; Tsoumakas, G.; Groves, W.; Vlahavas, I., Multi-target regression via input space expansion: treating targets as inputs, Mach. Learn., 104, 1, 55-98 (2016) · Zbl 1454.68134
[26] Sun, L., Ji, S., and Ye, J., A least squares formulation for canonical correlation analysis, in Proceeding of the 25th International Conference on Machine learning, A. McCallum and S. Roweis eds., ACM, New York, 2008, pp. 1024-1031.
[27] Sun, Q.-S.; Zeng, S.-G.; Liu, Y.; Heng, P.-A.; Xia, D.-S., A new method of feature fusion and its application in image recognition, Pattern. Recognit., 38, 12, 2437-2448 (2005)
[28] Sun, W.; Yuan, Y., Optimization Theory and Methods (2006), Springer: Springer, Berlin
[29] Tsanas, A.; Xifara, A., Accurate quantitative estimation of energy performance of residential buildings using statistical machine learning tools, Energy Build., 49, 560-567 (2012)
[30] Wen, Z.; Yin, W., A feasible method for optimization with orthogonality constraints, Math. Program., 142, 1-2, 397-434 (2013) · Zbl 1281.49030
[31] Yeh, I-C., Modeling slump flow of concrete using second-order regressions and artificial neural networks, Cem. Concr. Compos., 29, 6, 474-480 (2007)
[32] Zhang, L.-H.; Li, R.-C., Maximization of the sum of the trace ratio on the Stiefel manifold, I: theory, Sci. China Math., 57, 12, 2495-2508 (2014) · Zbl 1341.90128
[33] Zhang, L.-H.; Li, R.-C., Maximization of the sum of the trace ratio on the Stiefel manifold, II: computation, Sci. China Math., 58, 7, 1549-1566 (2015) · Zbl 1384.90102
This reference list is based on information provided by the publisher or from digital mathematics libraries. Its items are heuristically matched to zbMATH identifiers and may contain data conversion errors. It attempts to reflect the references listed in the original paper as accurately as possible without claiming the completeness or perfect precision of the matching.