Optimal selection of reduced rank estimators of high-dimensional matrices. (English) Zbl 1216.62086

Summary: We introduce a new criterion, the Rank Selection Criterion (RSC), for selecting the optimal reduced rank estimator of the coefficient matrix in multivariate response regression models. The corresponding RSC estimator minimizes the Frobenius norm of the fit plus a regularization term proportional to the number of parameters in the reduced rank model. The rank of the RSC estimator provides a consistent estimator of the rank of the coefficient matrix; in general, the rank of our estimator is a consistent estimate of the effective rank, which we define to be the number of singular values of the target matrix that are appropriately large. The consistency results are valid not only in the classic asymptotic regime, when \(n\), the number of responses, and \(p\), the number of predictors, stay bounded, and \(m\), the number of observations, grows, but also when either, or both, \(n\) and \(p\) grow, possibly much faster than \(m\).
We establish minimax optimal bounds on the mean squared errors of our estimators. Our finite sample performance bounds for the RSC estimator show that it achieves the optimal balance between the approximation error and the penalty term. Furthermore, our procedure has very low computational complexity, is linear in the number of candidate models, making it particularly appealing for large scale problems. We contrast our estimator with the nuclear norm penalized least squares (NNP) estimator, which has an inherently higher computational complexity than RSC, for multivariate regression models. We show that NNP has estimation properties similar to those of RSC, albeit under stronger conditions. However, it is not as parsimonious as RSC. We offer a simple correction of the NNP estimator which leads to consistent rank estimation.
We verify and illustrate our theoretical findings via an extensive simulation study.


62H12 Estimation in multivariate analysis
62J05 Linear regression; mixed models
65C60 Computational problems in statistics (MSC2010)


Full Text: DOI arXiv


[1] Anderson, T. W. (1951). Estimating linear restrictions on regression coefficients for multivariate normal distributions. Ann. Math. Statist. 22 327-351. · Zbl 0043.13902
[2] Anderson, T. W. (1999). Asymptotic distribution of the reduced rank regression estimator under general conditions. Ann. Statist. 27 1141-1154. · Zbl 0961.62011
[3] Anderson, T. W. (2002). Specification and misspecification in reduced rank regression. Sankhyā Ser. A 64 193-205. · Zbl 1192.62145
[4] Candès, E. J. and Plan, Y. (2010). Tight oracle bounds for low-rank matrix recovery from a minimal number of random measurements. Available at . · Zbl 1366.90160
[5] Candès, E. J. and Tao, T. (2010). The power of convex relaxation: Near-optimal matrix completion. IEEE Trans. Inform. Theory 56 2053-2080. · Zbl 1366.15021
[6] Cavalier, L., Golubev, G. K., Picard, D. and Tsybakov, A. B. (2002). Oracle inequalities for inverse problems. Ann. Statist. 30 843-874. · Zbl 1029.62032
[7] Fazel, M. (2002). Matrix rank minimization with applications. Ph.D. thesis, Stanford University.
[8] Horn, R. A. and Johnson, C. R. (1985). Matrix Analysis . Cambridge Univ. Press, Cambridge. · Zbl 0576.15001
[9] Izenman, A. J. (1975). Reduced-rank regression for the multivariate linear model. J. Multivariate Anal. 5 248-264. · Zbl 0313.62042
[10] Izenman, A. J. (2008). Modern Multivariate Statistical Techniques: Regression, Classification, and Manifold Learning . Springer, New York. · Zbl 1155.62040
[11] Kolmogorov, A. N. and Tihomirov, V. M. (1961). \epsilon -entropy and \epsilon -capacity of sets in functional spaces. Amer. Math. Soc. Transl. (2) 17 277-364. · Zbl 0133.06703
[12] Lu, Z., Monteiro, R. and Yuan, M. (2010). Convex optimization methods for dimension reduction and coefficient estimation in multivariate linear regression. Math. Program. · Zbl 1246.90120
[13] Ma, S., Goldfarb, D. and Chen, L. (2009). Fixed point and Bregman iterative methods for matrix rank minimization. Available at . · Zbl 1221.65146
[14] Negahban, S. and Wainwright, M. J. (2009). Estimation of (near) low-rank matrices with noise and high-dimensional scaling. Available at . · Zbl 1216.62090
[15] Rao, C. R. (1980). Matrix approximations and reduction of dimensionality in multivariate statistical analysis. In Multivariate Analysis, V (Proc. Fifth Internat. Sympos., Univ. Pittsburgh, Pittsburgh, PA, 1978) 3-22. North-Holland, Amsterdam. · Zbl 0442.62042
[16] Recht, B., Fazel, M. and Parrilo, P. A. (2010). Guaranteed minimum-rank solutions of linear matrix equations via nuclear norm minimization. SIAM Rev. 52 471-501. · Zbl 1198.90321
[17] Reinsel, G. C. and Velu, R. P. (1998). Multivariate Reduced-Rank Regression: Theory and Applications. Lecture Notes in Statist. 136 . Springer, New York. · Zbl 0909.62066
[18] Robinson, P. M. (1973). Generalized canonical analysis for time series. J. Multivariate Anal. 3 141-160. · Zbl 0263.62036
[19] Robinson, P. M. (1974). Identification, estimation and large-sample theory for regressions containing unobservable variables. Internat. Econom. Rev. 15 680-692. JSTOR: · Zbl 0298.62013
[20] Rohde, A. and Tsybakov, A. B. (2010). Estimation of high-dimensional low-rank matrices. Available at . · Zbl 1215.62056
[21] Rudelson, M. and Vershynin, R. (2010). Non-asymptotic theory of random matrices: Extreme singular values. In Proceedings of the International Congress of Mathematicians . Hyderabad, India. · Zbl 1227.60011
[22] Takane, Y. and Hunter, M. A. (2001). Constrained principal component analysis: A comprehensive theory. Appl. Algebra Engrg. Comm. Comput. 12 391-419. · Zbl 1040.62050
[23] Takane, Y. and Hwang, H. (2007). Regularized linear and kernel redundancy analysis. Comput. Statist. Data Anal. 52 394-405. · Zbl 1452.62421
[24] Toh, K. C., Todd, M. J. and Tütüncü, R. H. (1999). SDPT3-a MATLAB software package for semidefinite programming, version 1.3. Optim. Methods Softw. 11/12 545-581. · Zbl 0997.90060
[25] van der Vaart, A. W. and Wellner, J. A. (1996). Weak Convergence and Empirical Processes: With Applications to Statistics . Springer, New York. · Zbl 0862.60002
[26] Yuan, M., Ekici, A., Lu, Z. and Monteiro, R. (2007). Dimension reduction and coefficient estimation in multivariate linear regression. J. R. Stat. Soc. Ser. B Stat. Methodol. 69 329-346.
This reference list is based on information provided by the publisher or from digital mathematics libraries. Its items are heuristically matched to zbMATH identifiers and may contain data conversion errors. It attempts to reflect the references listed in the original paper as accurately as possible without claiming the completeness or perfect precision of the matching.