High-dimensionality effects in the Markowitz problem and other quadratic programs with linear constraints: risk underestimation. (English) Zbl 1274.62365

Summary: We first study the properties of solutions of quadratic programs with linear equality constraints whose parameters are estimated from data in the high-dimensional setting where \(p\), the number of variables in the problem, is of the same order of magnitude as \(n\), the number of observations used to estimate the parameters. The Markowitz problem in Finance is a subcase of our study. Assuming normality and independence of the observations we relate the efficient frontier computed empirically to the “true” efficient frontier. Our computations show that there is a separation of the errors induced by estimating the mean of the observations and estimating the covariance matrix. In particular, the price paid for estimating the covariance matrix is an underestimation of the variance by a factor roughly equal to \(1 - p/n\). Therefore the risk of the optimal population solution is underestimated when we estimate it by solving a similar quadratic program with estimated parameters.
We also characterize the statistical behavior of linear functionals of the empirical optimal vector and show that they are biased estimators of the corresponding population quantities.
We investigate the robustness of our Gaussian results by extending the study to certain elliptical models and models where our \(n\) observations are correlated (in “time”). We show a lack of robustness of the Gaussian results, but are still able to get results concerning first order properties of the quantities of interest, even in the case of relatively heavy-tailed data (we require two moments). Risk underestimation is still present in the elliptical case and more pronounced than in the Gaussian case.
We discuss properties of the nonparametric and parametric bootstrap in this context. We show several results, including the interesting fact that standard applications of the bootstrap generally yield inconsistent estimates of bias.
We propose some strategies to correct these problems and practically validate them in some simulations. Throughout this paper, we will assume that \(p, n\) and \(n - p\) tend to infinity, and \(p<n\).
Finally, we extend our study to the case of problems with more general linear constraints, including, in particular, inequality constraints.


62H12 Estimation in multivariate analysis
60B20 Random matrices (probabilistic aspects)
62G20 Asymptotic properties of nonparametric inference
62H10 Multivariate distribution of statistics
90C20 Quadratic programming
91G10 Portfolio theory


Full Text: DOI arXiv


[1] Anderson, T. W. (2003). An Introduction to Multivariate Statistical Analysis , 3rd ed. Wiley, Hoboken, NJ. · Zbl 1039.62044
[2] Bai, Z. D. (1999). Methodologies in spectral analysis of large-dimensional random matrices, a review. Statist. Sinica 9 611-677. With comments by G. J. Rodgers and Jack W. Silverstein; and a rejoinder by the author. · Zbl 0949.60077
[3] Bai, Z., Liu, H. and Wong, W.-K. (2009). Enhancement of the applicability of Markowitz’s portfolio optimization by utilizing random matrix theory. Math. Finance 19 639-667. · Zbl 1185.91155
[4] Bickel, P. J. and Levina, E. (2008a). Covariance regularization by thresholding. Ann. Statist. 36 2577-2604. · Zbl 1196.62062
[5] Bickel, P. J. and Levina, E. (2008b). Regularized estimation of large covariance matrices. Ann. Statist. 36 199-227. · Zbl 1132.62040
[6] Biroli, G., Bouchaud, J.-P. and Potters, M. (2007). The student ensemble of correlation matrices: Eigenvalue spectrum and Kullback-Leibler entropy. Acta Phys. Polon. B 38 4009-4026. · Zbl 1373.62283
[7] Black, F. and Litterman, R. (1990). Asset allocation: Combining investor views with market equilibrium. Golman Sachs Fixed Income Research .
[8] Boyd, S. and Vandenberghe, L. (2004). Convex Optimization . Cambridge Univ. Press, Cambridge. · Zbl 1058.90049
[9] Campbell, J., Lo, A. and MacKinlay, C. (1996). The Econometrics of Financial Markets . Princeton Univ. Press, Princeton, NJ. · Zbl 0927.62113
[10] Chikuse, Y. (2003). Statistics on Special Manifolds. Lecture Notes in Statistics 174 . Springer, New York. · Zbl 1026.62051
[11] Chow, Y. S. and Teicher, H. (1997). Probability Theory: Independence, Interchangeability, Martingales , 3rd ed. Springer, New York. · Zbl 0891.60002
[12] Davidson, K. R. and Szarek, S. J. (2001). Local operator theory, random matrices and Banach spaces. In Handbook of the Geometry of Banach Spaces, Vol. I 317-366. North-Holland, Amsterdam. · Zbl 1067.46008
[13] Eaton, M. L. (1983). Multivariate Statistics: A Vector Space Approach . Wiley, New York. · Zbl 0587.62097
[14] El Karoui, N. (2007). Tracy-Widom limit for the largest eigenvalue of a large class of complex sample covariance matrices. Ann. Probab. 35 663-714. · Zbl 1117.60020
[15] El Karoui, N. (2008). Operator norm consistent estimation of large dimensional sparse covariance matrices. Ann. Statist. 36 2717-2756. · Zbl 1196.62064
[16] El Karoui, N. (2009a). Concentration of measure and spectra of random matrices: Applications to correlation matrices, elliptical distributions and beyond. Ann. Appl. Probab. 19 2362-2405. · Zbl 1255.62156
[17] El Karoui, N. (2009b). On the realized risk of high-dimensional Markowitz portfolios. Technical Report No. 784, Dept. Statistics, Univ. California, Berkeley.
[18] Fang, K. T., Kotz, S. and Ng, K. W. (1990). Symmetric Multivariate and Related Distributions. Monographs on Statistics and Applied Probability 36 . Chapman and Hall, London. · Zbl 0699.62048
[19] Frahm, G. and Jaekel, U. (2005). Random matrix theory and robust covariance matrix estimation for financial data. Available at .
[20] Horn, R. A. and Johnson, C. R. (1990). Matrix Analysis . Cambridge Univ. Press, Cambridge. Corrected reprint of the 1985 original. · Zbl 0704.15002
[21] Horn, R. A. and Johnson, C. R. (1994). Topics in Matrix Analysis . Cambridge Univ. Press, Cambridge. Corrected reprint of the 1991 original. · Zbl 0801.15001
[22] Jobson, J. D. and Korkie, B. (1980). Estimation for Markowitz efficient portfolios. J. Amer. Statist. Assoc. 75 544-554. · Zbl 0446.62047
[23] Johnstone, I. (2001). On the distribution of the largest eigenvalue in principal component analysis. Ann. Statist. 29 295-327. · Zbl 1016.62078
[24] Kan, R. and Smith, D. R. (2008). The distribution of the sample minimum-variance frontier. Management Science 54 1364-1380. · Zbl 1232.62138
[25] Lai, T. L. and Xing, H. (2008). Statistical Models and Methods for Financial Markets. Springer Texts in Statistics . Springer, New York. · Zbl 1149.62086
[26] Laloux, L., Cizeau, P., Bouchaud, J.-P. and Potters, M. (2000). Random matrix theory and financial correlations. Internat. J. Theoret. Appl. Finance 3 391-397. · Zbl 0970.91059
[27] Ledoit, O. and Wolf, M. (2004). A well-conditioned estimator for large-dimensional covariance matrices. J. Multivariate Anal. 88 365-411. · Zbl 1032.62050
[28] Ledoux, M. (2001). The Concentration of Measure Phenomenon. Mathematical Surveys and Monographs 89 . Amer. Math. Soc., Providence, RI. · Zbl 0995.60002
[29] Lugosi, G. (2006). Concentration of measure inequalities. Lecture notes available online.
[30] Marčenko, V. A. and Pastur, L. A. (1967). Distribution of eigenvalues in certain sets of random matrices. Mat. Sb. (N.S.) 72 507-536. · Zbl 0152.16101
[31] Mardia, K. V., Kent, J. T. and Bibby, J. M. (1979). Multivariate Analysis . Academic Press, London. · Zbl 0432.62029
[32] Markowitz, H. (1952). Portfolio selection. J. Finance 7 77-91.
[33] McNeil, A. J., Frey, R. and Embrechts, P. (2005). Quantitative Risk Management: Concepts, Techniques and Tools . Princeton Univ. Press, Princeton, NJ. · Zbl 1089.91037
[34] Meucci, A. (2005). Risk and Asset Allocation. Springer Finance . Springer, Berlin. · Zbl 1102.91067
[35] Meucci, A. (2008). Enhancing the Black-Litterman and related approaches: Views and stress-test on risk factors. Available at SSRN, .
[36] Michaud, R. O. (1998). Efficient Asset Management: A Practical Guide to Stock Portfolio Optimization and Asset Allocation . Oxford Univ. Press.
[37] Pafka, S. and Kondor, I. (2003). Noisy covariance matrices and portfolio optimization. II. Phys. A 319 487-494. · Zbl 1008.91039
[38] Rothman, A. J., Bickel, P. J., Levina, E. and Zhu, J. (2008). Sparse permutation invariant covariance estimation. Electron. J. Statist. 2 494-515 (electronic). · Zbl 1320.62135
[39] Ruppert, D. (2006). Statistics and Finance: An Introduction . Springer, New York. Corrected second printing of the 2004 original. · Zbl 1049.91083
[40] Silverstein, J. W. (1995). Strong convergence of the empirical distribution of eigenvalues of large-dimensional random matrices. J. Multivariate Anal. 55 331-339. · Zbl 0851.62015
[41] Tyler, D. E. (1987). A distribution-free M -estimator of multivariate scatter. Ann. Statist. 15 234-251. · Zbl 0628.62053
[42] van der Vaart, A. W. (1998). Asymptotic Statistics . Cambridge Univ. Press, Cambridge. · Zbl 0910.62001
[43] Wachter, K. W. (1978). The strong limits of random matrix spectra for sample matrices of independent elements. Ann. Probab. 6 1-18. · Zbl 0374.60039
This reference list is based on information provided by the publisher or from digital mathematics libraries. Its items are heuristically matched to zbMATH identifiers and may contain data conversion errors. It attempts to reflect the references listed in the original paper as accurately as possible without claiming the completeness or perfect precision of the matching.