Ultra high-dimensional multivariate posterior contraction rate under shrinkage priors. (English) Zbl 1480.62105

Summary: In recent years, shrinkage priors have received much attention in high-dimensional data analysis from a Bayesian perspective. Compared with widely used spike-and-slab priors, shrinkage priors have better computational efficiency. But the theoretical properties, especially posterior contraction rate, which is important in uncertainty quantification, are not established in many cases. In this paper, we apply global-local shrinkage priors to high-dimensional multivariate linear regression with unknown covariance matrix. We show that when the prior is highly concentrated near zero and has heavy tail, the posterior contraction rates for both coefficients matrix and covariance matrix are nearly optimal. Our results hold when number of features \(p\) grows much faster than the sample size n, which is of great interest in modern data analysis. We show that a class of readily implementable scale mixture of normal priors satisfies the conditions of the main theorem.


62H12 Estimation in multivariate analysis
62J07 Ridge regression; shrinkage estimators (Lasso)
60F15 Strong limit theorems
62F15 Bayesian inference


Full Text: DOI arXiv


[1] Armagan, A.; Clyde, M.; Dunson, D. B., Generalized beta mixtures of Gaussians, Adv. Neural Inf. Process. Syst., 523-531 (2011)
[2] Armagan, A.; Dunson, D. B.; Lee, J., Generalized double Pareto shrinkage, Statist. Sinica, 23, 119 (2013) · Zbl 1259.62061
[3] Armagan, A.; Dunson, D. B.; Lee, J.; Bajwa, W. U.; Strawn, N., Posterior consistency in linear models under shrinkage priors, Biometrika, 100, 1011-1018 (2013) · Zbl 1279.62139
[4] Bai, R.; Ghosh, M., High-dimensional multivariate posterior consistency under global-local shrinkage priors, J. Multivariate Anal., 167, 157-170 (2018) · Zbl 1403.62134
[5] Bhadra, A.; Datta, J.; Polson, N. G.; Willard, B., The horseshoe+estimator of ultra-sparse signals, Bayesian Anal., 12, 1105-1131 (2017) · Zbl 1384.62079
[6] Bhattacharya, A.; Chakraborty, A.; Mallick, B. K., Fast sampling with Gaussian scale mixture priors in high-dimensional regression, Biometrika, asw042 (2016)
[7] Bhattacharya, A.; Pati, D.; Pillai, N. S.; Dunson, D. B., Dirichlet-Laplace priors for optimal shrinkage, J. Amer. Statist. Assoc., 110, 1479-1490 (2015) · Zbl 1373.62368
[8] Carvalho, C. M.; Polson, N. G.; Scott, J. G., The horseshoe estimator for sparse signals, Biometrika, 97, 465-480 (2010) · Zbl 1406.62021
[9] Castillo, I.; Schmidt-Hieber, J.; van der Vaart, A., Bayesian linear regression with sparse priors, Ann. Statist., 43, 1986-2018 (2015) · Zbl 1486.62197
[10] Chen, L.; Huang, J. Z., Sparse reduced-rank regression for simultaneous dimension reduction and variable selection, J. Amer. Statist. Assoc., 107, 1533-1545 (2012) · Zbl 1258.62075
[11] Chun, H.; Keleş, S., Sparse partial least squares regression for simultaneous dimension reduction and variable selection, J. R. Stat. Soc. Ser. B Stat. Methodol., 72, 3-25 (2010) · Zbl 1411.62184
[12] Deshpande, S. K.; Ročková, V.; George, E. I., Simultaneous variable and covariance selection with the multivariate spike-and-slab LASSO, J. Comput. Graph. Statist. (2019)
[13] Goh, G.; Dey, D. K.; Chen, K., Bayesian sparse reduced rank multivariate regression, J. Multivariate Anal., 157, 14-28 (2017) · Zbl 1362.62140
[14] Griffin, J. E.; Brown, P. J., Inference with normal-gamma prior distributions in regression problems, Bayesian Anal., 5, 171-188 (2010) · Zbl 1330.62128
[15] Li, Y.; Datta, J.; Craig, B. A.; Bhadra, A., Joint mean-covariance estimation via the horseshoe with an application in genomic data analysis (2019), arXiv preprint arXiv:1903.06768
[16] Li, Y.; Nan, B.; Zhu, J., Multivariate sparse group lasso for the multivariate multiple linear regression with an arbitrary group structure, Biometrics, 71, 354-363 (2015) · Zbl 1390.62285
[17] Liquet, B.; Mengersen, K.; Pettitt, T.; Sutton, M., Bayesian variable selection regression of multivariate responses for group data, Bayesian Anal., 12, 1039-1067 (2017) · Zbl 1384.62259
[18] Mitchell, T. J.; Beauchamp, J. J., Bayesian variable selection in linear regression, J. Amer. Statist. Assoc., 83, 1023-1032 (1988) · Zbl 0673.62051
[19] Moran, G. E.; Rocková, V.; George, E. I., On variance estimation for Bayesian variable selection, Bayesian Anal. (2019) · Zbl 1435.62272
[20] Ning, B.; Ghosal, S., Bayesian linear regression for multivariate responses under group sparsity (2018), arXiv preprint arXiv:1807.03439
[21] van der Pas, S.; Kleijn, B.; van der Vaart, A., The horseshoe estimator: Posterior concentration around nearly black vectors, Electron. J. Stat., 8, 2585-2618 (2014) · Zbl 1309.62060
[22] Piironen, J.; Vehtari, A., On the hyperprior choice for the global shrinkage parameter in the horseshoe prior, (Singh, A.; Zhu, J., Proceedings of the 20th International Conference on Artificial Intelligence and Statistics. Proceedings of the 20th International Conference on Artificial Intelligence and Statistics, Proceedings of Machine Learning Research, vol. 54 (2017), PMLR: PMLR Fort Lauderdale, FL, USA), 905-913
[23] Polson, N. G.; Scott, J. G., Shrink globally, act locally: Sparse Bayesian regularization and prediction, Bayesian Stat., 9, 501-538 (2010)
[24] Polson, N. G.; Scott, J. G., On the half-Cauchy prior for a global scale parameter, Bayesian Anal., 7, 887-902 (2012) · Zbl 1330.62148
[25] Raskutti, G.; Wainwright, M. J.; Yu, B., Minimax rates of estimation for high-dimensional linear regression over \(\ell_q\)-balls, IEEE Trans. Inf. Theory, 57, 6976-6994 (2011) · Zbl 1365.62276
[26] Ročková, V., Bayesian estimation of sparse signals with a continuous spike-and-slab prior, Ann. Statist., 46, 401-437 (2018) · Zbl 1395.62230
[27] Ročková, V.; George, E. I., The spike-and-slab lasso, J. Amer. Statist. Assoc., 113, 431-444 (2018) · Zbl 1398.62186
[28] Rothman, A. J.; Levina, E.; Zhu, J., Sparse multivariate regression with covariance estimation, J. Comput. Graph. Statist., 19, 947-962 (2010)
[29] Scott, D. W., Multivariate Density Estimation: Theory, Practice, and Visualization (2015), John Wiley & Sons · Zbl 1311.62004
[30] Shin, M.; Bhattacharya, A.; Johnson, V. E., Functional horseshoe priors for subspace shrinkage, J. Amer. Statist. Assoc., 1-14 (2019)
[31] Song, Q.; Liang, F., Nearly optimal Bayesian shrinkage for high dimensional regression (2017), arXiv preprint arXiv:1712.08964
[32] Stephens, M.; Balding, D. J., Bayesian statistical methods for genetic association studies, Nature Rev. Genet., 10, 681 (2009)
[33] Tang, X.; Xu, X.; Ghosh, M.; Ghosh, P., Bayesian variable selection and estimation based on global-local shrinkage priors, Sankhya A, 80, 215-246 (2018)
[34] Vershynin, R., Introduction to the non-asymptotic analysis of random matrices, 2011 (2012), arXiv preprint arXiv:1011.3027
[35] Wilms, I.; Croux, C., An algorithm for the multivariate group lasso with covariance estimation, J. Appl. Stat., 45, 668-681 (2018) · Zbl 07282452
[36] Xu, X.; Ghosh, M., Bayesian variable selection and estimation for group lasso, Bayesian Anal., 10, 909-936 (2015) · Zbl 1334.62132
[37] Ye, F.; Zhang, C.-H., Rate minimaxity of the Lasso and dantzig selector for the \(\ell_q\) loss in \(\ell_r\) balls, J. Mach. Learn. Res., 11, 3519-3540 (2010) · Zbl 1242.62074
This reference list is based on information provided by the publisher or from digital mathematics libraries. Its items are heuristically matched to zbMATH identifiers and may contain data conversion errors. It attempts to reflect the references listed in the original paper as accurately as possible without claiming the completeness or perfect precision of the matching.