High-dimensional multivariate posterior consistency under global-local shrinkage priors. (English) Zbl 1403.62134

Summary: We consider sparse Bayesian estimation in the classical multivariate linear regression model with \(p\) regressors and \(q\) response variables. In univariate Bayesian linear regression with a single response \(y\), shrinkage priors which can be expressed as scale mixtures of normal densities are popular for obtaining sparse estimates of the coefficients. In this paper, we extend the use of these priors to the multivariate case to estimate a \(p \times q\) coefficients matrix \(\mathbf{B}\). We derive sufficient conditions for posterior consistency under the Bayesian multivariate linear regression framework and prove that our method achieves posterior consistency even when \(p > n\) and even when \(p\) grows at nearly exponential rate with the sample size. We derive an efficient Gibbs sampling algorithm and provide the implementation in a comprehensive \(\mathsf{R}\) package called MBSP. Finally, we demonstrate through simulations and data analysis that our model has excellent finite sample performance.


62J07 Ridge regression; shrinkage estimators (Lasso)
62F15 Bayesian inference
62H12 Estimation in multivariate analysis
62F12 Asymptotic properties of parametric estimators


R2GUESS; R; MBSP; glmnet
Full Text: DOI arXiv


[1] Armagan, A.; Clyde, M.; Dunson, D. B., Generalized beta mixtures of gaussians, (Shawe-Taylor, J.; Zemel, R. S.; Bartlett, P.; Pereira, F. C.N.; Weinberger, K. Q., Advances in Neural Information Processing Systems, Vol. 24, (2011)), 523-531
[2] Armagan, A.; Dunson, D. B.; Lee, J., Generalized double Pareto shrinkage, Statist. Sinica, 23, 119-143, (2013) · Zbl 1259.62061
[3] Armagan, A.; Dunson, D. B.; Lee, J.; Bajwa, W. U.; Strawn, N., Posterior consistency in linear models under shrinkage priors, Biometrika, 100, 1011-1018, (2013) · Zbl 1279.62139
[4] Berger, J. O., A robust generalized Bayes estimator and confidence region for a multivariate normal mean, Ann. Statist., 8, 716-761, (1980) · Zbl 0464.62026
[5] Bhadra, A.; Datta, J.; Polson, N. G.; Willard, B., The horseshoe+ estimator of ultra-sparse signals, Bayesian Anal., 12, 1105-1131, (2017) · Zbl 1384.62079
[6] Bhadra, A.; Mallick, B. K., Joint high-dimensional Bayesian variable and covariance selection with an application to EQTL analysis, Biometrics, 69, 447-457, (2013) · Zbl 1274.62722
[7] Bhattacharya, A.; Chakraborty, A.; Mallick, B. K., Fast sampling with Gaussian scale mixture priors in high-dimensional regression, Biometrika, 103, 985-991, (2016)
[8] Bingham, N. H.; Goldie, C. M.; Teugels, J. L., (Regular Variation, Encyclopedia of Mathematics and its Applications, (1987), Cambridge University Press) · Zbl 0617.26001
[9] Brown, P. J.; Vannucci, M.; Fearn, T., Multivariate Bayesian variable selection and prediction, J. R. Stat. Soc. Ser. B, 60, 627-641, (1998) · Zbl 0909.62022
[10] Bunea, F.; She, Y.; Wegkamp, M. H., Joint variable and rank selection for parsimonious estimation of high-dimensional matrices, Ann. Statist., 40, 2359-2388, (2012) · Zbl 1373.62246
[11] Candès, E.; Tao, T., The Dantzig selector: statistical estimation when \(p\) is much larger than \(n\), Ann. Statist., 35, 2313-2351, (2007) · Zbl 1139.62019
[12] Carvalho, C. M.; Polson, N. G.; Scott, J. G., The horseshoe estimator for sparse signals, Biometrika, 97, 465-480, (2010) · Zbl 1406.62021
[13] Castillo, I.; Schmidt-Hieber, J.; van der Vaart, A. W., Bayesian linear regression with sparse priors, Ann. Statist., 43, 1986-2018, (2015) · Zbl 1486.62197
[14] Chen, L.; Huang, J. Z., Sparse reduced-rank regression for simultaneous dimension reduction and variable selection, J. Amer. Statist. Assoc., 107, 1533-1545, (2012) · Zbl 1258.62075
[15] Chun, H.; Keleş, S., Sparse partial least squares regression for simultaneous dimension reduction and variable selection, J. R. Stat. Soc. Ser. B, 72, 3-25, (2010)
[16] Datta, J.; Ghosh, J. K., Asymptotic properties of bayesbayes risk for the horseshoe prior, Bayesian Anal., 8, 111-132, (2013) · Zbl 1329.62122
[17] Fan, J.; Lv, J., Sure independence screening for ultrahigh dimensional feature space, J. R. Stat. Soc. Ser. B, 70, 849-911, (2008)
[18] Fan, J.; Song, R., Sure independence screening in generalized linear models with \(n p\)-dimensionality, Ann. Statist., 38, 3567-3604, (2010) · Zbl 1206.68157
[19] Friedman, J.; Hastie, T.; Tibshirani, R. J., Regularization paths for generalized linear models via coordinate descent, J. Stat. Softw., 33, 1-22, (2010)
[20] George, E. I.; McCulloch, R. E., Variable selection via Gibbs sampling, J. Amer. Statist. Assoc., 88, 881-889, (1993)
[21] Ghosh, P.; Chakrabarti, A., Asymptotic optimality of one-group shrinkage priors in sparse high-dimensional problems, Bayesian Anal., 12, 1133-1161, (2017) · Zbl 1384.62087
[22] Ghosh, P.; Tang, X.; Ghosh, M.; Chakrabarti, A., Asymptotic properties of Bayes risk of a general class of shrinkage priors in multiple hypothesis testing under sparsity, Bayesian Anal., 11, 753-796, (2016) · Zbl 1359.62309
[23] Goh, G.; Dey, D. K.; Chen, K., Bayesian sparse reduced rank multivariate regression, J. Multivariate Anal., 157, 14-28, (2017) · Zbl 1362.62140
[24] Griffin, J. E.; Brown, P. J., Some priors for sparse regression modelling, Bayesian Anal., 8, 691-702, (2013) · Zbl 1329.62132
[25] Ishwaran, H.; Rao, J. S., Spike and slab variable selection: frequentist and Bayesian strategies, Ann. Statist., 33, 730-773, (2005) · Zbl 1068.62079
[26] Li, Y.; Nan, B.; Zhu, J., Multivariate sparse group lasso for the multivariate multiple linear regression with an arbitrary group structure, Biometrics, 71, 354-363, (2015) · Zbl 1390.62285
[27] Libby, D. L.; Novick, M. R., Multivariate generalized beta distributions with applications to utility assessment, J. Educ. Stat., 7, 271-294, (1982)
[28] Liquet, B.; Bottolo, L.; Campanella, G.; Richardson, S.; Chadeau-Hyam, M., : A graphics processing unit-based R package for Bayesian variable selection regression of multivariate responses, J. Stat. Softw., 69, 1-32, (2016)
[29] Liquet, B.; Mengersen, K.; Pettitt, A. N.; Sutton, M., Bayesian variable selection regression of multivariate responses for group data, Bayesian Anal., 12, 1039-1067, (2017) · Zbl 1384.62259
[30] Mitchell, T. J.; Beauchamp, J. J., Bayesian variable selection in linear regression, J. Amer. Statist. Assoc., 83, 1023-1032, (1988) · Zbl 0673.62051
[31] Narisetty, N. N.; He, X., Bayesian variable selection with shrinking and diffusing priors, Ann. Statist., 42, 789-817, (2014) · Zbl 1302.62158
[32] Park, T.; Casella, G., The Bayesian lasso, J. Amer. Statist. Assoc., 103, 681-686, (2008) · Zbl 1330.62292
[33] Polson, N. G.; Scott, J. G., On the half-Cauchy prior for a global scale parameter, Bayesian Anal., 7, 887-902, (2012) · Zbl 1330.62148
[34] Ročková, V.; George, E. I., The spike-and-slab lasso, J. Amer. Statist. Assoc., (2016), To appear
[35] Rothman, A. J.; Levina, E.; Zhu, J., Sparse multivariate regression with covariance estimation, J. Comput. Graph. Statist., 19, 947-962, (2010)
[36] J.-B. Salomond, Risk quantification for the thresholding rule for multiple testing using Gaussian scale mixtures, ArXiv e-prints, 2017.
[37] Strawderman, W. E., Proper Bayes minimax estimators of the multivariate normal mean, Ann. Math. Stat., 42, 385-388, (1971) · Zbl 0222.62006
[38] Sun, T.; Zhang, C.-H., Scaled sparse linear regression, Biometrika, 99, 879-898, (2012) · Zbl 1452.62515
[39] Tang, X.; Xu, X.; Ghosh, M.; Ghosh, P., Bayesian variable selection and estimation based on global-local shrinkage priors, Sankhyā A, (2017)
[40] Tibshirani, R. J., Regression shrinkage and selection via the lasso, J. R. Stat. Soc. Ser. B, 58, 267-288, (1996) · Zbl 0850.62538
[41] van der Pas, S. L.; Kleijn, B. J.K.; van der Vaart, A. W., The horseshoe estimator: posterior concentration around nearly black vectors, Electron. J. Stat., 8, 2585-2618, (2014) · Zbl 1309.62060
[42] van der Pas, S. L.; Salomond, J.-B.; Schmidt-Hieber, J., Conditions for posterior contraction in the sparse normal means problem, Electron. J. Stat., 10, 976-1000, (2016) · Zbl 1343.62012
[43] van der Pas, S. L.; Szabó, B.; van der Vaart, A. W., Adaptive posterior contraction rates for the horseshoe, Electron. J. Stat., 11, 3196-3225, (2017) · Zbl 1373.62140
[44] van der Pas, S. L.; Szabó, B.; van der Vaart, A. W., Uncertainty quantification for the horseshoe (with discussion), Bayesian Anal., 12, 1221-1274, (2017) · Zbl 1384.62155
[45] Wilms, I.; Croux, C., An algorithm for the multivariate group lasso with covariance estimation, J. Appl. Stat., 45, 668-681, (2018)
[46] Xu, X.; Ghosh, M., Bayesian variable selection and estimation for group lasso, Bayesian Anal., 10, 909-936, (2015) · Zbl 1334.62132
[47] Yuan, M.; Lin, Y., Model selection and estimation in regression with grouped variables, J. R. Stat. Soc. Ser. B, 68, 49-67, (2006) · Zbl 1141.62030
[48] Zellner, A., On assessing prior distributions and Bayesian regression analysis with \(g\) prior distributions, (Goel, P. K.; Zellner, A., Bayesian Inference and Decision Techniques: Essays in Honor of Bruno de Finetti, Studies in Bayesian Econometrics, (1986)), 233-243 · Zbl 0655.62071
[49] Zhu, H.; Khondker, Z.; Lu, Z.; Ibrahim, J. G., Bayesian generalized low rank regression models for neuroimaging phenotypes and genetic markers, J. Amer. Statist. Assoc., 109, 990-997, (2014) · Zbl 1368.62298
[50] Zou, H., The adaptive lasso and its oracle properties, J. Amer. Statist. Assoc., 101, 1418-1429, (2006) · Zbl 1171.62326
[51] Zou, H.; Hastie, T., Regularization and variable selection via the elastic net, J. R. Stat. Soc. Ser. B, 67, 301-320, (2005) · Zbl 1069.62054
This reference list is based on information provided by the publisher or from digital mathematics libraries. Its items are heuristically matched to zbMATH identifiers and may contain data conversion errors. It attempts to reflect the references listed in the original paper as accurately as possible without claiming the completeness or perfect precision of the matching.