Parallel integrative learning for large-scale multi-response regression with incomplete outcomes. (English) Zbl 07422735

Summary: Multi-task learning is increasingly used to investigate the association structure between multiple responses and a single set of predictor variables in many applications. In the era of big data, the coexistence of incomplete outcomes, large number of responses, and high dimensionality in predictors poses unprecedented challenges in estimation, prediction and computation. In this paper, we propose a scalable and computationally efficient procedure, called PEER, for large-scale multi-response regression with incomplete outcomes, where both the numbers of responses and predictors can be high-dimensional. Motivated by sparse factor regression, we convert the multi-response regression into a set of univariate-response regressions, which can be efficiently implemented in parallel. Under some mild regularity conditions, we show that PEER enjoys nice sampling properties including consistency in estimation, prediction, and variable selection. Extensive simulation studies show that our proposal compares favorably with several existing methods in estimation accuracy, variable selection, and computation efficiency.


62-XX Statistics


rrpack; SOFAR; PEER; secure
Full Text: DOI arXiv


[1] Bickel, P. J.; Ritov, Y.; Tsybakov, A. B., Simultaneous analysis of lasso and Dantzig selector, Ann. Stat., 37, 1705-1732 (2009) · Zbl 1173.62022
[2] Bunea, F.; She, Y.; Wegkamp, M. H., Optimal selection of reduced rank estimators of high-dimensional matrices, Ann. Stat., 39, 1282-1309 (2011) · Zbl 1216.62086
[3] Bunea, F.; She, Y.; Wegkamp, M. H., Joint variable and rank selection for parsimonious estimation of high-dimensional matrices, Ann. Stat., 40, 2359-2388 (2012) · Zbl 1373.62246
[4] Bunea, F.; Tsybakov, A.; Wegkamp, M., Sparsity oracle inequalities for the lasso, Electron. J. Stat., 1, 169-194 (2007) · Zbl 1146.62028
[5] Candès, E.; Tao, T., The Dantzig selector: statistical estimation when p is much larger than n, Ann. Stat., 35, 2313-2351 (2007) · Zbl 1139.62019
[6] Chen, K., 2019. rrpack: reduced-rank regression. R package version 0.1-11.
[7] Chen, K.; Chan, K.-S., A note on rank reduction in sparse multivariate regression, J. Stat. Theory Pract., 10, 100-120 (2016) · Zbl 1420.62284
[8] Chen, K.; Chan, K.-S.; Stenseth, N. C., Reduced rank stochastic regression with a sparse singular value decomposition, J. R. Stat. Soc., Ser. B, Stat. Methodol., 74, 203-221 (2012) · Zbl 1411.62182
[9] Chen, L.; Huang, J. Z., Sparse reduced-rank regression for simultaneous dimension reduction and variable selection, J. Am. Stat. Assoc., 107, 1533-1545 (2012) · Zbl 1258.62075
[10] Dette, H.; Hoyden, L.; Kuhnt, S.; Schorning, K., Optimal designs for multi-response generalized linear models with applications in thermal spraying (2013), Preprint
[11] Efron, B.; Hastie, T.; Johnstone, I.; Tibshirani, R., Least angle regression, Ann. Stat., 32, 407-499 (2004) · Zbl 1091.62054
[12] Fan, J.; Gong, W.; Zhu, Z., Generalized high-dimensional trace regression via nuclear norm regularization, J. Econom., 212, 177-202 (2019) · Zbl 1452.62536
[13] Fan, J.; Li, R., Variable selection via nonconcave penalized likelihood and its oracle properties, J. Am. Stat. Assoc., 96, 1348-1360 (2001) · Zbl 1073.62547
[14] Fan, Y.; Lv, J., Asymptotic equivalence of regularization methods in thresholded parameter space, J. Am. Stat. Assoc., 108, 1044-1061 (2013) · Zbl 06224986
[15] Fan, Y.; Tang, C. Y., Tuning parameter selection in high dimensional penalized likelihood, J. R. Stat. Soc., Ser. B, Stat. Methodol., 75, 531-552 (2013) · Zbl 1411.62216
[16] Friedman, J.; Hastie, T.; Höfling, H.; Tibshirani, R., Pathwise coordinate optimization, Ann. Appl. Stat., 1, 302-332 (2007) · Zbl 1378.90064
[17] Van de Geer, S. A., High-dimensional generalized linear models and the lasso, Ann. Stat., 36, 614-645 (2008) · Zbl 1138.62323
[18] He, K.; Lian, H.; Ma, S.; Huang, J. Z., Dimensionality reduction and variable selection in multivariate varying-coefficient models with a large number of covariates, J. Am. Stat. Assoc., 113, 746-754 (2018) · Zbl 1398.62136
[19] Hilafu, H.; Safo, S. E.; Haine, L., Sparse reduced-rank regression for integrating omics data, BMC Bioinform., 21, 1-17 (2020)
[20] Izenman, A. J., Reduced-rank regression for the multivariate linear model, J. Multivar. Anal., 5, 248-264 (1975) · Zbl 0313.62042
[21] Kim, S.; Sohn, K.-A.; Xing, E. P., A multivariate regression approach to association analysis of a quantitative trait network, Bioinformatics, 25, i204-i212 (2009)
[22] Klopp, O., Noisy low-rank matrix completion with general sampling distribution, Bernoulli, 20, 282-303 (2014) · Zbl 1400.62115
[23] Lafond, J., Low rank matrix completion with exponential family noise, Proc. Mach. Learn. Res., 40, 1224-1243 (2015)
[24] Lee, T. I.; Rinaldi, N. J.; Robert, F.; Odom, D. T.; Bar-Joseph, Z.; Gerber, G. K.; Hannett, N. M., Transcriptional regulatory networks in Saccharomyces cerevisiae, Science, 298, 799-804 (2002)
[25] Liu, X.; Ma, S.; Chen, K., Multivariate functional regression via nested reduced-rank regularization (2020), Preprint
[26] Luo, C.; Liang, J.; Li, G.; Wang, F.; Zhang, C.; Dey, D. K.; Chen, K., Leveraging mixed and incomplete outcomes via reduced-rank modeling, J. Multivar. Anal., 167, 378-394 (2018) · Zbl 1395.62135
[27] Lv, J.; Fan, Y., A unified approach to model selection and sparse recovery using regularized least squares, Ann. Stat., 37, 3498-3528 (2009) · Zbl 1369.62156
[28] Ma, S.; Linton, O.; Gao, J., Estimation and inference in semiparametric quantile factor models, J. Econom., 222, 295-323 (2021) · Zbl 1471.62332
[29] Mishra, A., Chen, K., 2017. secure: sequential co-sparse factor regression. R package version 0.5.
[30] Mishra, A.; Dey, D. K.; Chen, K., Sequential co-sparse factor regression, J. Comput. Graph. Stat., 26, 814-825 (2017)
[31] Rothman, A. J.; Levina, E.; Zhu, J., Sparse multivariate regression with covariance estimation, J. Comput. Graph. Stat., 19, 947-962 (2010)
[32] Rudelson, M.; Vershynin, R., Non-asymptotic theory of random matrices: extreme singular values, (Proceedings of the International Congress of Mathematicians (2010), World Scientific), 1576-1602 · Zbl 1227.60011
[33] Spellman, P. T.; Sherlock, G.; Zhang, M. Q.; Iyer, V. R.; Anders, K.; Eisen, M. B.; Brown, P. O.; Botstein, D.; Futcher, B., Comprehensive identification of cell cycle-regulated genes of the yeast Saccharomyces cerevisiae by microarray hybridization, Mol. Biol. Cell, 9, 3273-3297 (1998)
[34] Stewart, G. W., Perturbation theory for the singular value decomposition (1998), Technical Report
[35] Tibshirani, R., Regression shrinkage and selection via the lasso, J. R. Stat. Soc., Ser. B, Stat. Methodol., 58, 267-288 (1996) · Zbl 0850.62538
[36] Uematsu, Y.; Fan, Y.; Chen, K.; Lv, J.; Lin, W., SOFAR: large-scale association network learning, IEEE Trans. Inf. Theory, 65, 4924-4939 (2019) · Zbl 1432.68402
[37] Velu, R.; Reinsel, G. C., Multivariate Reduced-Rank Regression: Theory and Applications, vol. 136 (2013), Springer Science & Business Media
[38] Wang, L.; Chen, G.; Li, H., Group SCAD regression analysis for microarray time course gene expression data, Bioinformatics, 23, 1486-1494 (2007)
[39] Wu, T. T.; Lange, K., Coordinate descent algorithms for lasso penalized regression, Ann. Appl. Stat., 2, 224-244 (2008) · Zbl 1137.62045
[40] Yu, Y.; Wang, T.; Samworth, R. J., A useful variant of the Davis-Kahan theorem for statisticians, Biometrika, 102, 315-323 (2015) · Zbl 1452.15010
[41] Zhang, C.-H., Nearly unbiased variable selection under minimax concave penalty, Ann. Stat., 38, 894-942 (2010) · Zbl 1183.62120
[42] Zhang, J.; Feng, Z.; Peng, H., Estimation and hypothesis test for partial linear multiplicative models, Comput. Stat. Data Anal., 128, 87-103 (2018) · Zbl 1469.62177
[43] Zhao, P.; Yu, B., On model selection consistency of lasso, J. Mach. Learn. Res., 7, 2541-2563 (2006) · Zbl 1222.62008
[44] Zheng, Z.; Bahadori, M. T.; Liu, Y.; Lv, J., Scalable interpretable multi-response regression via SEED, J. Mach. Learn. Res., 20, 1-34 (2019) · Zbl 1441.62214
[45] Zheng, Z.; Lv, J.; Lin, W., Nonsparse learning with latent variables, Oper. Res., 69, 346-359 (2021)
[46] Zhu, X.; Huang, D.; Pan, R.; Wang, H., Multivariate spatial autoregressive model for large scale social networks, J. Econom., 215, 591-606 (2020) · Zbl 1456.62229
[47] Zhu, Y.; Shen, X.; Ye, C., Personalized prediction and sparsity pursuit in latent factor models, J. Am. Stat. Assoc., 111, 241-252 (2016)
[48] Zou, H., The adaptive lasso and its oracle properties, J. Am. Stat. Assoc., 101, 1418-1429 (2006) · Zbl 1171.62326
This reference list is based on information provided by the publisher or from digital mathematics libraries. Its items are heuristically matched to zbMATH identifiers and may contain data conversion errors. It attempts to reflect the references listed in the original paper as accurately as possible without claiming the completeness or perfect precision of the matching.