Streamlined variational inference with higher level random effects. (English) Zbl 07306862

Nolan, Tui H.; Menictas, Marianne; Wand, Matt P.

J. Mach. Learn. Res. 21, Paper No. 157, 62 p. (2020).

Summary: We derive and present explicit algorithms to facilitate streamlined computing for variational inference for models containing higher level random effects. Existing literature is such that streamlined variational inference is restricted to mean field variational Bayes algorithms for two-level random effects models. Here we provide the following extensions: (1) explicit Gaussian response mean field variational Bayes algorithms for three-level models, (2) explicit algorithms for the alternative variational message passing approach in the case of two-level and three-level models, and (3) an explanation of how arbitrarily high levels of nesting can be handled based on the recently published matrix algebraic results of the authors. A pay-off from (2) is simple extension to non-Gaussian response models. In summary, we remove barriers for streamlining variational inference algorithms based on either the mean field variational Bayes approach or the variational message passing approach when higher level random effects are present.

MSC:

68T05

Learning and adaptive systems in artificial intelligence

Keywords:

factor graph fragment; longitudinal data analysis; mixed models; multilevel models; variational message passing

Software:

RStan; R; RcppArmadillo; Rcpp; RcppDist

References:

[1]	A. Atay-Kayis and H. Massam. A Monte Carlo method for computing marginal likelihood in nondecomposable gaussian graphical models.Biometrika, 92:317-335, 2005. · Zbl 1094.62028
[2]	B.H. Baltagi.Econometric Analysis of Panel Data, Fifth Edition. John Wiley & Sons, Chichester, U.K., 2013.
[3]	C.M. Bishop.Pattern Recognition and Machine Learning. Springer, New York, 2006. 60 · Zbl 1107.68072
[4]	D.M. Blei, A. Kucukelbir, and J.D. McAuliffe. Variational inference: a review for statisticians.Journal of the American Statistical Association, 112:859-877, 2017.
[5]	J.B. Duck-Mayr.RcppDist:Rcppintegration of additional probability distributions, 2018. URLhttps://CRAN.R-project.org.Rpackage version 0.1.1.
[6]	M. Durban, J. Harezlak, M.P. Wand, and R.J. Carroll. Simple fitting of subject-specific curves for longitudinal data.Statistics in Medicine, 24:1153-1167, 2005.
[7]	D. Eddelbeuttel, R. Francois, J.J. Allaire, K. Ushey, Q. Kou, N. Russell, D. Bates, and J. Chambers.Rcpp: SeamlessRandC++integration, 2020a. URLhttps://www.rcpp. org.Rpackage version 1.0.5.
[8]	D. Eddelbeuttel, R. Francois, D. Bates, and B. Ni.RcppArmadillo:Rcppintegration for the Armadillotemplated linear algebra library, 2020b. URLhttps://CRAN.R-project.org. Rpackage version 0.9.900.2.0.
[9]	G. Fitzmaurice, M. Davidian, G. Verbeke, and G. Molenberghs (Editors).Longitudinal Data Analysis. Chapman & Hall/CRC, Boca Raton, Florida, 2008. · Zbl 1144.62087
[10]	J.E. Gentle.Matrix Algebra. Springer, New York, 2007. · Zbl 1133.15001
[11]	H. Goldstein.Multilevel Statistical Models, Fourth Edition. John Wiley & Sons, Chichester, U.K., 2010.
[12]	G.H. Golub and C.F. Van Loan.Matrix Computations. The Johns Hopkins University Press, Baltimore, 1989. · Zbl 0733.65016
[13]	D.A. Harville.Matrix Algebra from a Statistician’s Perspective. Springer, New York, 2008. · Zbl 1142.15001
[14]	A. Huang and M.P. Wand. Simple marginally noninformative prior distributions for covariance matrices.Bayesian Analysis, 8:439-452, 2013. · Zbl 1329.62135
[15]	M.A. Klebanoff. The Collaborative Perinatal Project: a 50-year retrospective.Paediatric and Perinatal Epidemiology, 23(1):2-8, 2009.
[16]	C.Y.Y. Lee and M.P. Wand. Streamlined mean field variational bayes for longitudinal and multilevel data analysis.Biometrical Journal, 58:868-895, 2016. · Zbl 1386.62005
[17]	L. Maestrini and M.P. Wand. The Inverse G-Wishart distribution and variational message passing. 2020. URLhttps://arxiv.org/abs/2005.09876.
[18]	M. Menictas and M.P. Wand. Variational inference for marginal longitudinal semiparametric regression.Stat, 2:61-71, 2013.
[19]	M. Menictas, T.H. Nolan, D.G. Simpson, and M.P. Wand. Streamlined variational inference for higher level group-specific curve models.Statistical Modelling, 2020. in press.
[20]	T. Minka. Divergence measures and message passing.Microsoft Research Technical Report Series, MSR-TR-2005-173:1-17, 2005.
[21]	T.H. Nolan and M.P. Wand. Streamlined solutions to multilevel sparse matrix problems. ANZIAM Journal, 62:18-41, 2020. · Zbl 1450.65030
[22]	J.T. Ormerod and M.P. Wand. Explaining variational approximations.The American Statistician, 64:140-153, 2010. · Zbl 1200.65007
[23]	J.C. Pinheiro and D.M. Bates.Mixed-Effects Models inS. Springer-Verlag, New York, 2000. · Zbl 0953.62065
[24]	J.N.K. Rao and I. Molina.Small Area Estimation, Second Edition. John Wiley & Sons, Hoboken, New Jersey, 2015.
[25]	G.K. Robinson. That BLUP is a good thing: the estimation of random effects.Statistical Science, 6:15-51, 1991. · Zbl 0955.62500
[26]	H. Rue and L. Held.Gaussian Markov Random Fields. Chapman & Hall/CRC, Boca Raton, Florida, 2005. · Zbl 1093.60003
[27]	RCore Team.R: A language and environment for statistical computing. Vienna, Austria, 2020. URLhttps://www.R-project.org.RFoundation for Statistical Computing.
[28]	StanDevelopment Team.RStan:RStan: theRinterface toStan, 2020.URLhttps: //mc-stan.org.Rpackage version 2.21.2.
[29]	M.P. Wand. Fast approximate inference for arbitrarily large semiparametric regression models via message passing (with discussion).Journal of the American Statistical Association, 112:137-168, 2017.
[30]	M.P. Wand and J.T. Ormerod. Penalized wavelets: embedding wavelets into semiparametric regression.Electronic Journal of Statistics, 5:1654-1717, 2011. · Zbl 1271.62089