×

Variational inference for heteroscedastic semiparametric regression. (English) Zbl 1331.62236

Summary: We develop fast mean field variational methodology for Bayesian heteroscedastic semiparametric regression, in which both the mean and variance are smooth, but otherwise arbitrary, functions of the predictors. Our resulting algorithms are purely algebraic, devoid of numerical integration and Monte Carlo sampling. The locality property of mean field variational Bayes implies that the methodology also applies to larger models possessing variance function components. Simulation studies indicate good to excellent accuracy, and considerable time savings compared with Markov chain Monte Carlo. We also provide some illustrations from applications.

MSC:

62G08 Nonparametric regression and quantile regression
62F15 Bayesian inference
PDFBibTeX XMLCite
Full Text: DOI

References:

[1] Barber, Advances in Neural Information Processing Systems pp 395– (1997)
[2] Challis, Gaussian Kullback-Leibler approximate inference, J. Mach. Learn. Res. 14 pp 2239– (2013)
[3] Crainiceanu, Spatially adaptive Bayesian penalized splines with heteroscedastic errors, J. Comp. Graph. Stat. 16 pp 265– (2007) · doi:10.1198/106186007X208768
[4] Cressie, Statistics for Spatial Data (1993)
[5] Faes, Variational Bayesian inference for parametric and nonparametric regression with missing data, J. Amer. Statist. Assoc. 106 pp 959– (2011) · Zbl 1229.62028 · doi:10.1198/jasa.2011.tm10301
[6] Hinton, Proceedings of the 6th Annual Association Computing Machinery Conference Computational Learning Theory pp 5– (1993)
[7] Knowles, Advances in Neural Information Processing Systems pp 1701– (2011)
[8] Lázaro-Gredilla , M Titsias , M.K 2011 Variational heteroscedastic Gaussian process regression Proceedings of the 28th International Conference on Machine Learning
[9] Luts, Real-time semiparametric regression, J. Comp. Graph. Stat. 23 pp 589– (2014) · doi:10.1080/10618600.2013.810150
[10] Luts , J. Wang , S.S.J. Ormerod , J.T Wand , M.P 2015 Semiparametric regression analysis via Infer.NET
[11] Magnus, Matrix Differential Calculus with Applications in Statistics and Econometrics (1999) · Zbl 0912.15003
[12] Marley, Non-standard semiparametric regression via BRugs, J. Stat. Softw. 37 pp 1– (2010) · doi:10.18637/jss.v037.i05
[13] Menictas , M 2015 Variational inference for heteroscedastic and longitudinal regression models (PhD Thesis) · Zbl 1331.62236
[14] Menictas, Variational inference for marginal longitudinal semiparametric regression, Statistics 2 pp 61– (2013) · doi:10.1002/sta4.18
[15] Minka , T Winn , J 2008 Gates: A graphical notation for mixture models
[16] Minka , T. Winn , J. Guiver , J Knowles , D 2014 Infer.NET 2.6. http://research.microsoft.com/infernet
[17] Nott, Variational approximation for heteroscedastic linear models and matching pursuit algorithms, Stat. Comput. 22 pp 497– (2012) · Zbl 1322.62031 · doi:10.1007/s11222-011-9243-2
[18] Opper, The variational Gaussian approximation revisited, Neural Comput. 21 pp 786– (2009) · Zbl 1178.68450 · doi:10.1162/neco.2008.08-07-592
[19] Ormerod, Explaining variational approximations, Amer. Statist. 64 pp 140– (2010) · Zbl 1200.65007 · doi:10.1198/tast.2010.09058
[20] Pebesma, Multivariate geostatistics in R: The gstat package, Comput. Geosci. 30 pp 683– (2004) · doi:10.1016/j.cageo.2004.03.012
[21] Pebesma, Geostatistics for Environmental Applications: Proceedings of the Fifth European Conference on Geostatistics for Environmental Applications pp 367– (2005) · doi:10.1007/3-540-26535-X_31
[22] R Core Team 2015 R: A Language and Environment for Statistical Computing http://www.R-project.org
[23] Raiko, Building blocks for variational Bayesian learning of latent variable models, J. Mach. Learn. Res. 8 pp 155– (2007) · Zbl 1222.62040
[24] Rigby, Generalized additive models for location, scale and shape, J. R. Statist. Soc. Ser. C. Appl. Statist. 54 pp 507– (2005) · Zbl 1490.62201 · doi:10.1111/j.1467-9876.2005.00510.x
[25] Ruppert, Semiparametric Regression (2003) · Zbl 1038.62042 · doi:10.1017/CBO9780511755453
[26] Ruppert, Semiparametric regression during 2003-2007, Electron. J. Stat. 3 pp 1193– (2009) · Zbl 1326.62094 · doi:10.1214/09-EJS525
[27] Spiegelhalter , D.J. Thomas , A. Best , N.G. Gilks , W.R Lunn , D 2003 BUGS: Bayesian Inference Using Gibbs Sampling http://www.mrc-bsu.cam.ac.uk/bugs
[28] Stan Development Team 2013 Stan: A C++ library for probability and sampling http://mc-stan.org/
[29] Wainwright, Graphical models, exponential families, and variational inference, Found. Trends Mach. Learn. 1 pp 1– (2008) · Zbl 1193.62107 · doi:10.1561/2200000001
[30] Wand, Semiparametric regression and graphical models, Aust. N. Z. J. Stat. 51 pp 9– (2009) · Zbl 1334.62012 · doi:10.1111/j.1467-842X.2009.00538.x
[31] Wand, Fully simplified Multivariate Normal updates in non-conjugate variational message passing, J. Mach. Learn. Res. 15 pp 1351– (2014)
[32] Wand, Kernel Smoothing (1995) · doi:10.1007/978-1-4899-4493-1
[33] Wand, On semiparametric regression with O’Sullivan penalised splines, Aust. N. Z. J. Stat. 50 pp 179– (2008) · Zbl 1146.62030 · doi:10.1111/j.1467-842X.2008.00507.x
[34] Wand, Mean field variational Bayes for elaborate distributions, Bayesian Anal. 6 pp 847– (2011) · Zbl 1330.62158 · doi:10.1214/11-BA631
This reference list is based on information provided by the publisher or from digital mathematics libraries. Its items are heuristically matched to zbMATH identifiers and may contain data conversion errors. In some cases that data have been complemented/enhanced by data from zbMATH Open. This attempts to reflect the references listed in the original paper as accurately as possible without claiming completeness or a perfect matching.