×

Variational Bayes with synthetic likelihood. (English) Zbl 1384.65015

Summary: Synthetic likelihood is an attractive approach to likelihood-free inference when an approximately Gaussian summary statistic for the data, informative for inference about the parameters, is available. The synthetic likelihood method derives an approximate likelihood function from a plug-in normal density estimate for the summary statistic, with plug-in mean and covariance matrix obtained by Monte Carlo simulation from the model. In this article, we develop alternatives to Markov chain Monte Carlo implementations of Bayesian synthetic likelihoods with reduced computational overheads. Our approach uses stochastic gradient variational inference methods for posterior approximation in the synthetic likelihood context, employing unbiased estimates of the log likelihood. We compare the new method with a related likelihood-free variational inference technique in the literature, while at the same time improving the implementation of that approach in a number of ways. These new algorithms are feasible to implement in situations which are challenging for conventional approximate Bayesian computation methods, in terms of the dimensionality of the parameter and summary statistic.

MSC:

65C60 Computational problems in statistics (MSC2010)
62F15 Bayesian inference

Software:

ADADELTA; ADVI; PRMLT; GPS-ABC
PDFBibTeX XMLCite
Full Text: DOI arXiv

References:

[1] Adler, R.J., Feldman, R.E., Taqqu, M.S. (eds.): A Practical Guide to Heavy Tails: Statistical Techniques and Applications. Birkhauser Boston Inc., Cambridge (1998) · Zbl 0901.00010
[2] Allingham, D.R., King, A.R., Mengersen, K.L.: Bayesian estimation of quantile distributions. Stat. Comput. 19, 189-201 (2009)
[3] Amari, S.: Natural gradient works efficiently in learning. Neural Comput. 10, 251-276 (1998)
[4] Andrieu, C., Roberts, G.O.: The pseudo-marginal approach for efficient Monte Carlo computations. Ann. Stat. 37(2), 697-725 (2009) · Zbl 1185.60083
[5] Barthelmé, S., Chopin, N.: Expectation propagation for likelihood-free inference. J. Am. Stat. Assoc. 109(505), 315-333 (2014) · Zbl 1367.62063
[6] Beaumont, M.A.: Estimation of population growth or decline in genetically monitored populations. Genetics 164(3), 1139-1160 (2003)
[7] Bishop, C.M.: Pattern Recognition and Machine Learning. Springer, Berlin (2006) · Zbl 1107.68072
[8] Blum, M.G.B., Nunes, M.A., Prangle, D., Sisson, S.A.: A comparative review of dimension reduction methods in approximate Bayesian computation. Stat. Sci. 28(2), 189-208 (2013) · Zbl 1331.62123
[9] Bottou, L.: Large-scale machine learning with stochastic gradient descent. In: Lechevallier, Y., Saporta, G. (eds.) Proceedings of the 19th International Conference on Computational Statistics (COMPSTAT’2010), pp 177-187. Springer, Berlin (2010) · Zbl 1436.68293
[10] Box, G.E.P.: Sampling and Bayes’ inference in scientific modelling and robustness (with discussion). J. R. Stat. Soc. Ser. A 143, 383-430 (1980) · Zbl 0471.62036
[11] Brown, V.L., Drake, J.M., Barton, H.D., Stallknecht, D.E., Brown, J.D., Rohani, P.: Neutrality, cross-immunity and subtype dominance in avian influenza viruses. PLOS ONE 9(2), 1-10 (2014)
[12] Doucet, A., Pitt, M.K., Deligiannidis, G., Kohn, R.: Efficient implementation of Markov chain Monte Carlo when using an unbiased likelihood estimator. Biometrika 102(2), 295-313 (2015). doi:10.1093/biomet/asu075 · Zbl 1452.62055
[13] Drovandi, C.C., Pettitt, A.N.: Likelihood-free Bayesian estimation of multivariate quantile distributions. Comput. Stat. Data Anal. 55, 2541-2556 (2011) · Zbl 1464.62062
[14] Drovandi, C.C., Pettitt, A.N., Faddy, M.J.: Approximate Bayesian computation using indirect inference. J. R. Stat. Soc. Ser. C (Appl. Stat.) 60(3), 503-524 (2011)
[15] Dutta, R., Corander, J., Kaski, S., Gutmann, M.U.: Likelihood-Free Inference by Penalised Logistic Regression. arXiv:1611.10242 (2016) · Zbl 1384.62089
[16] Everitt, R.G., Johansen, A.M., Rowing, E., Evdemon-Hogan, M.: Bayesian model comparison with un-normalised likelihoods. Stat. Comput. 27(2), 403-422 (2017) · Zbl 1505.62139
[17] Fasiolo, M., Pya, N., Wood, S.N.: A comparison of inferential methods for highly nonlinear state space models in ecology and epidemiology. Stat. Sci. 31, 96-118 (2016a) · Zbl 1442.62349
[18] Fasiolo, M., Wood, S.N., Hartig, F., Bravington, M.V.: An extended empirical saddlepoint approximation for intractable likelihoods. arXiv:1601.01849 (2016b) · Zbl 1395.62338
[19] Fisher, R.A., Yates, F.: Statistical Tables for Biological, Agricultural and Medical Research. Hafner, New York (1948) · Zbl 0030.31503
[20] Gelman, A., Meng, X.L., Stern, H.: Posterior predictive assessment of model fitness via realized discrepancies. Stat. Sin. 6, 733-807 (1996) · Zbl 0859.62028
[21] Ghurye, S.G., Olkin, I.: Unbiased estimation of some multivariate probability densities and related functions. Ann. Math. Stat. 40(4), 1261-1271 (1969) · Zbl 0202.17103
[22] Gunawan, D., Tran, M.N., Kohn, R.: Fast inference for intractable likelihood problems using variational Bayes. Working Paper, Discipline of Business Analytics, University of Sydney. http://hdl.handle.net/2123/14594 (2016)
[23] Gutmann, M.U., Corander, J.: Bayesian optimization for likelihood-free inference of simulator-based statistical models. J. Mach. Learn. Res. 17(125), 1-47 (2015) · Zbl 1392.62072
[24] Hartig, F., Dislich, C., Wiegand, T., Huth, A.: Technical note: approximate Bayesian parameterization of a process-based tropical forest model. Biogeosciences 11, 1261-1272 (2014)
[25] Hoffman, M.D., Blei, D.M., Wang, C., Paisley, J.: Stochastic variational inference. J. Mach. Learn. Res. 14(1), 1303-1347 (2013) · Zbl 1317.68163
[26] Ji, C., Shen, H., West, M.: Bounded approximations for marginal likelihoods. Technical Report 10-05, Institute of Decision Sciences, Duke University. http://ftp.stat.duke.edu/WorkingPapers/10-05.html (2010) · Zbl 1443.62499
[27] Joe, H.: Multivariate models and dependence concepts. Chapman & Hall, London (1997) · Zbl 0990.62517
[28] Kingma, D.P., Welling, M.: Auto-encoding variational Bayes. arXiv: 1312.6114 (2013)
[29] Kucukelbir, A., Tran, D., Ranganath, R., Gelman, A., Blei, D.M.: Automatic differentiation variational inference. J. Mach. Learn. Res. 18(14), 1-45 (2017) · Zbl 1437.62109
[30] Li, J., Nott, D.J., Fan, Y., Sisson, S.A.: Extending approximate Bayesian computation methods to high dimensions via Gaussian copula. Comput. Stat. Data Anal. 106, 77-89 (2017) · Zbl 1466.62136
[31] Magnus, J.R., Neudecker, H.: Matrix Differential Calculus with Applications in Statistics and Econometrics. Wiley, New York (1999) · Zbl 0912.15003
[32] Marin, J.M., Pudlo, P., Robert, C.P., Ryder, R.J.: Approximate Bayesian computational methods. Stat. Comput. 22(6), 1167-1180 (2012) · Zbl 1252.62022
[33] McCulloch, J.: Simple consistent estimators of stable distribution parameters. Commun. Stat. Simul. Comput. 15(4), 1109-1136 (1986) · Zbl 0612.62028
[34] Meeds, E., Welling, M.: GPS-ABC: Gaussian process surrogate approximate Bayesian computation. In: Proceedings of the Thirtieth Conference Annual Conference on Uncertainty in Artificial Intelligence (UAI-14), pp. 593-602 (2014) · Zbl 0859.62028
[35] Moores, M.T., Drovandi, C.C., Mengersen, K.L., Robert, C.P.: Pre-processing for approximate Bayesian computation in image analysis. Stat. Comput. 25(1), 23-33 (2015) · Zbl 1331.62158
[36] Moreno, A., Adel, T., Meeds, E., Rehg, J.M., Welling, M.: Automatic variational ABC. arXiv:1606.08549 (2016)
[37] Nott, D., Tan, S., Villani, M., Kohn, R.: Regression density estimation with variational methods and stochastic approximation. J. Comput. Graph. Stat. 21(3), 797-820 (2012)
[38] Ormerod, J., Wand, M.: Explaining variational approximations. Am. Stat. 64, 140-153 (2010) · Zbl 1200.65007
[39] Paisley, J.W., Blei, D.M., Jordan, M.I.: Variational Bayesian inference with stochastic search. In: Proceedings of the 29th International Conference on Machine Learning (ICML-12) (2012)
[40] Peters, G.W., Sisson, S.A., Fan, Y.: Likelihood-free Bayesian inference for \[\alpha\] α-stable models. Comput. Stat. Data Anal. 56, 3743-3756 (2012) · Zbl 1255.62071
[41] Pinheiro, J.C., Bates, D.M.: Unconstrained parametrizations for variance-covariance matrices. Stat. Comput. 6(3), 289-296 (1996)
[42] Pitt, M.K., Silva, RdS, Giordani, P., Kohn, R.: On some properties of Markov chain Monte Carlo simulation methods based on the particle filter. J. Econ. 171(2), 134-151 (2012) · Zbl 1443.62499
[43] Price, L.F., Drovandi, C.C., Lee, A.C., Nott, D.J.: Bayesian synthetic likelihood. J. Comput. Graph. Stat. (2016) (to appear) (2016) · Zbl 07498962
[44] Ranganath, R., Wang, C., Blei, D.M., Xing, E.P.: An adaptive learning rate for stochastic variational inference. In: Proceedings of the 30th International Conference on Machine Learning (ICML-13), pp. 298-306 (2013) · Zbl 1252.62022
[45] Ranganath, R., Gerrish, S., Blei, D.M.: Black box variational inference. Int. Conf. Artif. Intell. Stat. 33, 814-822 (2014)
[46] Rayner, G., MacGillivray, H.: Weighted quantile-based estimation for a class of transformation distributions. Comput. Stat. Data Anal. 39(4), 401-433 (2002) · Zbl 1101.62323
[47] Reserve Bank of Australia (2014) Historical data. http://www.rba.gov.au/statistics/historical-data.html. Accessed 16 Sept 2014
[48] Rezende, D.J., Mohamed, S., Wierstra, D.: Stochastic backpropagation and approximate inference in deep generative models. In: Proceedings of the 31st International Conference on Machine Learning (ICML-14), pp. 1278-1286 (2014)
[49] Ripley, B.D.: Pattern Recognition and Neural Networks. Cambridge University Press, Cambridge (1996) · Zbl 0853.62046
[50] Robbins, H., Monro, S.: A stochastic approximation method. Ann. Math. Stat. 22(3), 400-407 (1951) · Zbl 0054.05901
[51] Salimans, T., Knowles, D.A.: Fixed-form variational posterior approximation through stochastic linear regression. Bayesian Anal. 8(4), 837-882 (2013) · Zbl 1329.62142
[52] Tan, L.S.L., Nott, D.J.: Gaussian variational approximation with sparse precision matrices. Stat. Comput. (2017). doi:10.1007/s11222-017-9729-7 · Zbl 1384.62105
[53] Titsias, M., Lázaro-Gredilla, M.: Doubly stochastic variational Bayes for non-conjugate inference. In: Proceedings of the 31st International Conference on Machine Learning (ICML-14), pp. 1971-1979 (2014) · Zbl 1101.62323
[54] Titsias, M.; Lázaro-Gredilla, M.; Cortes, C. (ed.); Lawrence, ND (ed.); Lee, DD (ed.); Sugiyama, M. (ed.); Garnett, R. (ed.), Local expectation gradients for black box variational inference, No. 28, 2638-2646 (2015), Red Hook
[55] Tran, M.N., Nott, D.J., Kohn, R.: Variational Bayes with intractable likelihood. arXiv:1503.08621v1 (2015)
[56] Tran, M.N., Nott, D.J., Kohn, R.: Variational Bayes with intractable likelihood J. Comput. Graph. Stat. (2016) (to appear)
[57] Wand, M.P.: Fully simplified multivariate normal updates in non-conjugate variational message passing. J. Mach. Learn. Res. 15, 1351-1369 (2014) · Zbl 1319.62066
[58] Wilkinson, R.: Accelerating ABC methods using Gaussian processes. J. Mach. Learn. Res. 33, 1015-1023 (2014)
[59] Wood, S.N.: Statistical inference for noisy nonlinear ecological dynamic systems. Nature 466, 1102-1107 (2010)
[60] Zeiler, M.D.: ADADELTA: an adaptive learning rate method. arXiv:1212.5701 (2012)
This reference list is based on information provided by the publisher or from digital mathematics libraries. Its items are heuristically matched to zbMATH identifiers and may contain data conversion errors. In some cases that data have been complemented/enhanced by data from zbMATH Open. This attempts to reflect the references listed in the original paper as accurately as possible without claiming completeness or a perfect matching.