×

zbMATH — the first resource for mathematics

Posterior inference for sparse hierarchical non-stationary models. (English) Zbl 07212308
Summary: Gaussian processes are valuable tools for non-parametric modelling, where typically an assumption of stationarity is employed. While removing this assumption can improve prediction, fitting such models is challenging. Hierarchical models are constructed based on Gaussian Markov random fields with stochastic spatially varying parameters. Importantly, this allows for non-stationarity while also addressing the computational burden through a sparse banded representation of the precision matrix. In this setting, efficient Markov chain Monte Carlo (MCMC) sampling is challenging due to the strong coupling a posteriori of the parameters and hyperparameters. Three adaptive MCMC schemes are developed and compared making use of banded matrix operations for faster inference. Furthermore, a novel extension to higher dimensional input spaces is proposed through an additive structure that retains the flexibility and scalability of the model, while also inheriting interpretability from the additive approach. A thorough assessment of the efficiency and accuracy of the methods in nonstationary settings is presented for both simulated experiments and a computer emulation problem.

MSC:
62 Statistics
PDF BibTeX XML Cite
Full Text: DOI
References:
[1] Anderes, E. B.; Stein, M. L., Estimating deformations of isotropic Gaussian random fields on the plane, Ann. Statist., 36, 2, 719-741 (2008) · Zbl 1133.62077
[2] Berrocal, V. J.; Raftery, A. E.; Gneiting, T.; Steed, R. C., Probabilistic weather forecasting for winter road maintenance, J. Amer. Statist. Assoc., 105, 490, 522-537 (2010) · Zbl 1392.62330
[3] Blomqvist, K.; Kaski, S.; Heinonen, M., Deep convolutional Gaussian processes (2018), arXiv:1810.03052, Unpublished results
[4] Buja, A.; Hastie, T.; Tibshirani, R., Linear smoothers and additive models, Ann. Statist., 17, 2, 453-510 (1989) · Zbl 0689.62029
[5] Chen, V.; Dunlop, M. M.; Papaspiliopoulos, O.; Stuart, A. M., Dimension-robust MCMC in Bayesian inverse problems (2019), arXiv:1803.03344, Unpublished results
[6] Cheng, L.; Ramchandran, S.; Vatanen, T.; Lietzén, N.; Lahesmaa, R.; Vehtari, A.; Lähdesmäki, H., An additive Gaussian process regression model for interpretable non-parametric analysis of longitudinal data, Nature Commun., 10, 1, 1798 (2019)
[7] Damianou, A.; Lawrence, N., Deep Gaussian processes, (Carvalho, C. M.; Ravikumar, P., Proceedings of the Sixteenth International Conference on Artificial Intelligence and Statistics. Proceedings of the Sixteenth International Conference on Artificial Intelligence and Statistics, Proceedings of Machine Learning Research, vol. 31 (2013), PMLR), 207-215
[8] Donoho, D. L.; Johnstone, I. M., Adapting to unknown smoothness via wavelet shrinkage, J. Amer. Statist. Assoc., 90, 432, 1200-1224 (1995) · Zbl 0869.62024
[9] Dunlop, M. M.; Girolami, M.; Stuart, A. M.; Teckentrup, A. L., How deep are deep Gaussian processes?, J. Mach. Learn. Res., 19, 1-46 (2018) · Zbl 06982345
[10] Durrande, N.; Adam, V.; Bordeaux, L.; Eleftheriadis, S.; Hensman, J., Banded matrix operators for Gaussian Markov models in the automatic differentiation era, (Chaudhuri, K.; Sugiyama, M., Proceedings of the 22nd International Conference on Artificial Intelligence and Statistics. Proceedings of the 22nd International Conference on Artificial Intelligence and Statistics, Proceedings of Machine Learning Research, vol. 89 (2019), PMLR), 2780-2789
[11] Duvenaud, D.; Nickisch, H.; Rasmussen, C. E., Additive Gaussian processes, (Proceedings of the 24th International Conference on Neural Information Processing Systems. Proceedings of the 24th International Conference on Neural Information Processing Systems, NIPS’11 (2011), Curran Associates Inc.), 226-234
[12] Duvenaud, D.; Rippel, O.; Adams, R.; Ghahramani, Z., Avoiding pathologies in very deep networks, (Proceedings of the Seventeenth International Conference on Artificial Intelligence and Statistics. Proceedings of the Seventeenth International Conference on Artificial Intelligence and Statistics, Proceedings of Machine Learning Research, vol. 33 (2014), PMLR), 202-210
[13] Fagan, F.; Bhandari, J.; Cunningham, J. P., Elliptical slice sampling with expectation propagation, (Proceedings of the Thirty-Second Conference on Uncertainty in Artificial Intelligence. Proceedings of the Thirty-Second Conference on Uncertainty in Artificial Intelligence, UAI’16 (2016), AUAI Press), 172-181
[14] Filippone, M.; Zhong, M.; Girolami, M., A comparative evaluation of stochastic-based inference methods for Gaussian process models, Mach. Learn., 93, 1, 93-114 (2013) · Zbl 1294.62048
[15] Fouedjio, F.; Desassis, N.; Rivoirard, J., A generalized convolution model and estimation for non-stationary random functions, Spat. Stat., 16, 35-52 (2016)
[16] Friedman, J. H.; Stuetzle, W., Projection pursuit regression, J. Amer. Statist. Assoc., 76, 376, 817-823 (1981)
[17] Fuglstad, G. A.; Lindgren, F.; Simpson, D.; Rue, H., Exploring a new class of non-stationary spatial Gaussian random fields with varying local anisotropy, Statist. Sinica, 25, 1, 115-133 (2015) · Zbl 06497337
[18] Fuglstad, G. A.; Simpson, D.; Lindgren, F.; Rue, H., Does non-stationary spatial data always require non-stationary random fields?, Spat. Stat., 14, 505-531 (2015)
[19] Gilboa, E.; Saatçi, Y.; Cunningham, J. P., Scaling multidimensional Gaussian processes using projected additive approximations, (Proceedings of the 30th International Conference on Machine Learning (2013)), 454-461
[20] Gilboa, E.; Saatçi, Y.; Cunningham, J. P., Scaling multidimensional inference for structured Gaussian processes, IEEE Trans. Pattern Anal. Mach. Intell., 37, 2, 424-436 (2015)
[21] Gramacy, R. B., Tgp: an R package for Bayesian nonstationary, semiparametric nonlinear regression and design by treed Gaussian process models, J. Stat. Softw., 19, 9, 1-46 (2007)
[22] Gramacy, R. B., LaGP: Large-scale spatial modeling via local approximate Gaussian processes in R, J. Stat. Softw., 72, 1, 1-46 (2016)
[23] Gramacy, R. B.; Lee, H. K., Bayesian treed Gaussian process models with an application to computer modeling, J. Amer. Statist. Assoc., 103, 483, 1119-1130 (2008) · Zbl 1205.62218
[24] Harville, D. A., Matrix Algebra from a Statistician’s Perspective, Vol. 1 (1997), Springer · Zbl 0881.15001
[25] Heaton, M. J.; Datta, A.; Finley, A. O.; Furrer, R.; Guinness, J.; Guhaniyogi, R.; Gerber, F.; Gramacy, R. B.; Hammerling, D.; Katzfuss, M.; Lindgren, F.; Nychka, D. W.; Sun, F.; Zammit-Mangion, A., A case study competition among methods for analyzing large spatial data, J. Agric. Biol. Environ. Stat., 24, 3, 398-425 (2018) · Zbl 1426.62345
[26] Hegde, P.; Heinonen, M.; Lähdesmäki, H.; Kaski, S., Deep learning with differential Gaussian process flows, (Chaudhuri, K.; Sugiyama, M., Proceedings of the 22nd International Conference on Artificial Intelligence and Statistics. Proceedings of the 22nd International Conference on Artificial Intelligence and Statistics, Proceedings of Machine Learning Research, vol. 89 (2019), PMLR), 1812-1821
[27] Heinonen, M.; Mannerström, H.; Rousu, J.; Kaski, S.; Lähdesmäki, H., Non-stationary Gaussian process regression with Hamiltonian Monte Carlo, (Proceedings of the 19th International Conference on Artificial Intelligence and Statistics. Proceedings of the 19th International Conference on Artificial Intelligence and Statistics, Proceedings of Machine Learning Research, vol. 51 (2016), PMLR), 732-740
[28] Kaipio, J.; Somersalo, E., Statistical and Computational Inverse Problems (2006), Springer Science & Business Media · Zbl 0927.35134
[29] Katzfuss, M., Bayesian nonstationary spatial modeling for very large datasets, Environmetrics, 24, 3, 189-200 (2013)
[30] Kim, H.-M.; Mallick, B. K.; Holmes, C., Analyzing nonstationary spatial data using piecewise Gaussian processes, J. Amer. Statist. Assoc., 100, 470, 653-668 (2005) · Zbl 1117.62368
[31] Kleiber, W.; Nychka, D., Nonstationary modeling for multivariate spatial processes, J. Multivariate Anal., 112, 76-91 (2012) · Zbl 1273.62235
[32] Lang, T.; Plagemann, C.; Burgard, W., Adaptive non-stationary kernel regression for terrain modeling, (Robotics: Science and Systems (RSS) (2007))
[33] Lindgren, F.; Rue, H.; Lindström, J., An explicit link between Gaussian fields and Gaussian Markov random fields: the stochastic partial differential equation approach, J. R. Stat. Soc. Ser. B Stat. Methodol., 73, 4, 423-498 (2011) · Zbl 1274.62360
[34] Matheron, G., The intrinsic random functions and their applications, Adv. Appl. Probab., 439-468 (1973) · Zbl 0324.60036
[35] Matthews, A. G.; van der Wilk, M.; Nickson, T.; Fujii, K.; Boukouvalas, A.; Leoón-Villagrá, P.; Ghahramani, Z.; Hensman, J., GPflow: A Gaussian process library using TensorFlow, J. Mach. Learn. Res., 18, 1, 1299-1304 (2017) · Zbl 1437.62127
[36] Montagna, S.; Tokdar, S. T., Computer emulation with nonstationary Gaussian processes, SIAM/ASA J. Uncertain. Quantif., 4, 1, 26-47 (2016) · Zbl 1387.62091
[37] Murray, I.; Adams, R. P., Slice sampling covariance hyperparameters of latent Gaussian models, (Lafferty, J.; Williams, C.; Shawe-Taylor, J.; Zemel, R.; Culotta, A., Advances in Neural Information Processing Systems (2010), Curran Associates, Inc.), 1732-1740
[38] Murray, I.; Adams, R. P.; MacKay, D. J., Elliptical slice sampling, (Proceedings of the 13th International Conference on Artificial Intelligence and Statistics. Proceedings of the 13th International Conference on Artificial Intelligence and Statistics, Proceedings of Machine Learning Research, vol. 9 (2010)), 541-548
[39] Neto, J. H.V.; Schmidt, A. M.; Guttorp, P., Accounting for spatially varying directional effects in spatial covariance structures, J. R. Stat. Soc. Ser. C. Appl. Stat., 63, 1, 103-122 (2014)
[40] Paciorek, C. J.; Schervish, M. J., Spatial modelling using a new class of nonstationary covariance functions, Environmetrics, 17, 5, 483-506 (2006)
[41] Pintore, A.; Holmes, C., Spatially Adaptive Non-Stationary Covariance Functions Via Spatially Adaptive SpectraTechnical report (2004), University of Oxford
[42] Raftery, A. E.; Lewis, S. M., [Practical Markov Chain Monte Carlo]: Comment: One long run with diagnostics: Implementation strategies for Markov Chain Monte Carlo, Statist. Sci., 7, 4, 493-497 (1992)
[43] Risser, M. D., Nonstationary spatial modeling, with emphasis on process convolution and covariate-driven approaches (2016), arXiv:1610.02447, Unpublished results
[44] Risser, M. D.; Calder, C. A., Local likelihood estimation for covariance functions with spatially-varying parameters: The convoSPAT package for R, J. Stat. Softw., 81, 1, 1-32 (2017)
[45] Roberts, G. O.; Rosenthal, J. S., Examples of adaptive MCMC, J. Comput. Graph. Statist., 18, 2, 349-367 (2009)
[46] Roininen, L.; Girolami, M.; Lasanen, S.; Markkanen, M., Hyperpriors for Matérn fields with applications in Bayesian inversion, Inverse Probl. Imaging, 13, 1, 1-29 (2019) · Zbl 07031512
[47] Rue, H.; Held, L., Gaussian Markov Random Fields: Theory and Applications (2005), Chapman and Hall/CRC · Zbl 1093.60003
[48] Sampson, P.; Damian, D.; Guttorp, P., Advances in modeling and inference for environmental processes with nonstationary spatial covariance, (GeoENV III — Geostatistics for Environmental Applications, Vol. 11 (2001), Springer), 17-32
[49] Seiler, M. C.; Seiler, F. A., Numerical recipes in C: the art of scientific computing, Risk Anal., 9, 3, 415-416 (1989)
[50] Stathopoulos, V.; Zamora-Gutierrez, V.; Jones, K.; Girolami, M., Bat call identification with Gaussian process multinomial probit regression and a dynamic time warping kernel, (Proceedings of the Seventeenth International Conference on Artificial Intelligence and Statistics. Proceedings of the Seventeenth International Conference on Artificial Intelligence and Statistics, Proceedings of Machine Learning Research, vol. 33 (2014), PMLR), 913-921
[51] Stein, M. L., Nonstationary Spatial Covariance FunctionsTechnical Report 21 (2005), Center for Integrating Statistical and Environmental Science, University of Chicago: Center for Integrating Statistical and Environmental Science, University of Chicago Chicago
[52] Titsias, M. K.; Papaspiliopoulos, O., Auxiliary gradient-based sampling algorithms, J. R. Stat. Soc. Ser. B Stat. Methodol., 80, 749-767 (2018) · Zbl 1398.62035
[53] Vannucci, M.; Corradi, F., Covariance structure of wavelet coefficients: Theory and models in a Bayesian perspective, J. R. Stat. Soc. Ser. B Stat. Methodol., 61, 4, 971-986 (1999) · Zbl 0940.62023
[54] Volodina, V.; Williamson, D. B., Diagnostic-driven nonstationary emulators using kernel mixtures, SIAM/ASA J. Uncertain. Quantif., 8, 1, 1-26 (2020) · Zbl 1436.62365
[55] Wood, S., Mgcv: Mixed GAM computation vehicle with automatic smoothness estimation (2019), R package version 1.8-31
[56] Yu, Y.; Meng, X.-L., To center or not to center: That is not the question -anancillarity-sufficiency interweaving strategy (ASIS) for boosting MCMC efficiency, J. Comput. Graph. Statist., 20, 3, 531-570 (2011)
[57] Yue, Y. R.; Simpson, D.; Lindgren, F.; Rue, H., Bayesian adaptive smoothing splines using stochastic differential equations, Bayesian Anal., 9, 2, 397-424 (2014) · Zbl 1327.62234
[58] Zhang, H., Inconsistent estimation and asymptotically equal interpolations in model-based geostatistics, J. Amer. Statist. Assoc., 99, 465, 250-261 (2004) · Zbl 1089.62538
This reference list is based on information provided by the publisher or from digital mathematics libraries. Its items are heuristically matched to zbMATH identifiers and may contain data conversion errors. It attempts to reflect the references listed in the original paper as accurately as possible without claiming the completeness or perfect precision of the matching.