Analyzing stochastic computer models: a review with opportunities. (English) Zbl 07474198

Summary: In modern science, computer models are often used to understand complex phenomena and a thriving statistical community has grown around analyzing them. This review aims to bring a spotlight to the growing prevalence of stochastic computer models – providing a catalogue of statistical methods for practitioners, an introductory view for statisticians (whether familiar with deterministic computer models or not), and an emphasis on open questions of relevance to practitioners and statisticians. Gaussian process surrogate models take center stage in this review, and these, along with several extensions needed for stochastic settings, are explained. The basic issues of designing a stochastic computer experiment and calibrating a stochastic computer model are prominent in the discussion. Instructive examples, with data and code, are used to describe the implementation of, and results from, various methods.


62-XX Statistics
Full Text: DOI arXiv


[1] Abt, M. and Welch, W. J. (1998). Fisher information and maximum-likelihood estimation of covariance parameters in Gaussian stochastic processes. Canad. J. Statist. 26 127-137. · Zbl 0899.62124
[2] Alkema, L., Raftery, A. E. and Clark, S. J. (2007). Probabilistic projections of HIV prevalence using Bayesian melding. Ann. Appl. Stat. 1 229-248. · Zbl 1129.62098
[3] Andrianakis, I., Vernon, I. R., McCreesh, N., McKinley, T. J., Oakley, J. E., Nsubuga, R. N., Goldstein, M. and White, R. G. (2015). Bayesian history matching of complex infectious disease models using emulation: A tutorial and a case study on HIV in Uganda. PLoS Comput. Biol. 11 e1003968. · Zbl 1387.62111
[4] Andrianakis, I., McCreesh, N., Vernon, I., McKinley, T. J., Oakley, J. E., Nsubuga, R. N., Goldstein, M. and White, R. G. (2017). Efficient history matching of a high dimensional individual-based HIV transmission model. SIAM/ASA J. Uncertain. Quantificat. 5 694-719. · Zbl 1387.62111
[5] Ankenman, B., Nelson, B. L. and Staum, J. (2010). Stochastic kriging for simulation metamodeling. Oper. Res. 58 371-382. · Zbl 1342.62134
[6] Ba, S. (2019). SLHD: Maximin-distance (sliced) Latin hypercube designs. R package version 1.1.2.
[7] Ba, S. and Joseph, V. R. (2012). Composite Gaussian process models for emulating expensive functions. Ann. Appl. Stat. 6 1838-1860. · Zbl 1257.62089
[8] Baker, E., Challenor, P. and Eames, M. (2020). Predicting the output from a stochastic computer model when a deterministic approximation is available. J. Comput. Graph. Statist. 29 786-797. · Zbl 07500357
[9] Baker, E., Barbillon, P., Fadikar, A., Gramacy, R. B., Herbei, R., Higdon, D., Huang, J., Johnson, L. R., Ma, P., Mondal, A., Pires, B., Sacks, J. and Sokolov, V. (2022). Supplement to “Analyzing stochastic computer models: A review with opportunities.” https://doi.org/10.1214/21-STS822SUPPA, https://doi.org/10.1214/21-STS822SUPPB · Zbl 07474198
[10] Barton, R., Nakayama, M. K. and Schruben, L. (2017). History of improving statistical efficiency. In 2017 Winter Simulation Conference (WSC) 158-180. IEEE, New York.
[11] Bastos, L. S. and O’Hagan, A. (2009). Diagnostics for Gaussian process emulators. Technometrics 51 425-438.
[12] Bayarri, M. J., Berger, J. O., Cafeo, J., Garcia-Donato, G., Liu, F., Palomo, J., Parthasarathy, R. J., Paulo, R., Sacks, J. et al. (2007a). Computer model validation with functional output. Ann. Statist. 35 1874-1906. · Zbl 1144.62368
[13] Bayarri, M. J., Berger, J. O., Paulo, R., Sacks, J., Cafeo, J. A., Cavendish, J., Lin, C.-H. and Tu, J. (2007b). A framework for validation of computer models. Technometrics 49 138-154.
[14] Bayarri, M. J., Berger, J. O., Kennedy, M. C., Kottas, A., Paulo, R., Sacks, J., Cafeo, J. A., Lin, C.-H. and Tu, J. (2009). Predicting vehicle crashworthiness: Validation of computer models for functional and hierarchical data. J. Amer. Statist. Assoc. 104 929-943. · Zbl 1388.62382
[15] Begon, M. et al. (1979). Investigating Animal Abundance: Capture-Recapture for Biologists. Edward Arnold Publishers Ltd., London.
[16] Bernardo, M. C., Buck, R., Liu, L., Nazaret, W. A., Sacks, J. and Welch, W. J. (1992). Integrated circuit design optimization using a sequential strategy. IEEE Trans. Comput.-Aided Des. Integr. Circuits Syst. 11 361-372.
[17] Binois, M. and Gramacy, R. B. (2018). hetGP: Heteroskedastic Gaussian process modeling and design under replication. R package version 1.1.1.
[18] Binois, M., Gramacy, R. B. and Ludkovski, M. (2018). Practical heteroscedastic Gaussian process modeling for large simulation experiments. J. Comput. Graph. Statist. 27 808-821. · Zbl 07498993
[19] Binois, M., Huang, J., Gramacy, R. B. and Ludkovski, M. (2019). Replication or exploration? Sequential design for stochastic simulation experiments. Technometrics 61 7-23.
[20] Bisset, K. R., Chen, J., Feng, X., Kumar, V. and Marathe, M. V. (2009). Epifast: A fast algorithm for large scale realistic epidemic simulations on distributed memory systems. In Proceedings of the 23rd International Conference on Supercomputing 430-439. ACM.
[21] Boukouvalas, A. and Cornford, D. (2009). Learning heteroscedastic Gaussian processes for complex datasets. Technical report, Aston University, Neural Computing Research Group.
[22] Boukouvalas, A., Cornford, D. and Stehlík, M. (2014). Optimal design for correlated processes with input-dependent noise. Comput. Statist. Data Anal. 71 1088-1102. · Zbl 1471.62031
[23] Boukouvalas, A., Sykes, P., Cornford, D. and Maruri-Aguilar, H. (2014). Bayesian precalibration of a large stochastic microsimulation model. IEEE Trans. Intell. Transp. Syst. 15 1337-1347.
[24] Breiman, L., Friedman, J. H., Olshen, R. A. and Stone, C. J. (1984). Classification and Regression Trees. Wadsworth Statistics/Probability Series. Wadsworth Advanced Books and Software, Belmont, CA. · Zbl 0541.62042
[25] Broderick, T. and Gramacy, R. B. (2011). Classification and categorical inputs with treed Gaussian process models. J. Classification 28 244-270. · Zbl 1226.62057
[26] Brynjarsdóttir, J. and O’Hagan, A. (2014). Learning about physical parameters: The importance of model discrepancy. Inverse Probl. 30 114007, 24. · Zbl 1307.60042
[27] Chalabi, Y., Dutang, C., Savicky, P. and Wuertz, D. (2019). randtoolbox: Toolbox for pseudo and quasi random number generation and random generator tests. R package version 1.30.0.
[28] Chen, X., Wang, K. and Yang, F. (2013). Stochastic kriging with qualitative factors. In 2013 Winter Simulations Conference (WSC) 790-801. IEEE, New York.
[29] Chen, H., Loeppky, J. L., Sacks, J. and Welch, W. J. (2016). Analysis methods for computer experiments: How to assess and what counts? Statist. Sci. 31 40-60. · Zbl 1442.62179
[30] Chipman, H. A., George, E. I. and McCulloch, R. E. (2010). BART: Bayesian additive regression trees. Ann. Appl. Stat. 4 266-298. · Zbl 1189.62066
[31] Chipman, H. and McCulloch, R. (2016). BayesTree: Bayesian additive regression trees. R package version 0.3-1.4.
[32] Chung, M., Binois, M., Gramacy, R. B., Bardsley, J. M., Moquin, D. J., Smith, A. P. and Smith, A. M. (2019). Parameter and uncertainty estimation for dynamical systems using surrogate stochastic processes. SIAM J. Sci. Comput. 41 A2212-A2238. · Zbl 1420.60047
[33] Conti, S. and O’Hagan, A. (2010). Bayesian emulation of complex multi-output and dynamic computer models. J. Statist. Plann. Inference 140 640-651. · Zbl 1177.62033
[34] Craig, P. S., Goldstein, M., Seheult, A. H. and Smith, J. A. (1997). Pressure matching for hydrocarbon reservoirs: A case study in the use of Bayes linear strategies for large computer experiments. In Case Studies in Bayesian Statistics 37-93. Springer, Berlin. · Zbl 0895.62105
[35] Cressie, N. A. C. (1993). Statistics for Spatial Data. Wiley Series in Probability and Mathematical Statistics: Applied Probability and Statistics. Wiley, New York. Revised reprint of the 1991 edition, A Wiley-Interscience Publication. · Zbl 1347.62005
[36] Currin, C., Mitchell, T., Morris, M. and Ylvisaker, D. (1991). Bayesian prediction of deterministic functions, with applications to the design and analysis of computer experiments. J. Amer. Statist. Assoc. 86 953-963.
[37] Duan, W., Ankenman, B. E., Sanchez, S. M. and Sanchez, P. J. (2017). Sliced full factorial-based Latin hypercube designs as a framework for a batch sequential design algorithm. Technometrics 59 11-22.
[38] Erickson, C. B., Ankenman, B. E., Plumlee, M. and Sanchez, S. M. (2018). Gradient based criteria for sequential design. In 2018 Winter Simulation Conference (WSC) 467-478. IEEE, New York.
[39] Fadikar, A., Higdon, D., Chen, J., Lewis, B., Venkatramanan, S. and Marathe, M. (2018). Calibrating a stochastic, agent-based model using quantile-based emulation. SIAM/ASA J. Uncertain. Quantificat. 6 1685-1706. · Zbl 1409.62111
[40] Farah, M., Birrell, P., Conti, S. and De Angelis, D. (2014). Bayesian emulation and calibration of a dynamic epidemic model for A/H1N1 influenza. J. Amer. Statist. Assoc. 109 1398-1411.
[41] Feynman, R. P. (1948). Space-time approach to non-relativistic quantum mechanics. Rev. Modern Phys. 20 367-387. · Zbl 1371.81126
[42] Frazier, P. I. (2018). A tutorial on Bayesian optimization. arXiv preprint, arXiv:1807.02811.
[43] Frenklach, M., Packard, A., Garcia-Donato, G., Paulo, R. and Sacks, J. (2016). Comparison of statistical and deterministic frameworks of uncertainty quantification. SIAM/ASA J. Uncertain. Quantificat. 4 875-901. · Zbl 1356.62037
[44] Fricker, T. E., Oakley, J. E. and Urban, N. M. (2013). Multivariate Gaussian process emulators with nonseparable covariance structures. Technometrics 55 47-56.
[45] Fuentes, M. and Raftery, A. E. (2005). Model evaluation and spatial interpolation by Bayesian combination of observations with outputs from numerical models. Biometrics 61 36-45. · Zbl 1077.62124
[46] Gal, Y. and Ghahramani, Z. (2016). Dropout as a Bayesian approximation: Representing model uncertainty in deep learning. In Proceedings of the 33rd International Conference on International Conference on Machine Learning 1050-1059.
[47] Gao, F., Sacks, J. and Welch, W. J. (1996). Predicting urban ozone levels and trends with semiparametric modeling. J. Agric. Biol. Environ. Stat. 1 404-425.
[48] Gneiting, T. and Raftery, A. E. (2007). Strictly proper scoring rules, prediction, and estimation. J. Amer. Statist. Assoc. 102 359-378. · Zbl 1284.62093
[49] Goldberg, P. W., Williams, C. K. and Bishop, C. M. (1997). Regression with input-dependent noise: A Gaussian process treatment. In Proceedings of the 10th International Conference on Neural Information Processing Systems 493-499.
[50] Gramacy, R. (2007). tgp: An R package for Bayesian nonstationary, semiparametric nonlinear regression and design by treed Gaussian process models. J. Stat. Softw. 19 6.
[51] Gramacy, R. B. (2020). Surrogates: Gaussian Process Modeling, Design and Optimization for the Applied Sciences. Chapman Hall/CRC, Boca Raton, FL.
[52] Gramacy, R. B. and Lee, H. K. H. (2008). Bayesian treed Gaussian process models with an application to computer modeling. J. Amer. Statist. Assoc. 103 1119-1130. · Zbl 1205.62218
[53] Gramacy, R. B. and Lee, H. K. H. (2009). Adaptive design and analysis of supercomputer experiments. Technometrics 51 130-145.
[54] Gramacy, R. B. and Lee, H. K. H. (2011). Optimization under unknown constraints. In Bayesian Statistics 9 (J. Bernardo, S. Bayarri, J. O. Berger, A. P. Dawid, D. Heckerman, A. F. M. Smith and M. West, eds.) 229-256. Oxford Univ. Press, Oxford.
[55] Gramacy, R. and Taddy, M. (2010). Categorical inputs, sensitivity analysis, optimization and importance tempering with tgp version 2, an R package for treed Gaussian process models. J. Stat. Softw. 33 1-48.
[56] Gramacy, R. B. and Taddy, M. A. (2016). tgp: Bayesian treed Gaussian process models. R package version 2.4-14.
[57] Graves, A. (2011). Practical variational inference for neural networks. In Proceedings of the 24th International Conference on Neural Information Processing Systems 2348-2356.
[58] Grimm, V., Berger, U., Bastiansen, F., Eliassen, S., Ginot, V., Giske, J., Goss-Custard, J., Grand, T., Heinz, S. K. et al. (2006). A standard protocol for describing individual-based and agent-based models. Ecol. Model. 198 115-126.
[59] Gu, M. and Wang, L. (2018). Scaled Gaussian stochastic process for computer model calibration and prediction. SIAM/ASA J. Uncertain. Quantificat. 6 1555-1583. · Zbl 1409.62185
[60] Hart, J. L., Alexanderian, A. and Gremaud, P. A. (2017). Efficient computation of Sobol’ indices for stochastic models. SIAM J. Sci. Comput. 39 A1514-A1530. · Zbl 1371.60094
[61] Harville, D. A. (1997). Matrix Algebra from a Statistician’s Perspective. Springer, New York. · Zbl 0881.15001
[62] Henderson, D. A., Boys, R. J., Krishnan, K. J., Lawless, C. and Wilkinson, D. J. (2009). Bayesian emulation and calibration of a stochastic computer model of mitochondrial DNA deletions in substantia nigra neurons. J. Amer. Statist. Assoc. 104 76-87. · Zbl 1388.92007
[63] Herbei, R. and Berliner, L. M. (2014). Estimating ocean circulation: An MCMC approach with approximated likelihoods via the Bernoulli factory. J. Amer. Statist. Assoc. 109 944-954.
[64] Higdon, D., Kennedy, M., Cavendish, J. C., Cafeo, J. A. and Ryne, R. D. (2004). Combining field data and computer simulations for calibration and prediction. SIAM J. Sci. Comput. 26 448-466. · Zbl 1072.62018
[65] Higdon, D., Gattiker, J., Williams, B. and Rightley, M. (2008). Computer model calibration using high-dimensional output. J. Amer. Statist. Assoc. 103 570-583. · Zbl 1469.62414
[66] Jalali, H., Van Nieuwenhuyse, I. and Picheny, V. (2017). Comparison of Kriging-based algorithms for simulation optimization with heterogeneous noise. European J. Oper. Res. 261 279-301. · Zbl 1403.90553
[67] Johnson, L. R. (2010). Implications of dispersal and life history strategies for the persistence of linyphiid spider populations. Ecol. Model. 221 1138-1147.
[68] Johnson, L. R. and Briggs, C. J. (2011). Parameter inference for an individual based model of chytridiomycosis in frogs. J. Theoret. Biol. 277 90-98. · Zbl 1405.92289
[69] Johnson, L. R., Gramacy, R. B., Cohen, J., Mordecai, E., Murdock, C., Rohr, J., Ryan, S. J., Stewart-Ibarra, A. M. and Weikel, D. (2018). Phenomenological forecasting of disease incidence using heteroskedastic Gaussian processes: A Dengue case study. Ann. Appl. Stat. 12 27-66. · Zbl 1393.62071
[70] Jolliffe, I. (2011). Principal Component Analysis. Springer, Berlin.
[71] Jones, D. R., Schonlau, M. and Welch, W. J. (1998). Efficient global optimization of expensive black-box functions. J. Global Optim. 13 455-492. Workshop on Global Optimization (Trier, 1997). · Zbl 0917.90270
[72] Kac, M. (1949). On distributions of certain Wiener functionals. Trans. Amer. Math. Soc. 65 1-13. · Zbl 0032.03501
[73] Kennedy, J. C., Henderson, D. A. and Wilson, K. J. (2020). Multilevel emulation for stochastic computer models with an application to large offshore windfarms. arXiv preprint, arXiv:2003.08921.
[74] Kennedy, M. C. and O’Hagan, A. (2000). Predicting the output from a complex computer code when fast approximations are available. Biometrika 87 1-13. · Zbl 0974.62024
[75] Kennedy, M. C. and O’Hagan, A. (2001). Bayesian calibration of computer models. J. R. Stat. Soc. Ser. B. Stat. Methodol. 63 425-464. · Zbl 1007.62021
[76] Kersaudy, P., Sudret, B., Varsier, N., Picon, O. and Wiart, J. (2015). A new surrogate modeling technique combining kriging and polynomial chaos expansions—application to uncertainty analysis in computational dosimetry. J. Comput. Phys. 286 103-117. · Zbl 1351.94003
[77] Kersting, K., Plagemann, C., Pfaff, P. and Burgard, W. (2007). Most likely heteroscedastic Gaussian process regression. In Proceedings of the 24th International Conference on Machine Learning 393-400.
[78] Kim, H.-M., Mallick, B. K. and Holmes, C. C. (2005). Analyzing nonstationary spatial data using piecewise Gaussian processes. J. Amer. Statist. Assoc. 100 653-668. · Zbl 1117.62368
[79] Kleijnen, J. P. C. (2009). Kriging metamodeling in simulation: A review. European J. Oper. Res. 192 707-716. · Zbl 1157.90544
[80] Kleijnen, J. P. C. (2017). Regression and kriging metamodels with their experimental designs in simulation: A review. European J. Oper. Res. 256 1-16. · Zbl 1394.90004
[81] Koenker, R. and Bassett, G. Jr. (1978). Regression quantiles. Econometrica 46 33-50. · Zbl 0373.62038
[82] Konomi, B. A., Karagiannis, G., Lai, K. and Lin, G. (2017). Bayesian treed calibration: An application to carbon capture with AX sorbent. J. Amer. Statist. Assoc. 112 37-53.
[83] Kushner, H. J. (1964). A new method of locating the maximum point of an arbitrary multipeak curve in the presence of noise. J. Basic Eng. 86 97-106.
[84] Lakshminarayanan, B., Pritzel, A. and Blundell, C. (2017). Simple and scalable predictive uncertainty estimation using deep ensembles. In Advances in Neural Information Processing Systems 6402-6413.
[85] Lee, A. (2015). pyDOE: The experimental design package for python. Python package version 0.3.8.
[86] Liu, F., Bayarri, M. J. and Berger, J. O. (2009). Modularization in Bayesian analysis, with emphasis on analysis of computer models. Bayesian Anal. 4 119-150. · Zbl 1330.65033
[87] Loeppky, J. L., Moore, L. M. and Williams, B. J. (2010). Batch sequential designs for computer experiments. J. Statist. Plann. Inference 140 1452-1464. · Zbl 1185.62142
[88] Loeppky, J. L., Sacks, J. and Welch, W. J. (2009). Choosing the sample size of a computer experiment: A practical guide. Technometrics 51 366-376.
[89] Lyu, X., Binois, M. and Ludkovski, M. (2018). Evaluating Gaussian process metamodels and sequential designs for noisy level set estimation. arXiv preprint, arXiv:1807.06712. · Zbl 1475.62048
[90] Ma, P., Mondal, A., Konomi, B., Hobbs, J., Song, J. and Kang, E. (2021). Computer model emulation with high-dimensional functional output in large-scale observing system uncertainty experiments. Technometrics.
[91] Marrel, A., Iooss, B., Laurent, B. and Roustant, O. (2009). Calculations of Sobol indices for the Gaussian process metamodel. Reliab. Eng. Syst. Saf. 94 742-751.
[92] Marrel, A., Iooss, B., Da Veiga, S. and Ribatet, M. (2012). Global sensitivity analysis of stochastic computer models with joint metamodels. Stat. Comput. 22 833-847. · Zbl 1252.62120
[93] McCulloch, R., Sparapani, R., Gramacy, R., Spanbauer, C. and Pratola, M. (2019). BART: Bayesian additive regression trees. R package version 2.7.
[94] McKay, M. D., Beckman, R. J. and Conover, W. J. (1979). A comparison of three methods for selecting values of input variables in the analysis of output from a computer code. Technometrics 21 239-245. · Zbl 0415.62011
[95] McKeague, I. W., Nicholls, G., Speer, K. and Herbei, R. (2005). Statistical inversion of South Atlantic circulation in an abyssal neutral density layer. J. Mar. Res. 63 683-704.
[96] McKinley, T. J., Vernon, I., Andrianakis, I., McCreesh, N., Oakley, J. E., Nsubuga, R. N., Goldstein, M. and White, R. G. (2018). Approximate Bayesian computation and simulation-based inference for complex stochastic epidemic models. Statist. Sci. 33 4-18. · Zbl 1407.62406
[97] Mockus, J., Tiesis, V. and Zilinskas, A. (1978). The application of Bayesian methods for seeking the extremum. In Towards Global Optimization 2 117-129. · Zbl 0394.90090
[98] Molina, G., Bayarri, M. J. and Berger, J. O. (2005). Statistical inverse analysis for a network microsimulator. Technometrics 47 388-398.
[99] Mortveit, H., Adiga, A., Agashe, A., Alam, M., Alexander, K., Arifuzzaman, S., Barrett, C., Beckman, R., Bisset, K. et al. (2015). Synthetic populations and interaction networks for Guinea, Liberia and Sierra Leone. Technical report, NDSSL.
[100] Moutoussamy, V., Nanty, S. and Pauwels, B. (2015). Emulators for stochastic simulation codes. In CEMRACS 2013—Modelling and Simulation of Complex Systems: Stochastic and Deterministic Approaches. ESAIM Proc. Surveys 48 116-155. EDP Sci., Les Ulis. · Zbl 1338.62179
[101] Neal, R. M. (1996). Bayesian Learning for Neural Networks. Lecture Notes in Statistics 118. Springer, New York. · Zbl 0888.62021
[102] Nguyen, H., Cressie, N. and Braverman, A. (2017). Multivariate spatial data fusion for very large remote sensing datasets. Remote Sens. 9 142.
[103] O’Hagan, A. (2006). Bayesian analysis of computer code outputs: A tutorial. Reliab. Eng. Syst. Saf. 91 1290-1300.
[104] Oakley, J. E. and O’Hagan, A. (2004). Probabilistic sensitivity analysis of complex models: A Bayesian approach. J. R. Stat. Soc. Ser. B. Stat. Methodol. 66 751-769. · Zbl 1046.62027
[105] Oakley, J. E. and Youngman, B. D. (2017). Calibration of stochastic computer simulators using likelihood emulation. Technometrics 59 80-92.
[106] Opitz, T., Huser, R., Bakka, H. and Rue, H. (2018). INLA goes extreme: Bayesian tail regression for the estimation of high spatio-temporal quantiles. Extremes 21 441-462. · Zbl 1407.62167
[107] Papamakarios, G., Nalisnick, E., Rezende, D. J., Mohamed, S. and Lakshminarayanan, B. (2019). Normalizing flows for probabilistic modeling and inference. arXiv preprint, arXiv:1912.02762. · Zbl 07370574
[108] Park, C. and Apley, D. (2018). Patchwork Kriging for large-scale Gaussian process regression. J. Mach. Learn. Res. 19 Paper No. 7, 43. · Zbl 1444.62088
[109] Paulo, R., García-Donato, G. and Palomo, J. (2012). Calibration of computer models with multivariate output. Comput. Statist. Data Anal. 56 3959-3974. · Zbl 1255.62187
[110] Pedregosa, F., Varoquaux, G., Gramfort, A. et al. (2011). Scikit-learn: Machine learning in Python. J. Mach. Learn. Res. 12 2825-2830. · Zbl 1280.68189
[111] Peleg, N., Fatichi, S., Paschalis, A., Molnar, P. and Burlando, P. (2017). An advanced stochastic weather generator for simulating 2-d high-resolution climate variables. J. Adv. Model. Earth Syst. 9 1595-1627.
[112] Picheny, V. and Ginsbourger, D. (2014). Noisy kriging-based optimization methods: A unified implementation within the DiceOptim package. Comput. Statist. Data Anal. 71 1035-1053. · Zbl 1471.62161
[113] Picheny, V., Ginsbourger, D. and Roustant, O. (2016). DiceOptim: Kriging-based optimization for computer experiments. R package version 2.0.
[114] Picheny, V., Wagner, T. and Ginsbourger, D. (2013). A benchmark of kriging-based infill criteria for noisy optimization. Struct. Multidiscip. Optim. 48 607-626.
[115] Plumlee, M. (2017). Bayesian calibration of inexact computer models. J. Amer. Statist. Assoc. 112 1274-1285.
[116] Plumlee, M. and Tuo, R. (2014). Building accurate emulators for stochastic simulations via quantile kriging. Technometrics 56 466-473.
[117] Poole, D. and Raftery, A. E. (2000). Inference for deterministic simulation models: The Bayesian melding approach. J. Amer. Statist. Assoc. 95 1244-1255. · Zbl 1072.62544
[118] Pratola, M. T., Chipman, H. A., George, E. I. and McCulloch, R. E. (2020). Heteroscedastic BART via multiplicative regression trees. J. Comput. Graph. Statist. 29 405-417. · Zbl 07499266
[119] Pronzato, L. and Müller, W. G. (2012). Design of computer experiments: Space filling and beyond. Stat. Comput. 22 681-701. · Zbl 1252.62080
[120] Pukelsheim, F. (1994). The three sigma rule. Amer. Statist. 48 88-91.
[121] Qian, P. Z. G., Wu, H. and Wu, C. F. J. (2008). Gaussian process models for computer experiments with qualitative and quantitative factors. Technometrics 50 383-396.
[122] Radtke, P. J., Burk, T. E. and Bolstad, P. V. (2002). Bayesian melding of a forest ecosystem model with correlated inputs. For. Sci. 48 701-711.
[123] Raftery, A. E., Givens, G. H. and Zeh, J. E. (1995). Inference from a deterministic population dynamics model for bowhead whales. J. Amer. Statist. Assoc. 90 402-416. · Zbl 0925.62473
[124] Ramsey, D. S. and Efford, M. G. (2010). Management of bovine tuberculosis in brushtail possums in New Zealand: Predictions from a spatially explicit, individual-based model. J. Appl. Ecol. 47 911-919.
[125] Rannou, V., Brouaye, F., Hélier, M. and Tabbara, W. (2002). Kriging the quantile: Application to a simple transmission line model. Inverse Probl. 18 37. · Zbl 1009.62002
[126] Rasmussen, C. E. and Williams, C. K. I. (2006). Gaussian Processes for Machine Learning. Adaptive Computation and Machine Learning. MIT Press, Cambridge, MA.
[127] Reynolds, C. W. (1987). Flocks, herds and schools: A distributed behavioral model. In Proceedings of the 14th Annual Conference on Computer Graphics and Interactive Techniques 25-34.
[128] Richardson, C. W. (1981). Stochastic simulation of daily precipitation, temperature, and solar radiation. Water Resour. Res. 17 182-190.
[129] Roustant, O., Ginsbourger, D. and Deville, Y. (2018). DiceKriging: Kriging methods for computer experiments. R package version 1.5.6.
[130] Rullière, D., Durrande, N., Bachoc, F. and Chevalier, C. (2018). Nested Kriging predictions for datasets with a large number of observations. Stat. Comput. 28 849-867. · Zbl 1384.62246
[131] Rushdi, A. A., Swiler, L. P., Phipps, E. T., D’Elia, M. and Ebeida, M. S. (2017). VPS: Voronoi piecewise surrogate models for high-dimensional data fitting. Int. J. Uncertain. Quantif. 7 1-21. · Zbl 1498.62157
[132] Rutter, C. M., Ozik, J., DeYoreo, M. and Collier, N. (2019). Microsimulation model calibration using incremental mixture approximate Bayesian computation. Ann. Appl. Stat. 13 2189-2212. · Zbl 1435.62398
[133] Sacks, J., Welch, W. J., Mitchell, T. J. and Wynn, H. P. (1989). Design and analysis of computer experiments. Statist. Sci. 4 409-435. With comments and a rejoinder by the authors. · Zbl 0955.62619
[134] Salter, J. M., Williamson, D. B., Scinocca, J. and Kharin, V. (2019). Uncertainty quantification for computer models with spatial output using calibration-optimal bases. J. Amer. Statist. Assoc. 114 1800-1814. · Zbl 1428.62117
[135] Santner, T. J., Williams, B. J. and Notz, W. I. (2018). The Design and Analysis of Computer Experiments. Springer Series in Statistics. Springer, New York. Second edition of [MR2160708]. · Zbl 1405.62110
[136] Schonlau, M. and Welch, W. J. (2006). Screening the input variables to a computer model via analysis of variance and visualization. In Screening 308-327. Springer.
[137] Schultz, L. and Sokolov, V. (2018). Practical Bayesian optimization for transportation simulators. arXiv preprint, arXiv:1810.03688.
[138] Ševčíková, H., Raftery, A. E. and Waddell, P. A. (2007). Assessing uncertainty in urban simulations using Bayesian melding. Transp. Res., Part B, Methodol. 41 652-669.
[139] Shah, A., Wilson, A. and Ghahramani, Z. (2014). Student-t processes as alternatives to Gaussian processes. In Proceedings of the 17th International Conference on Artificial Intelligence and Statistics. PMLR 33 877-885.
[140] Smieszek, T., Balmer, M., Hattendorf, J., Axhausen, K. W., Zinsstag, J. and Scholz, R. W. (2011). Reconstructing the 2003/2004 H3N2 influenza epidemic in Switzerland with a spatially explicit, individual-based model. BMC Infect. Dis. 11 115.
[141] Sobol, I. M. (1967). On the distribution of points in a cube and the approximate evaluation of integrals. Comput. Math. Math. Phys. 7 86-112. · Zbl 0185.41103
[142] Sobol, I. M. (1993). Sensitivity estimates for nonlinear mathematical models. Math. Model. Comput. Exp. 1 407-414. · Zbl 1039.65505
[143] Spiller, E. T., Bayarri, M. J., Berger, J. O., Calder, E. S., Patra, A. K., Pitman, E. B. and Wolpert, R. L. (2014). Automating emulator construction for geophysical hazard maps. SIAM/ASA J. Uncertain. Quantificat. 2 126-152. · Zbl 1308.62182
[144] Srinivas, N., Krause, A., Kakade, S. M. and Seeger, M. W. (2012). Information-theoretic regret bounds for Gaussian process optimization in the bandit setting. IEEE Trans. Inf. Theory 58 3250-3265. · Zbl 1365.94131
[145] Stein, M. L. (2012). Interpolation of Spatial Data: Some Theory for Kriging. Springer, New York.
[146] Sullivan, T. J. (2015). Introduction to Uncertainty Quantification. Texts in Applied Mathematics 63. Springer, Cham. · Zbl 1336.60002
[147] Sun, F., Gramacy, R. B., Haaland, B., Lu, S. and Hwang, Y. (2019). Synthesizing simulation and field data of solar irradiance. Stat. Anal. Data Min. 12 311-324. · Zbl 07260639
[148] Sung, C.-L., Barber, B. D. and Walker, B. J. (2019). Calibration of computer models with heteroscedastic errors and application to plant relative growth rates. arXiv preprint, arXiv:1910.11518.
[149] Tuo, R. and Jeff Wu, C. F. (2016). A theoretical framework for calibration in computer models: Parametrization, estimation and convergence properties. SIAM/ASA J. Uncertain. Quantificat. 4 767-795. · Zbl 1383.62354
[150] Tuo, R. and Wu, C. F. J. (2015). Efficient calibration for imperfect computer models. Ann. Statist. 43 2331-2352. · Zbl 1326.62228
[151] Vazquez, E., Villemonteix, J., Sidorkiewicz, M. and Walter, E. (2008). Global optimization based on noisy evaluations: An empirical study of two statistical approaches. J. Phys., Conf. Ser. 135. 012100. · Zbl 1169.90460
[152] Vernon, I., Goldstein, M. and Bower, R. G. (2010). Galaxy formation: A Bayesian uncertainty analysis. Bayesian Anal. 5 619-669. · Zbl 1330.85005
[153] Wang, W. and Haaland, B. (2019). Controlling sources of inaccuracy in stochastic kriging. Technometrics 61 309-321.
[154] Wang, S. and Ng, S. H. (2020). Enhancing response predictions with a joint Gaussian process model for stochastic simulation models. ACM Trans. Model. Comput. Simul. 30 Art. 4, 25.
[155] Wang, Z., Shi, J. Q. and Lee, Y. (2017). Extended \(t\)-process regression models. J. Statist. Plann. Inference 189 38-60. · Zbl 1376.62026
[156] Welling, M. and Teh, Y. W. (2011). Bayesian learning via stochastic gradient Langevin dynamics. In Proceedings of the 28th International Conference on Machine Learning (ICML-11) 681-688.
[157] Wilensky, U. (1999). NetLogo. Available at http://ccl.northwestern.edu/netlogo/. Center for Connected Learning and Computer-Based Modeling, Northwestern Univ., Evanston, IL.
[158] Wilkinson, R. D. (2013). Approximate Bayesian computation (ABC) gives exact results under the assumption of model error. Stat. Appl. Genet. Mol. Biol. 12 129-141.
[159] Xie, G. and Chen, X. (2017). A heteroscedastic t-process simulation metamodeling approach and its application in inventory control and optimization. In Simulation Conference (WSC), 2017 Winter 3242-3253. IEEE, New York.
[160] Zhang, B., Cole, D. A. and Gramacy, R. B. (2021). Distance-Distributed Design for Gaussian Process Surrogates. Technometrics 63 40-52.
[161] Zhang, J., Cressie, N. and Craigmile, P. F. (2008). Loss function approaches to predict a spatial quantile and its exceedance region. Technometrics 50 216-227.
[162] Zhang, Q. and Xie, W. (2017). Asymmetric kriging emulator for stochastic simulation. In Proceedings of the 2017 Winter Simulation Conference 137. IEEE Press, New York.
[163] Zhang, Y., Tao, S., Chen, W. and Apley, D. W. (2020). A latent variable approach to Gaussian process modeling with qualitative and quantitative factors. Technometrics 62 291-302.
[164] Zhou, J., Chang, H. H. and Fuentes, M. (2012). Estimating the health impact of climate change with calibrated climate model output. J. Agric. Biol. Environ. Stat. 17 377-394. · Zbl 1302.62288
[165] Zhou, Q., Qian, P. Z. G. and Zhou, S. (2011). A simple approach to emulation for computer models with qualitative and quantitative factors. Technometrics 53 266-273
This reference list is based on information provided by the publisher or from digital mathematics libraries. Its items are heuristically matched to zbMATH identifiers and may contain data conversion errors. In some cases that data have been complemented/enhanced by data from zbMATH Open. This attempts to reflect the references listed in the original paper as accurately as possible without claiming completeness or a perfect matching.