Comparison of Gaussian process modeling software. (English) Zbl 1403.62006

Summary: Gaussian process fitting, or Kriging, is often used to create a model from a set of data. Many available software packages do this, but we show that very different results can be obtained from different packages even when using the same data and model. We describe the parameterization, features, and optimization used by eight different fitting packages that run on four different platforms. We then compare these eight packages using various data functions and data sets, revealing that there are stark differences between the packages. In addition to comparing the prediction accuracy, the predictive variance – which is important for evaluating precision of predictions and is often used in stopping criteria – is also evaluated.


62-04 Software, source code, etc. for problems pertaining to statistics
62Kxx Design of statistical experiments
62Pxx Applications of statistics
60G15 Gaussian processes
Full Text: DOI arXiv


[1] Ankenman, B. E.; Nelson, B. L.; Staum, J., Stochastic Kriging for simulation metamodeling, Operations Research, 58, 2, 371-382, (2010) · Zbl 1342.62134
[2] Ba, S., & Joseph, V. R. (2015). Maxpro: Maximum projection designs. R package version 3.1-2. Retrieved 15 February 2017 from https://CRAN.R-project.org/package=MaxPro.
[3] Bastos, L. S.; O’Hagan, A., Diagnostics for Gaussian process emulators, Technometrics, 51, 4, 425-438, (2009)
[4] Ben-Ari, E. N.; Steinberg, D. M., Modeling data from computer experiments: an empirical comparison of Kriging with MARS and projection pursuit regression, Quality Engineering, 19, 4, 327-338, (2007)
[5] Binois, M., Gramacy, R. B., & Ludkovski, M. (2016). Practical heteroskedastic Gaussian process modeling for large simulation experiments. arXiv preprint:1611.05902. Retrieved 15 February 2017 from https://arxiv.org/pdf/1611.05902.pdf.
[6] Byrd, R. H.; Lu, P.; Nocedal, J.; Zhu, C., A limited memory algorithm for bound constrained optimization, SIAM Journal on Scientific Computing, 16, 5, 1190-1208, (1995) · Zbl 0836.65080
[7] Chen, H.; Loeppky, J. L.; Sacks, J.; Welch, W. J., Analysis methods for computer experiments: how to assess and what counts?, Statistical Science, 31, 1, 40-60, (2016)
[8] Chen, X.; Wang, K.; Yang, F., Stochastic Kriging with qualitative factors, Proceedings of the 2013 Winter Simulation Conference, 790-801, (2013), IEEE Press
[9] Chipman, H. A.; George, E. I.; McCulloch, R. E., Bayesian CART model search, Journal of the American Statistical Association, 93, 443, 935-948, (1998)
[10] Christen, J.; Ichchou, M.; Troclet, B.; Ouisse, M., Global sensitivity analysis of acoustic transmission models through infinite plates, Proceedings of ISMA, 4177-4188, (2014)
[11] Ciuffo, B.; Casas, J.; Montanino, M.; Perarnau, J.; Punzo, V., Gaussian process metamodels for sensitivity analysis of traffic simulation models: case study of AIMSUN mesoscopic model, Transportation Research Record: Journal of the Transportation Research Board, 2390, 87-98, (2013)
[12] Damianou, A.; Lawrence, N., Deep Gaussian processes, Proceedings of the Sixteenth International Workshop on Artificial Intelligence and Statistics (AISTATS), 207-215, (2013)
[13] Dancik, G. (2011). mlegp: An R package for Gaussian process modeling and sensitivity analysis. Retrieved April 14, 2016, from http://download.nextag.com/cran/web/packages/mlegp/vignettes/mlegp.pdf.
[14] Dancik, G. M. (2013). mlegp: Maximum likelihood estimates of Gaussian processes. R package version 3.1.4. Retrieved 15 February 2017 from https://CRAN.R-project.org/package=mlegp.
[15] Dancik, G. M.; Dorman, K. S., Mlegp: statistical analysis for computer models of biological systems using R, Bioinformatics, 24, 17, 1966-1967, (2008)
[16] Dette, H.; Pepelyshev, A., Generalized Latin hypercube design for computer experiments, Technometrics, 52, 4, 421-429, (2010)
[17] scikit-learn developers (2016a). scikit-learn release history. Retrieved January 16, 2017, from http://scikit-learn.org/stable/whats_new.html.
[18] scikit-learn developers (2016b). sklearn.gaussian_process.gaussianprocessregressor. Retrieved January 14, 2017, from http://scikit-learn.org/stable/modules/generated/sklearn.gaussian_process.GaussianProcessRegressor.html.
[19] Du, W.; Xue, N.; Shyy, W.; Martins, J. R., A surrogate-based multi-scale model for mass transport and electrochemical kinetics in lithium-ion battery electrodes, Journal of the Electrochemical Society, 161, 8, E3086-E3096, (2014)
[20] Erickson, C. B.; Ankenman, B. E.; Sanchez, S. M., Data from Fitting gaussian process models to various data sets using eight Gaussian process software packages, Data in Brief, (2017)
[21] Gidaris, I.; Taflanidis, A. A., Performance assessment and optimization of fluid viscous dampers through life-cycle cost criteria and comparison to alternative design approaches, Bulletin of Earthquake Engineering, 13, 4, 1003-1028, (2015)
[22] Gramacy, R., Tgp: an R package for Bayesian nonstationary, semiparametric nonlinear regression and design by treed Gaussian process models, Journal of Statistical Software, 19, 9, 1-46, (2007)
[23] Gramacy, R. B., Lagp: large-scale spatial modeling via local approximate Gaussian processes in R, Technical Report, (2014), The University of Chicago
[24] Gramacy, R. B. (2015). lagp: Local approximate Gaussian process regression. R package version 1.3-2. Retrieved 14 February 2017 from https://CRAN.R-project.org/package=laGP.
[25] Gramacy, R. B.; Apley, D. W., Local Gaussian process approximation for large computer experiments, Journal of Computational and Graphical Statistics, 24, 2, 561-578, (2015)
[26] Gramacy, R. B.; Lee, H. K., Cases for the nugget in modeling computer experiments, Statistics and Computing, 22, 3, 713-722, (2012) · Zbl 1252.62098
[27] Gramacy, R. B.; Lee, H. K.H., Bayesian treed Gaussian process models with an application to computer modeling, Journal of the American Statistical Association, 103, 483, 1119-1130, (2008) · Zbl 1205.62218
[28] Hensman, J.; Fusi, N.; Lawrence, N. D., Gaussian processes for big data, (2013)
[29] JMP: Gaussian Process, (2016). JMP: Gaussian process. Retrieved August 01, 2016, from http://www.jmp.com/support/help/Gaussian_Process.shtml.
[30] Jones, D. R.; Schonlau, M.; Welch, W. J., Efficient global optimization of expensive black-box functions, Journal of Global Optimization, 13, 4, 455-492, (1998) · Zbl 0917.90270
[31] Jones, E., Oliphant, T., Peterson, P., et al., (2001). SciPy: Open source scientific tools for Python. http://www.scipy.org/, Online; accessed 19.04.16.
[32] Kleijnen, J. P.C., Design and analysis of simulation experiments, vol. 230, (2015), Springer · Zbl 1321.62006
[33] Le Gratiet, L.; Marelli, S.; Sudret, B., Metamodel-based sensitivity analysis: polynomial chaos expansions and Gaussian processes, (Ghanem, R.; Higdon, D.; Owhadi, H., Handbook of Uncertainty Quantification - Part III: Sensitivity analysis, (2016), Springer International Publishing)
[34] Liu, D. C.; Nocedal, J., On the limited memory BFGS method for large scale optimization, Mathematical programming, 45, 1-3, 503-528, (1989) · Zbl 0696.90048
[35] Liu, M.; Staum, J., Stochastic Kriging for efficient nested simulation of expected shortfall, The Journal of Risk, 12, 3, 3, (2010)
[36] Loeppky, J. L.; Sacks, J.; Welch, W. J., Choosing the sample size of a computer experiment: A practical guide, Technometrics, 51, 4, 366-376, (2009)
[37] Lophaven, S. N.; Nielsen, H. B.; Søndergaard, J., Aspects of the MATLAB toolbox DACE, Technical Report, (2002), Technical University of Denmark, DTU
[38] Lophaven, S. N., Nielsen, H. B., & Søndergaard, J. (2002b). DACE-A MATLAB Kriging toolbox, version 2.0. Technical Report. Technical University of Denmark, DTU. Informatics and Mathematical Modelling.
[39] MacDonald, B.; Ranjan, P.; Chipman, H., Gpfit: an R package for Fitting a Gaussian process model to deterministic simulator outputs, Journal of Statistical Software, 64, 12, 1-23, (2015)
[40] Mebane Jr, W. R.; Sekhon, J. S., Genetic optimization using derivatives: the rgenoud package for R, Journal of Statistical Software, 42, 11, 1-26, (2011)
[41] Morris, M. D., Factorial sampling plans for preliminary computational experiments, Technometrics, 33, 2, 161-174, (1991)
[42] Morris, M. D.; Mitchell, T. J.; Ylvisaker, D., Bayesian design and analysis of computer experiments: use of derivatives in surface prediction, Technometrics, 35, 3, 243-255, (1993) · Zbl 0785.62025
[43] Pedregosa, F.; Varoquaux, G.; Gramfort, A.; Michel, V.; Thirion, B.; Grisel, O., Scikit-learn: machine learning in python, The Journal of Machine Learning Research, 12, 2825-2830, (2011) · Zbl 1280.68189
[44] Platt, J. C.; Burges, C. J.C.; Swenson, S.; Weare, C.; Zheng, A., Learning a Gaussian process prior for automatically generating music playlists, (Dietterich, T. G.; Becker, S.; Ghahramani, Z., Advances in Neural Information Processing Systems, 14, (2002), MIT Press)
[45] Pousi, J.; Poropudas, J.; Virtanen, K., Game theoretic simulation metamodeling using stochastic Kriging, Proceedings of the 2010 Winter Simulation Conference, 1456-1467, (2010)
[46] R Core Team (2014). R: A language and environment for statistical computing. R Foundation for Statistical Computing. Vienna, Austria.
[47] Ranjan, P.; Haynes, R.; Karsten, R., A computationally stable approach to Gaussian process interpolation of deterministic computer simulation data, Technometrics, 53, 4, 366-378, (2011)
[48] Rasmussen, C. E.; Williams, C. K.I., Gaussian processes for machine learning, (2006), The MIT Press · Zbl 1177.68165
[49] Roustant, O.; Ginsbourger, D.; Deville, Y., Dicekriging, diceoptim: two R packages for the analysis of computer experiments by Kriging-based metamodeling and optimization, Journal of Statistical Software, 51, 1, 1-55, (2012)
[50] Sacks, J.; Welch, W. J.; Mitchell, T. J.; Wynn, H. P., Design and analysis of computer experiments, Statistical Science, 4, 4, 409-423, (1989) · Zbl 0955.62619
[51] Santner, T. J.; Williams, B. J.; Notz, W., The design and analysis of computer experiments, (2003), Springer Science & Business Media · Zbl 1041.62068
[52] SAS Products: JMP, (2016), Sas products: JMP. Retrieved August 01, 2016, from http://support.sas.com/software/products/jmp/.
[53] Snelson, E.; Ghahramani, Z., Sparse Gaussian processes using pseudo-inputs, Proceedings of the Advances in Neural Information Processing Systems, 1257-1264, (2005)
[54] Sung, C.-L., Gramacy, R. B., & Haaland, B. (2016). Potentially predictive variance reducing subsample locations in local Gaussian process regression. arXiv preprint:1604.04980.
[55] Surjanovic, S., & Bingham, D. (2016). Virtual library of simulation experiments: Test functions and datasets. Retrieved March 29, 2016, from http://www.sfu.ca/ ssurjano.
[56] The GPy authors (2012-2015). GPy: A Gaussian process framework in Python.
[57] The GPy authors (2015). Welcome to GPy’s documentation! Retrieved Oct 03, 2016, from https://pythonhosted.org/GPy/index.html.
[58] Villa-Vialaneix, N.; Follador, M.; Ratto, M.; Leip, A., A comparison of eight metamodeling techniques for the simulation of N_{2}O fluxes and N leaching from corn crops, Environmental Modelling & Software, 34, 51-66, (2012)
[59] Williams, C. K.; Barber, D., Bayesian classification with Gaussian processes, IEEE Transactions on Pattern Analysis and Machine Intelligence, 20, 12, 1342-1351, (1998)
[60] Woodbury, M. A., Inverting modified matrices, Memorandum report, 42, 106, (1950)
[61] Worley, B., Deterministic uncertainty analysis, Technical Report ORN-0628, (1987), National Technical Information Service 5285 Port Royal Road, Springfield, VA 22161, USA
[62] Yin, H.; Wen, G.; Fang, H.; Qing, Q.; Kong, X.; Xiao, J., Multiobjective crashworthiness optimization design of functionally graded foam-filled tapered tube based on dynamic ensemble metamodel, Materials & Design, 55, 747-757, (2014)
This reference list is based on information provided by the publisher or from digital mathematics libraries. Its items are heuristically matched to zbMATH identifiers and may contain data conversion errors. It attempts to reflect the references listed in the original paper as accurately as possible without claiming the completeness or perfect precision of the matching.