Multi-fidelity classification using Gaussian processes: accelerating the prediction of large-scale computational models. (English) Zbl 1442.62008

Summary: Machine learning techniques typically rely on large datasets to create accurate classifiers. However, there are situations when data is scarce and expensive to acquire. This is the case of studies that rely on state-of-the-art computational models which typically take days to run, thus hindering the potential of machine learning tools. In this work, we present a novel classifier that takes advantage of lower fidelity models and inexpensive approximations to predict the binary output of expensive computer simulations. We postulate an autoregressive model between the different levels of fidelity with Gaussian process priors. We adopt a fully Bayesian treatment for the hyper-parameters and use Markov chain Monte Carlo samplers. We take advantage of the probabilistic nature of the classifier to implement active learning strategies. We also introduce a sparse approximation to enhance the ability of the multi-fidelity classifier to handle a large amount of low fidelity samples. We test these multi-fidelity classifiers against their single-fidelity counterpart with synthetic data, showing a median computational cost reduction of 23% for a target accuracy of 90%. In an application to cardiac electrophysiology, the multi-fidelity classifier achieves an F1 score, the harmonic mean of precision and recall, of 99.6% compared to 74.1% of a single-fidelity classifier when both are trained with 50 samples. In general, our results show that the multi-fidelity classifiers outperform their single-fidelity counterpart in terms of accuracy in all cases. We envision that this new tool will enable researchers to study classification problems that would otherwise be prohibitively expensive. Source code is available at https://github.com/fsahli/MFclass.


62-08 Computational methods for problems pertaining to statistics
62H30 Classification and discrimination; cluster analysis (statistical aspects)


MFclass; GitHub
Full Text: DOI arXiv


[1] Hannun, A. Y.; Rajpurkar, P.; Haghpanahi, M.; Tison, G. H.; Bourn, C.; Turakhia, M. P.; Ng, A. Y., Cardiologist-level arrhythmia detection and classification in ambulatory electrocardiograms using a deep neural network, Nature Med., 25, 1, 65 (2019)
[2] Morris, P. D.; Narracott, A.; von Tengg-Kobligk, H.; Soto, D. A.S.; Hsiao, S.; Lungu, A.; Evans, P.; Bressloff, N. W.; Lawford, P. V.; Hose, D. R., Computational fluid dynamics modelling in cardiovascular medicine, Heart, 102, 1, 18-28 (2016)
[3] Lee, T.; Turin, S. Y.; Gosain, A. K.; Bilionis, I.; Tepole, A. B., Propagation of material behavior uncertainty in a nonlinear finite element model of reconstructive surgery, Biomech. Model. Mechanobiol., 17, 6, 1857-1873 (2018)
[4] Forrester, A.; Sobester, A.; Keane, A., Engineering Design Via Surrogate Modelling: a Practical Guide (2008), John Wiley & Sons
[5] Santner, T. J.; Williams, B. J.; Notz, W. I., The Design and Analysis of Computer Experiments, Vol. 1 (2003), Springer · Zbl 1041.62068
[6] Sahli Costabal, F.; Yao, J.; Kuhl, E., Predicting drug-induced arrhythmias by multiscale modeling, Int. J. Numer. Methods Biomed. Eng., 34, Article e2964 pp. (2018)
[7] Sahli Costabal, F.; Matsuno, K.; Yao, J.; Perdikaris, P.; Kuhl, E., Machine learning in drug development: Characterizing the effect of 30 drugs on the QT interval using Gaussian process regression, sensitivity analysis, and uncertainty quantification, Comput. Methods Appl. Mech. Engrg., 348, 313-333 (2019) · Zbl 1440.62371
[8] Schiavazzi, D.; Doostan, A.; Iaccarino, G.; Marsden, A., A generalized multi-resolution expansion for uncertainty propagation with application to cardiovascular modeling, Comput. Methods Appl. Mech. Engrg., 314, 196-221 (2017) · Zbl 1439.74213
[9] Hurtado, D. E.; Castro, S.; Madrid, P., Uncertainty quantification of two models of cardiac electromechanics, Int. J. Numer. Methods Biomed. Eng., e2984, 1-21 (2017)
[10] Biehler, J.; Gee, M. W.; Wall, W. A., Towards efficient uncertainty quantification in complex and large-scale biomechanical problems based on a Bayesian multi-fidelity scheme, Biomech. Model. Mechanobiol., 14, 3, 489-513 (2015)
[11] Peherstorfer, B.; Willcox, K.; Gunzburger, M., Survey of multifidelity methods in uncertainty propagation, inference, and optimization, SIAM Rev., 60, 3, 550-591 (2018) · Zbl 1458.65003
[12] Qian, E.; Peherstorfer, B.; O’Malley, D.; Vesselinov, V.; Willcox, K., Multifidelity Monte Carlo estimation of variance and sensitivity indices, SIAM/ASA J. Uncertain. Quantif., 6, 2, 683-706 (2018) · Zbl 1394.62175
[13] Quaglino, A.; Pezzuto, S.; Koutsourelakis, P.; Auricchio, A.; Krause, R., Fast uncertainty quantification of activation sequences in patient-specific cardiac electrophysiology meeting clinical time constraints, Int. J. Numer. Methods Biomed. Eng., 34, 7, Article e2985 pp. (2018)
[14] Koutsourelakis, P.-S., A multi-resolution, non-parametric, Bayesian framework for identification of spatially-varying model parameters, J. Comput. Phys., 228, 17, 6184-6211 (2009) · Zbl 1190.62211
[15] Kennedy, M. C.; O’Hagan, A., Predicting the output from a complex computer code when fast approximations are available, Biometrika, 87, 1, 1-13 (2000) · Zbl 0974.62024
[16] Parussini, L.; Venturi, D.; Perdikaris, P.; Karniadakis, G. E., Multi-fidelity Gaussian process regression for prediction of random fields, J. Comput. Phys., 336, 36-50 (2017) · Zbl 1419.62272
[17] Le Gratiet, L.; Garnier, J., Recursive co-kriging model for design of computer experiments with multiple levels of fidelity, Int. J. Uncertain. Quantif., 4, 5 (2014)
[18] Perdikaris, P.; Venturi, D.; Karniadakis, G. E., Multifidelity information fusion algorithms for high-dimensional systems and massive data sets, SIAM J. Sci. Comput., 38, 4, B521-B538 (2016) · Zbl 1342.62110
[19] Gray, R. A.; Pertsov, A. M.; Jalife, J., Spatial and temporal organization during cardiac fibrillation, Nature, 392, 6671, 75 (1998)
[20] Budday, S.; Raybaud, C.; Kuhl, E., A mechanical model predicts morphological abnormalities in the developing human brain, Sci. Rep., 4, 5644 (2014)
[21] Holland, M.; Li, B.; Feng, X.; Kuhl, E., Instabilities of soft films on compliant substrates, J. Mech. Phys. Solids, 98, 350-365 (2017)
[22] Rasmussen, C. E.; Williams, C. K.I., Gaussian Processes for Machine Learning (2006), MIT Press, Cambridge · Zbl 1177.68165
[23] Robert, C.; Casella, G., Monte Carlo Statistical Methods (2013), Springer Science & Business Media
[24] Neal, R., Regression and classification using Gaussian process priors, Bayesian Stat., 6, 475-501 (1999) · Zbl 0974.62072
[25] Kapoor, A.; Grauman, K.; Urtasun, R.; Darrell, T., Active learning with gaussian processes for object categorization, (2007 IEEE 11th International Conference on Computer Vision (2007), IEEE), 1-8
[26] Gramacy, R. B.; Polson, N. G., Particle learning of Gaussian process models for sequential design and optimization, J. Comput. Graph. Statist., 20, 1, 102-118 (2017), arXiv:arXiv:0909.5262v3
[27] Nickisch, H.; Rasmussen, C. E., Approximations for binary Gaussian process classification, Mach. Learn. Res., 9, 2035-2078 (2008) · Zbl 1225.62087
[28] Hoffman, M. D.; Gelman, A., The No-U-Turn sampler: adaptively setting path lengths in Hamiltonian Monte Carlo, J. Mach. Learn. Res., 15, 1, 1593-1623 (2014) · Zbl 1319.60150
[29] Salvatier, J.; Wiecki, T. V.; Fonnesbeck, C., Probabilistic programming in python using Pymc3, PeerJ Comput. Sci., 2, e55 (2016)
[30] Perdikaris, P.; Raissi, M.; Damianou, A.; Lawrence, N.; Karniadakis, G. E., Nonlinear information fusion algorithms for data-efficient multi-fidelity modelling, Proc. R. Soc. Lond. Ser. A Math. Phys. Eng. Sci., 473, 2198, Article 20160751 pp. (2017) · Zbl 1407.62252
[31] Snelson, E.; Ghahramani, Z., Sparse Gaussian processes using pseudo-inputs, (Advances in Neural Information Processing Systems (2006)), 1257-1264
[32] Quiñonero-Candela, J.; Rasmussen, C. E., A unifying view of sparse approximate Gaussian process regression, J. Mach. Learn. Res., 6, 1939-1959 (2005) · Zbl 1222.68282
[33] Titsias, M., Variational learning of inducing variables in sparse Gaussian processes, (Artificial Intelligence and Statistics (2009)), 567-574
[34] Hensman, J.; Matthews, A. G.; Filippone, M.; Ghahramani, Z., MCMC for variationally sparse Gaussian processes, (Advances in Neural Information Processing Systems (2015)), 1648-1656
[35] Cohn, D. A.; Ghahramani, Z.; Jordan, M. I., Active learning with statistical models, J. Artificial Intelligence Res., 4, 129-145 (1996) · Zbl 0900.68366
[36] MacKay, D. J.; Mac Kay, D. J., Information Theory, Inference and Learning Algorithms (2003), Cambridge university press
[37] Stein, M., Large sample properties of simulations using Latin hypercube sampling, Technometrics, 29, 2, 143-151 (1987) · Zbl 0627.62010
[38] Wilcoxon, F., Individual comparisons by ranking methods, Biom. Bull., 1, 6, 80-83 (1945)
[39] Mann, H. B.; Whitney, D. R., On a test of whether one of two random variables is stochastically larger than the other, Ann. Math. Stat., 50-60 (1947) · Zbl 0041.26103
[40] Moreno, J. D.; Zhu, Z. I.; Yang, P.-c.; Bankston, J. R.; Jeng, M.-t.; Kang, C.; Wang, L.; Bayer, J. D.; Christini, D. J.; Trayanova, N.a.; Ripplinger, C. M.; Kass, R. S.; Clancy, C. E., A computational model to predict the effects of class I Anti- Arrhythmic drugs on ventricular rhythms, Sci. Transl. Med., 3, 98, 1-20 (2012)
[41] Sahli Costabal, F.; Yao, J.; Sher, A.; Kuhl, E., Predicting critical drug concentrations and torsadogenic risk using a multiscale exposure-response simulator, Prog. Biophys. Mol. Biol. (2018)
[42] Aliev, R. R.; Panfilov, A. V., A simple two-variable model of Cardiac excitation, Chaos Solitons Fractals, 7, 3, 293-301 (1996)
[43] Sahli Costabal, F.; Zaman, J.; Kuhl, E.; Narayan, S., Interpreting activation mapping of atrial fibrillation: A hybrid computational/physiological study, Ann. Biomed. Eng., 46, 2 (2018)
[44] Sahli Costabal, F.; Hurtado, D.; Kuhl, E., Generating Purkinje networks in the human heart, J. Biomech., 49, 2455-2465 (2016)
[45] Hurtado, D.; Kuhl, E., Computational modelling of electrocardiograms: repolarisation and T-wave polarity in the human heart, Comput. Methods Biomech. Biomed. Eng., 17, 9, 986-996 (2014)
This reference list is based on information provided by the publisher or from digital mathematics libraries. Its items are heuristically matched to zbMATH identifiers and may contain data conversion errors. It attempts to reflect the references listed in the original paper as accurately as possible without claiming the completeness or perfect precision of the matching.