zbMATH — the first resource for mathematics

Multi-level convolutional autoencoder networks for parametric prediction of spatio-temporal dynamics. (English) Zbl 07337875
Summary: A data-driven framework is proposed towards the end of predictive modeling of complex spatio-temporal dynamics, leveraging nested non-linear manifolds. Three levels of neural networks are used, with the goal of predicting the future state of a system of interest in a parametric setting. A convolutional autoencoder is used as the top level to encode the high dimensional input data along spatial dimensions into a sequence of latent variables. A temporal convolutional autoencoder (TCAE) serves as the second level, which further encodes the output sequence from the first level along the temporal dimension, and outputs a set of latent variables that encapsulate the spatio-temporal evolution of the dynamics. The use of dilated temporal convolutions grows the receptive field exponentially with network depth, allowing for efficient processing of long temporal sequences typical of scientific computations. A fully-connected network is used as the third level to learn the mapping between these latent variables and the global parameters from training data, and predict them for new parameters. For future state predictions, the second level uses a temporal convolutional network to predict subsequent steps of the output sequence from the top level. Latent variables at the bottom-most level are decoded to obtain the dynamics in physical space at new global parameters and/or at a future time. Predictive capabilities are evaluated on a range of problems involving discontinuities, wave propagation, strong transients, and coherent structures. The sensitivity of the results to different modeling choices is assessed. The results suggest that given adequate data and careful training, effective data-driven predictive models can be constructed. Perspectives are provided on the present approach and its place in the landscape of model reduction.

68-XX Computer science
92-XX Biology and other natural sciences
WaveNet; GNMT; Adam; LSTM
Full Text: DOI
[1] Benner, P.; Gugercin, S.; Willcox, K., A survey of projection-based model reduction methods for parametric dynamical systems, SIAM Rev., 57, 4, 483-531 (2015) · Zbl 1339.37089
[2] Berkooz, G.; Holmes, P.; Lumley, J. L., The proper orthogonal decomposition in the analysis of turbulent flows, Annu. Rev. Fluid Mech., 25, 1, 539-575 (1993)
[3] Rowley, C. W.; Colonius, T.; Murray, R. M., Model reduction for compressible flows using POD and Galerkin projection, Physica D, 189, 1-2, 115-129 (2004) · Zbl 1098.76602
[4] Moore, B., Principal component analysis in linear systems: Controllability, observability, and model reduction, IEEE Trans. Automat. Control, 26, 1, 17-32 (1981) · Zbl 0464.93022
[5] Safonov, M. G.; Chiang, R., A Schur method for balanced-truncation model reduction, IEEE Trans. Automat. Control, 34, 7, 729-733 (1989) · Zbl 0687.93027
[6] Peterson, J. S., The reduced basis method for incompressible viscous flow calculations, SIAM J. Sci. Stat. Comput., 10, 4, 777-786 (1989) · Zbl 0672.76034
[7] Prud’Homme, C.; Rovas, D. V.; Veroy, K.; Machiels, L.; Maday, Y.; Patera, A. T.; Turinici, G., Reliable real-time solution of parametrized partial differential equations: Reduced-basis output bound methods, J. Fluids Eng., 124, 1, 70-80 (2001)
[8] Rozza, G.; Huynh, D. B.P.; Patera, A. T., Reduced basis approximation and a posteriori error estimation for affinely parametrized elliptic coercive partial differential equations, Arch. Comput. Methods Eng., 15, 3, 1 (2007)
[9] Baur, U.; Beattie, C.; Benner, P.; Gugercin, S., Interpolatory projection methods for parameterized model reduction, SIAM J. Sci. Comput., 33, 5, 2489-2518 (2011) · Zbl 1254.93032
[10] de Almeida, J. M., A basis for bounding the errors of proper generalised decomposition solutions in solid mechanics, Internat. J. Numer. Methods Engrg., 94, 10, 961-984 (2013) · Zbl 1352.74041
[11] Berger, J.; Gasparin, S.; Chhay, M.; Mendes, N., Estimation of temperature-dependent thermal conductivity using proper generalised decomposition for building energy management, J. Build. Phys., 40, 3, 235-262 (2016)
[12] Rowley, C. W., Model reduction for fluids, using balanced proper orthogonal decomposition, Int. J. Bifurcation Chaos, 15, 03, 997-1013 (2005) · Zbl 1140.76443
[13] Carlberg, K.; Bou-Mosleh, C.; Farhat, C., Efficient non-linear model reduction via a least-squares Petrov-Galerkin projection and compressive tensor approximations, Internat. J. Numer. Methods Engrg., 86, 2, 155-181 (2011) · Zbl 1235.74351
[14] Hijazi, S.; Stabile, G.; Mola, A.; Rozza, G., Data-driven POD-Galerkin reduced order model for turbulent flows (2019), arXiv preprint arXiv:1907.09909
[15] Xu, J.; Huang, C.; Duraisamy, K., Reduced-order modeling framework for combustor instabilities using truncated domain training, AIAA J., 1-15 (2019)
[16] Huang, C.; Xu, J.; Duraisamy, K.; Merkle, C., Exploration of reduced-order models for rocket combustion applications, (2018 AIAA Aerospace Sciences Meeting (2018)), 1183
[17] Carlberg, K.; Barone, M.; Antil, H., Galerkin v. least-squares Petrov-Galerkin projection in nonlinear model reduction, J. Comput. Phys., 330, 693-734 (2017) · Zbl 1378.65145
[18] Wang, Z.; Akhtar, I.; Borggaard, J.; Iliescu, T., Two-level discretizations of nonlinear closure models for proper orthogonal decomposition, J. Comput. Phys., 230, 1, 126-146 (2011) · Zbl 1427.76226
[19] Wang, Z.; Akhtar, I.; Borggaard, J.; Iliescu, T., Proper orthogonal decomposition closure models for turbulent flows: a numerical comparison, Comput. Methods Appl. Mech. Engrg., 237, 10-26 (2012) · Zbl 1253.76050
[20] Parish, E. J.; Wentland, C.; Duraisamy, K., The adjoint Petrov-Galerkin method for non-linear model reduction (2018), arXiv preprint arXiv:1810.03455
[21] Gouasmi, A.; Parish, E. J.; Duraisamy, K., A priori estimation of memory effects in reduced-order models of nonlinear systems using the Mori-Zwanzig formalism, Proc. R. Soc. A, 473, 2205, Article 20170385 pp. (2017) · Zbl 1402.76065
[22] Peherstorfer, B.; Willcox, K., Online adaptive model reduction for nonlinear systems via low-rank updates, SIAM J. Sci. Comput., 37, 4, A2123-A2150 (2015) · Zbl 1323.65102
[23] Peherstorfer, B., Model reduction for transport-dominated problems via online adaptive bases and adaptive sampling (2018), arXiv preprint arXiv:1812.02094
[24] Amsallem, D.; Farhat, C., An online method for interpolating linear parametric reduced-order models, SIAM J. Sci. Comput., 33, 5, 2169-2198 (2011) · Zbl 1269.65059
[25] Hartman, D.; Mestha, L. K., A deep learning framework for model reduction of dynamical systems, (2017 IEEE Conference on Control Technology and Applications (CCTA) (2017), IEEE), 1917-1922
[26] DeMers, D.; Cottrell, G. W., Non-linear dimensionality reduction, (Advances in Neural Information Processing Systems (1993)), 580-587
[27] Omata, N.; Shirayama, S., A novel method of low-dimensional representation for temporal behavior of flow fields using deep autoencoder, AIP Adv., 9, 1, Article 015006 pp. (2019)
[28] Lee, K.; Carlberg, K., Model reduction of dynamical systems on nonlinear manifolds using deep convolutional autoencoders (2018), arXiv preprint arXiv:1812.08373
[29] Guo, X.; Li, W.; Iorio, F., Convolutional neural networks for steady flow approximation, (Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining (2016), ACM), 481-490
[30] Puligilla, S. C.; Jayaraman, B., Deep multilayer convolution frameworks for data-driven learning of fluid flow dynamics, (2018 Fluid Dynamics Conference (2018)), 3091
[31] Carlberg, K. T.; Jameson, A.; Kochenderfer, M. J.; Morton, J.; Peng, L.; Witherden, F. D., Recovering missing CFD data for high-order discretizations using deep neural networks and dynamics learning, J. Comput. Phys. (2019)
[32] Couplet, M.; Basdevant, C.; Sagaut, P., Calibrated reduced-order POD-Galerkin system for fluid flow modelling, J. Comput. Phys., 207, 1, 192-220 (2005) · Zbl 1177.76283
[33] Astrid, P., Reduction of Process Simulation Models: A Proper Orthogonal Decomposition Approach (2004), Technische Universiteit Eindhoven Eindhoven: Technische Universiteit Eindhoven Eindhoven Netherlands
[34] Astrid, P.; Weiland, S.; Willcox, K.; Backx, T., Missing point estimation in models described by proper orthogonal decomposition, IEEE Trans. Automat. Control, 53, 10, 2237-2251 (2008) · Zbl 1367.93110
[35] Barrault, M.; Maday, Y.; Nguyen, N. C.; Patera, A. T., An ‘empirical interpolation’method: application to efficient reduced-basis discretization of partial differential equations, C. R. Math., 339, 9, 667-672 (2004) · Zbl 1061.65118
[36] Chaturantabut, S.; Sorensen, D. C., Discrete empirical interpolation for nonlinear model reduction, (Decision and Control, 2009 Held Jointly with the 2009 28th Chinese Control Conference. CDC/CCC 2009. Proceedings of the 48th IEEE Conference on (2009), IEEE), 4316-4321
[37] Drmac, Z.; Gugercin, S., A new selection operator for the discrete empirical interpolation method—Improved a priori error bound and extensions, SIAM J. Sci. Comput., 38, 2, A631-A648 (2016) · Zbl 1382.65193
[38] Peherstorfer, B.; Drmač, Z.; Gugercin, S., Stabilizing discrete empirical interpolation via randomized and deterministic oversampling (2018), arXiv preprint arXiv:1808.10473
[39] Brunton, S. L.; Proctor, J. L.; Kutz, J. N., Discovering governing equations from data by sparse identification of nonlinear dynamical systems, Proc. Natl. Acad. Sci., 113, 15, 3932-3937 (2016) · Zbl 1355.94013
[40] Gu, C., QLMOR: A projection-based nonlinear model order reduction approach using quadratic-linear representation of nonlinear systems, IEEE Trans. Comput.-Aided Des. Integr. Circuits Syst., 30, 9, 1307-1320 (2011)
[41] Kramer, B.; Willcox, K. E., Nonlinear model order reduction via lifting transformations and proper orthogonal decomposition, AIAA J., 57, 6, 2297-2307 (2019)
[42] Kramer, B.; Willcox, K. E., Balanced truncation model reduction for lifted nonlinear systems (2019), arXiv preprint arXiv:1907.12084
[43] Schmidt, M. D.; Vallabhajosyula, R. R.; Jenkins, J. W.; Hood, J. E.; Soni, A. S.; Wikswo, J. P.; Lipson, H., Automated refinement and inference of analytical models for metabolic networks, Phys. Biol., 8, 5, Article 055011 pp. (2011)
[44] Daniels, B. C.; Nemenman, I., Automated adaptive inference of phenomenological dynamical models, Nature Commun., 6, 8133 (2015)
[45] Barthelmann, V.; Novak, E.; Ritter, K., High dimensional polynomial interpolation on sparse grids, Adv. Comput. Math., 12, 4, 273-288 (2000) · Zbl 0944.41001
[46] Guo, M.; Hesthaven, J. S., Reduced order modeling for nonlinear structural analysis using gaussian process regression, Comput. Methods Appl. Mech. Engrg., 341, 807-826 (2018) · Zbl 1440.65206
[47] Hesthaven, J. S.; Ubbiali, S., Non-intrusive reduced order modeling of nonlinear problems using neural networks, J. Comput. Phys., 363, 55-78 (2018) · Zbl 1398.65330
[48] Wang, Q.; Hesthaven, J. S.; Ray, D., Non-intrusive reduced order modeling of unsteady flows using artificial neural networks with application to a combustion problem, J. Comput. Phys., 384, 289-307 (2019)
[49] Mohan, A.; Daniel, D.; Chertkov, M.; Livescu, D., Compressed convolutional LSTM: An efficient deep learning framework to model high fidelity 3D turbulence (2019), arXiv preprint arXiv:1903.00033
[50] Lee, S.; You, D., Data-driven prediction of unsteady flow fields over a circular cylinder using deep learning (2018), arXiv preprint arXiv:1804.06076
[51] Rumelhart, D. E.; Hinton, G. E.; Williams, R. J., Learning representations by back-propagating errors, Cogn. Model., 5, 3, 1 (1988)
[52] Hochreiter, S.; Schmidhuber, J., LSTM can solve hard long time lag problems, (Advances in Neural Information Processing Systems (1997)), 473-479
[53] Gonzalez, F. J.; Balajewicz, M., Deep convolutional recurrent autoencoders for learning low-dimensional feature dynamics of fluid systems (2018), arXiv preprint arXiv:1808.01346
[54] Maulik, R.; Mohan, A.; Lusch, B.; Madireddy, S.; Balaprakash, P.; Livescu, D., Time-series learning of latent-space dynamics for reduced-order model closure, Physica D, Article 132368 pp. (2020)
[55] Xingjian, S.; Chen, Z.; Wang, H.; Yeung, D.-Y.; Wong, W.-K.; Woo, W.-c., Convolutional LSTM network: A machine learning approach for precipitation nowcasting, (Advances in Neural Information Processing Systems (2015)), 802-810
[56] Yu, B.; Yin, H.; Zhu, Z., Spatio-temporal graph convolutional networks: A deep learning framework for traffic forecasting (2017), arXiv preprint arXiv:1709.04875
[57] Elman, J. L., Finding structure in time, Cogn. Sci., 14, 2, 179-211 (1990)
[58] Hochreiter, S.; Schmidhuber, J., Long short-term memory, Neural Comput., 9, 8, 1735-1780 (1997)
[59] Cho, K.; Van Merriënboer, B.; Bahdanau, D.; Bengio, Y., On the properties of neural machine translation: Encoder-decoder approaches (2014), arXiv preprint arXiv:1409.1259
[60] Y. Wang, M. Huang, X. Zhu, L. Zhao, Attention-based LSTM for aspect-level sentiment classification, in: Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, 2016, pp. 606-615.
[61] Wu, Y.; Schuster, M.; Chen, Z.; Le, Q. V.; Norouzi, M.; Macherey, W.; Krikun, M.; Cao, Y.; Gao, Q.; Macherey, K., Google’s neural machine translation system: Bridging the gap between human and machine translation (2016), arXiv preprint arXiv:1609.08144
[62] Qing, X.; Niu, Y., Hourly day-ahead solar irradiance prediction using weather forecasts by LSTM, Energy, 148, 461-468 (2018)
[63] Rather, A. M.; Agarwal, A.; Sastry, V., Recurrent neural network and a hybrid model for prediction of stock returns, Expert Syst. Appl., 42, 6, 3234-3241 (2015)
[64] Eck, D.; Schmidhuber, J., A First Look at Music Composition using Lstm Recurrent Neural Networks, Vol. 103, 48 (2002), Istituto Dalle Molle Di Studi Sull Intelligenza Artificiale
[65] Oord, A.v.d.; Dieleman, S.; Zen, H.; Simonyan, K.; Vinyals, O.; Graves, A.; Kalchbrenner, N.; Senior, A.; Kavukcuoglu, K., Wavenet: A generative model for raw audio (2016), arXiv preprint arXiv:1609.03499
[66] Bai, S.; Kolter, J. Z.; Koltun, V., An empirical evaluation of generic convolutional and recurrent networks for sequence modeling (2018), arXiv preprint arXiv:1803.01271
[67] Gehring, J.; Auli, M.; Grangier, D.; Dauphin, Y. N., A convolutional encoder model for neural machine translation (2016), arXiv preprint arXiv:1611.02344
[68] C. Lea, M.D. Flynn, R. Vidal, A. Reiter, G.D. Hager, Temporal convolutional networks for action segmentation and detection, in: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2017, pp. 156-165.
[69] Dauphin, Y. N.; Fan, A.; Auli, M.; Grangier, D., Language modeling with gated convolutional networks, (International Conference on Machine Learning (2017)), 933-941
[70] K. He, X. Zhang, S. Ren, J. Sun, Delving deep into rectifiers: Surpassing human-level performance on imagenet classification, in: Proceedings of the IEEE International Conference on Computer Vision, 2015, pp. 1026-1034.
[71] Dumoulin, V.; Visin, F., A guide to convolution arithmetic for deep learning (2016), arXiv preprint arXiv:1603.07285
[72] Kingma, D. P.; Ba, J., Adam: A method for stochastic optimization (2014), arXiv preprint arXiv:1412.6980
[73] Sod, G. A., A survey of several finite difference methods for systems of nonlinear hyperbolic conservation laws, J. Comput. Phys., 27, 1, 1-31 (1978) · Zbl 0387.76063
[74] Danaila, I.; Joly, P.; Kaber, S. M.; Postel, M., Gas dynamics: The Riemann problem and discontinuous solutions: Application to the shock tube problem, (Introduction to Scientific Computing (2007), Springer: Springer New York, NY), 213-233
[75] Parish, E. J.; Carlberg, K. T., Time-series machine-learning error models for approximate solutions to parameterized dynamical systems (2019), arXiv preprint arXiv:1907.11822
[76] Huang, C.; Duraisamy, K.; Merkle, C. L., Investigations and improvement of robustness of reduced-order models of reacting flow, AIAA J., 57, 12, 5377-5389 (2019)
[77] Lee, D.; Sezer-Uzol, N.; Horn, J. F.; Long, L. N., Simulation of helicopter shipboard launch and recovery with time-accurate airwakes, J. Aircr., 42, 2, 448-461 (2005)
[78] Wilcox, D. C., Reassessment of the scale-determining equation for advanced turbulence models, AIAA J., 26, 11, 1299-1310 (1988) · Zbl 0664.76057
[79] Huang, C.; Duraisamy, K.; Merkle, C., Data-informed species limiters for local robustness control of reduced-order models of reacting flow, (AIAA Scitech 2020 Forum (2020)), 2141
[80] Raissi, M.; Perdikaris, P.; Karniadakis, G. E., Physics-informed neural networks: A deep learning framework for solving forward and inverse problems involving nonlinear partial differential equations, J. Comput. Phys., 378, 686-707 (2019) · Zbl 1415.68175
[81] Pan, S.; Duraisamy, K., Physics-informed probabilistic learning of linear embeddings of non-linear dynamics with guaranteed stability, SIAM J. Appl. Dyn. Syst. (2020), arXiv preprint arXiv:1906.03663 · Zbl 1442.37090
[82] Swischuk, R.; Kramer, B.; Huang, C.; Willcox, K., Learning physics-based reduced-order models for a single-injector combustion process (2019), arXiv preprint arXiv:1908.03620
[83] Gin, C.; Lusch, B.; Brunton, S. L.; Kutz, J. N., Deep learning models for global coordinate transformations that linearize PDEs (2019), arXiv preprint arXiv:1911.02710
[84] Kipf, T. N.; Welling, M., Semi-supervised classification with graph convolutional networks (2016), arXiv preprint arXiv:1609.02907
This reference list is based on information provided by the publisher or from digital mathematics libraries. Its items are heuristically matched to zbMATH identifiers and may contain data conversion errors. It attempts to reflect the references listed in the original paper as accurately as possible without claiming the completeness or perfect precision of the matching.