×

Deep learning of dynamics and signal-noise decomposition with time-stepping constraints. (English) Zbl 1452.68170

Summary: A critical challenge in the data-driven modeling of dynamical systems is producing methods robust to measurement error, particularly when data is limited. Many leading methods either rely on denoising prior to learning or on access to large volumes of data to average over the effect of noise. We propose a novel paradigm for data-driven modeling that simultaneously learns the dynamics and estimates the measurement noise at each observation. By constraining our learning algorithm, our method explicitly accounts for measurement error in the map between observations, treating both the measurement error and the dynamics as unknowns to be identified, rather than assuming idealized noiseless trajectories. We model the unknown vector field using a deep neural network, imposing a Runge-Kutta integrator structure to isolate this vector field, even when the data has a non-uniform timestep, thus constraining and focusing the modeling effort. We demonstrate the ability of this framework to form predictive models on a variety of canonical test problems of increasing complexity and show that it is robust to substantial amounts of measurement error. We also discuss issues with the generalizability of neural network models for dynamical systems and provide open-source code for all examples.

MSC:

68T07 Artificial neural networks and deep learning
34C28 Complex behavior and chaotic systems of ordinary differential equations
34C60 Qualitative investigation and simulation of ordinary differential equation models
37M10 Time series analysis of dynamical systems
PDFBibTeX XMLCite
Full Text: DOI arXiv

References:

[1] Tyrus, Berry; Sauer, Timothy, Adaptive ensemble Kalman filtering of non-linear systems, Tellus, Ser. A Dyn. Meteorol. Oceanogr., 65, 1, Article 20331 pp. (2013)
[2] Billings, Stephen A., Nonlinear System Identification: NARMAX Methods in the Time, Frequency, and Spatio-Temporal Domains (2013), John Wiley & Sons · Zbl 1287.93101
[3] Bongard, Josh; Lipson, Hod, Automated reverse engineering of nonlinear dynamical systems, Proc. Natl. Acad. Sci. USA, 104, 24, 9943-9948 (2007) · Zbl 1155.37044
[4] Boninsegna, Lorenzo; Nüske, Feliks; Clementi, Cecilia, Sparse learning of stochastic dynamic equations (2017), arXiv preprint
[5] Brunton, S. L.; Brunton, B. W.; Proctor, J. L.; Kaiser, E.; Kutz, J. N., Chaos as an intermittently forced linear system, Nat. Commun., 8, 19, 1-9 (2017)
[6] Brunton, S. L.; Noack, B. R., Closed-loop turbulence control: progress and challenges, Appl. Mech. Rev., 67, Article 050801 pp. (2015)
[7] Brunton, S. L.; Proctor, J. L.; Kutz, J. N., Discovering governing equations from data by sparse identification of nonlinear dynamical systems, Proc. Natl. Acad. Sci. USA, 113, 15, 3932-3937 (2016) · Zbl 1355.94013
[8] Caruana, Rich; Lawrence, Steve; Giles, C. Lee, Overfitting in neural nets: backpropagation, conjugate gradient, and early stopping, (Advances in Neural Information Processing Systems (2001)), 402-408
[9] Chen, Sheng; Billings, S. A.; Grant, P. M., Non-linear system identification using neural networks, Int. J. Control, 51, 6, 1191-1214 (1990) · Zbl 0703.93071
[10] Chen, Sheng; Billings, Steve A., Representations of non-linear systems: the narmax model, Int. J. Control, 49, 3, 1013-1032 (1989) · Zbl 0674.93009
[11] Chen, Tian Qi; Rubanova, Yulia; Bettencourt, Jesse; Duvenaud, David, Neural ordinary differential equations (2018), arXiv preprint
[12] Choromanska, Anna; Henaff, Mikael; Mathieu, Michael; Arous, Gérard Ben; LeCun, Yann, The loss surfaces of multilayer networks, (Artificial Intelligence and Statistics (2015)), 192-204
[13] Colonius, T.; Taira, K., A fast immersed boundary method using a nullspace approach and multi-domain far-field boundary conditions, Comput. Methods Appl. Mech. Eng., 197, 2131-2146 (2008) · Zbl 1158.76395
[14] Courtier, Philippe; Thépaut, J-N.; Hollingsworth, Anthony, A strategy for operational implementation of 4d-var, using an incremental approach, Q. J. R. Meteorol. Soc., 120, 519, 1367-1387 (1994)
[15] Daniels, Bryan; Nemenman, Ilya, Automated adaptive inference of phenomenological dynamical models, Nat. Commun., 6 (2015)
[16] Daniels, Bryan; Nemenman, Ilya, Efficient inference of parsimonious phenomenological models of cellular dynamics using s-systems and alternating regression, PLoS ONE, 10, 3, Article e0119821 pp. (2015)
[17] Dauphin, Yann N.; Pascanu, Razvan; Gulcehre, Caglar; Cho, Kyunghyun; Ganguli, Surya; Bengio, Yoshua, Identifying and attacking the saddle point problem in high-dimensional non-convex optimization, (Advances in Neural Information Processing Systems (2014)), 2933-2941
[18] Glorot, Xavier; Bengio, Yoshua, Understanding the difficulty of training deep feedforward neural networks, (Proceedings of the Thirteenth International Conference on Artificial Intelligence and Statistics (2010)), 249-256
[19] Gonzalez-Garcia, R.; Rico-Martinez, R.; Kevrekidis, I. G., Identification of distributed parameter systems: a neural net based approach, Comput. Chem. Eng., 22, S965-S968 (1998)
[20] Ian Goodfellow, Yoshua Bengio, Aaron Courville, Deep learning.; Ian Goodfellow, Yoshua Bengio, Aaron Courville, Deep learning. · Zbl 1373.68009
[21] Ho, B. L.; Kalman, R. E., Effective construction of linear state-variable models from input/output data, (3rd Annual Allerton Conference (1965)), 449-459
[22] Hornik, Kurt, Approximation capabilities of multilayer feedforward networks, Neural Netw., 4, 2, 251-257 (1991)
[23] Juang, J. N., Applied System Identification (1994), Prentice Hall PTR: Prentice Hall PTR Upper Saddle River, New Jersey · Zbl 0838.93009
[24] Juang, J. N.; Pappa, R. S., An eigensystem realization algorithm for modal parameter identification and model reduction, J. Guid. Control Dyn., 8, 5, 620-627 (1985) · Zbl 0589.93008
[25] Juang, J. N.; Phan, M.; Horta, L. G.; Longman, R. W., Identification of Observer/Kalman Filter Markov Parameters: Theory and Experiments (1991), NASA, Technical Memorandum 104069
[26] Kalman, Rudolph Emil, A new approach to linear filtering and prediction problems, J. Fluids Eng., 82, 1, 35-45 (1960)
[27] Kutz, J. Nathan; Brunton, Steven L.; Brunton, Bingni W.; Proctor, Joshua L., Dynamic Mode Decomposition: Data-Driven Modeling of Complex Systems, vol. 149 (2016), SIAM · Zbl 1365.65009
[28] LeVeque, Randall J., Finite Difference Methods for Ordinary and Partial Differential Equations: Steady-State and Time-Dependent Problems, vol. 98 (2007), Siam · Zbl 1127.65080
[29] Ljung, L., System Identification: Theory for the User (1999), Prentice Hall
[30] Loiseau, J.-C.; Brunton, S. L., Constrained sparse Galerkin regression, J. Fluid Mech., 838, 42-67 (2018) · Zbl 1419.76205
[31] Loiseau, J.-C.; Noack, B. R.; Brunton, S. L., Sparse reduced-order modeling: sensor-based dynamics to full-state estimation, J. Fluid Mech., 844, 459-490 (2018) · Zbl 1461.76369
[32] Longman, Richard W.; Juang, Jer-Nan, Recursive form of the eigensystem realization algorithm for system identification, J. Guid. Control Dyn., 12, 5, 647-652 (1989) · Zbl 0695.93015
[33] Lu, Zhixin; Hunt, Brian R.; Ott, Edward, Attractor reconstruction by machine learning, Chaos, 28, 6, Article 061104 pp. (2018)
[34] Lusch, Bethany; Kutz, J. Nathan; Brunton, Steven L., Deep learning for universal linear embeddings of nonlinear dynamics (2017), arXiv preprint
[35] Mallat, Stéphane, Understanding deep convolutional networks, Philos. Trans. R. Soc. A, 374, 2065, Article 20150203 pp. (2016)
[36] Mangan, Niall M.; Kutz, J. Nathan; Brunton, Steven L.; Proctor, Joshua L., Model selection for dynamical systems via sparse regression and information criteria, (Proc. R. Soc. A, vol. 473 (2017), The Royal Society), Article 20170009 pp. · Zbl 1404.65308
[37] Mardt, Andreas; Pasquali, Luca; Wu, Hao; Noé, Frank, VAMPnets: deep learning of molecular kinetics, Nat. Commun., 9, 5 (2018)
[38] Marsden, Jerrold E.; West, Matthew, Discrete mechanics and variational integrators, Acta Numer., 10, 357-514 (2001) · Zbl 1123.37327
[39] Milano, Michele; Koumoutsakos, Petros, Neural network modeling for near wall turbulent flow, J. Comput. Phys., 182, 1, 1-26 (2002) · Zbl 1090.76535
[40] Noack, B. R.; Afanasiev, K.; Morzynski, M.; Tadmor, G.; Thiele, F., A hierarchy of low-dimensional models for the transient and post-transient cylinder wake, J. Fluid Mech., 497, 335-363 (2003) · Zbl 1067.76033
[41] Pan, Shaowu; Duraisamy, Karthik, Data-driven discovery of closure models (2018), arXiv preprint · Zbl 1411.70023
[42] Pan, Shaowu; Duraisamy, Karthik, Long-time predictive modeling of nonlinear dynamical systems using neural networks (2018), arXiv preprint · Zbl 1411.70023
[43] Pascanu, Razvan; Mikolov, Tomas; Bengio, Yoshua, On the difficulty of training recurrent neural networks, (International Conference on Machine Learning (2013)), 1310-1318
[44] Pathak, Jaideep; Lu, Zhixin; Hunt, Brian R.; Girvan, Michelle; Ott, Edward, Using machine learning to replicate chaotic attractors and calculate Lyapunov exponents from data, Chaos, 27, 12, Article 121102 pp. (2017) · Zbl 1390.37138
[45] Pathak, Jaideep; Wikner, Alexander; Fussell, Rebeckah; Chandra, Sarthak; Hunt, Brian R.; Girvan, Michelle; Ott, Edward, Hybrid forecasting of chaotic processes: using machine learning in conjunction with a knowledge-based model, Chaos, 28, 4, Article 041101 pp. (2018)
[46] Phan, M.; Horta, L. G.; Juang, J. N.; Longman, R. W., Linear system identification via an asymptotically stable observer, J. Optim. Theory Appl., 79, 59-86 (1993) · Zbl 0794.93022
[47] Phan, M.; Juang, J. N.; Longman, R. W., Identification of linear-multivariable systems by identification of observers with assigned real eigenvalues, J. Astronaut. Sci., 40, 2, 261-279 (1992)
[48] Qin, Tong; Wu, Kailiang; Xiu, Dongbin, Data driven governing equations approximation using deep neural networks (2018), arXiv preprint
[49] Raissi, Maziar, Deep hidden physics models: deep learning of nonlinear partial differential equations (2018), arXiv preprint · Zbl 1439.68021
[50] Raissi, Maziar; Karniadakis, George Em, Hidden physics models: machine learning of nonlinear partial differential equations (2017), arXiv preprint · Zbl 1381.68248
[51] Raissi, Maziar; Perdikaris, Paris; Karniadakis, George Em, Physics informed deep learning (part i): data-driven solutions of nonlinear partial differential equations (2017), arXiv preprint · Zbl 1382.65229
[52] Raissi, Maziar; Perdikaris, Paris; Karniadakis, George Em, Physics informed deep learning (part ii): data-driven discovery of nonlinear partial differential equations (2017), arXiv preprint · Zbl 1380.68339
[53] Raissi, Maziar; Perdikaris, Paris; Karniadakis, George Em, Multistep neural networks for data-driven discovery of nonlinear dynamical systems (2018), arXiv preprint · Zbl 1386.65030
[54] Rudy, Samuel H.; Brunton, Steven L.; Proctor, Joshua L.; Kutz, J. Nathan, Data-driven discovery of partial differential equations, Sci. Adv., 3, 4, Article e1602614 pp. (2017)
[55] Schaeffer, Hayden, Learning Partial Differential Equations via Data Discovery and Sparse Optimization, Proc. R. Soc. A, vol. 473, 20160446 (2017), The Royal Society · Zbl 1404.35397
[56] Schaeffer, Hayden; McCalla, Scott G., Sparse model selection via integral terms, Phys. Rev. E, 96, 2, Article 023302 pp. (2017)
[57] Schaeffer, Hayden; Tran, Giang; Ward, Rachel, Extracting sparse high-dimensional dynamics from limited data (2017), arXiv preprint · Zbl 1405.62127
[58] Schaeffer, Hayden; Tran, Giang; Ward, Rachel; Zhang, Linan, Extracting structured dynamical systems using sparse optimization with very few samples (2018), arXiv preprint
[59] Schmid, P. J., Dynamic mode decomposition of numerical and experimental data, J. Fluid Mech., 656, 5-28 (August 2010) · Zbl 1197.76091
[60] Schmidt, Michael; Lipson, Hod, Distilling free-form natural laws from experimental data, Science, 324, 5923, 81-85 (2009)
[61] Schmidt, Michael; Vallabhajosyula, Ravishankar; Jenkins, J.; Hood, Jonathan; Soni, Abhishek; Wikswo, John; Lipson, Hod, Automated refinement and inference of analytical models for metabolic networks, Phys. Biol., 8, 5, Article 055011 pp. (2011)
[62] Srivastava, Nitish; Hinton, Geoffrey; Krizhevsky, Alex; Sutskever, Ilya; Salakhutdinov, Ruslan, Dropout: a simple way to prevent neural networks from overfitting, J. Mach. Learn. Res., 15, 1, 1929-1958 (2014) · Zbl 1318.68153
[63] Taira, K.; Colonius, T., The immersed boundary method: a projection approach, J. Comput. Phys., 225, 2, 2118-2137 (2007) · Zbl 1343.76027
[64] Takeishi, Naoya; Kawahara, Yoshinobu; Yairi, Takehisa, Learning Koopman invariant subspaces for dynamic mode decomposition, (Advances in Neural Information Processing Systems (2017)), 1130-1140
[65] Tran, Giang; Ward, Rachel, Exact recovery of chaotic systems from highly corrupted data, Multiscale Model. Simul., 15, 3, 1108-1129 (2017)
[66] Tu, J. H.; Rowley, C. W.; Luchtenburg, D. M.; Brunton, S. L.; Kutz, J. N., On dynamic mode decomposition: theory and applications, J. Comput. Dyn., 1, 2, 391-421 (2014) · Zbl 1346.37064
[67] Vlachas, Pantelis R.; Byeon, Wonmin; Wan, Zhong Y.; Sapsis, Themistoklis P.; Koumoutsakos, Petros, Data-driven forecasting of high-dimensional chaotic systems with long short-term memory networks, Proc. R. Soc. A, 474, 2213, Article 20170844 pp. (2018) · Zbl 1402.92030
[68] Wan, Zhong Yi; Vlachas, Pantelis; Koumoutsakos, Petros; Sapsis, Themistoklis, Data-assisted reduced-order modeling of extreme events in complex dynamical systems, PLoS ONE, 13, 5, Article e0197704 pp. (2018) · Zbl 1402.92030
[69] Wang, W. X.; Yang, R.; Lai, Y. C.; Kovanis, V.; Grebogi, C., Predicting catastrophes in nonlinear dynamical systems by compressive sensing, Phys. Rev. Lett., 106, Article 154101 pp. (2011)
[70] Wehmeyer, Christoph; Noé, Frank, Time-lagged autoencoders: deep learning of slow collective variables for molecular kinetics (2017), arXiv preprint
[71] Weinan, E., A proposal on machine learning via dynamical systems, Commun. Math. Stat., 5, 1, 1-11 (2017) · Zbl 1380.37154
[72] Yeung, Enoch; Kundu, Soumya; Hodas, Nathan, Learning deep neural network representations for Koopman operators of nonlinear dynamical systems (2017), arXiv preprint
[73] Yoshida, Haruo, Construction of higher order symplectic integrators, Phys. Lett. A, 150, 5-7, 262-268 (1990)
[74] Zhu, Ciyou; Byrd, Richard H.; Lu, Peihuang; Nocedal, Jorge, Algorithm 778: L-bfgs-b: Fortran subroutines for large-scale bound-constrained optimization, ACM Trans. Math. Softw., 23, 4, 550-560 (1997) · Zbl 0912.65057
This reference list is based on information provided by the publisher or from digital mathematics libraries. Its items are heuristically matched to zbMATH identifiers and may contain data conversion errors. In some cases that data have been complemented/enhanced by data from zbMATH Open. This attempts to reflect the references listed in the original paper as accurately as possible without claiming completeness or a perfect matching.