×

Deep neural networks can stably solve high-dimensional, noisy, non-linear inverse problems. (English) Zbl 1511.41010

Authors’ abstract: We study the problem of reconstructing solutions of inverse problems when only noisy measurements are available. We assume that the problem can be modeled with an infinite-dimensional forward operator that is not continuously invertible. Then, we restrict this forward operator to finite-dimensional spaces so that the inverse is Lipschitz continuous. For the inverse operator, we demonstrate that there exists a neural network which is a robust-to-noise approximation of the operator. In addition, we show that these neural networks can be learned from appropriately perturbed training data. We demonstrate the admissibility of this approach to a wide range of inverse problems of practical interest. Numerical examples are given that support the theoretical findings.

MSC:

41A27 Inverse theorems in approximation theory
41A25 Rate of convergence, degree of approximation
41A30 Approximation by other special function classes
68T05 Learning and adaptive systems in artificial intelligence
PDFBibTeX XMLCite
Full Text: DOI arXiv

References:

[1] Alberti, G. S. and Santacesaria, M., Infinite-dimensional inverse problems with finite measurements, Arch. Ration. Mech. Anal.243(1) (2022) 1-31. · Zbl 1481.35390
[2] G. S. Alberti, M. Santacesaria and S. Sciutto, Continuous generative neural networks, preprint (2022), arXiv:2205.14627.
[3] Amodei, D., Ananthanarayanan, S., Anubhai, R., Bai, J., Battenberg, E., Case, C., Casper, J., Catanzaro, B., Cheng, Q. and Chen, G., Deep speech 2: End-to-end speech recognition in english and mandarin, in Int. Conf. Machine Learning (PMLR, 2016), pp. 173-182.
[4] Anthony, M. and Bartlett, P. L., Neural Network Learning: Theoretical Foundations (Cambridge University Press, Cambridge, 1999). · Zbl 0968.68126
[5] V. Antun, F. Renna, C. Poon, B. Adcock and A. C. Hansen, On instabilities of deep learning in image reconstruction-does AI come at a cost? preprint (2019), arXiv:1902.05300.
[6] Armanious, K., Jiang, C., Abdulatif, S., Küstner, T., Gatidis, S. and Yang, B., Unsupervised medical image translation using cycle-MedGAN, in 2019 27th Eur. Signal Processing Conf. (EUSIPCO) (IEEE, 2019), pp. 1-5.
[7] Arridge, S., Maass, P., Öktem, O. and Schönlieb, C. B., Solving inverse problems using data-driven models, Acta Numer.28 (2019) 1-174. · Zbl 1429.65116
[8] Bacchelli, V. and Vessella, S., Lipschitz stability for a stationary 2d inverse problem with unknown polygonal boundary, Inverse Probl.22(5) (2006) 1627. · Zbl 1106.35128
[9] D. Bahdanau, K. Cho and Y. Bengio, Neural machine translation by jointly learning to align and translate, preprint (2014), arXiv:1409.0473.
[10] J. Berner, P. Grohs, G. Kutyniok and P. Petersen, The modern mathematics of deep learning, preprint (2021), arXiv:2105.04026. · Zbl 07571594
[11] Bickel, P. J. and Li, B., Local polynomial regression on unknown manifolds, in Complex Datasets and Inverse Problems, Vol. 54 (Institute of Mathematical Statistics, 2007), pp. 177-187.
[12] H. Boche, A. Fono and G. Kutyniok, Limitations of deep learning for inverse problems on digital hardware, preprint (2022), arXiv:2202.13490.
[13] Boucheron, S., Lugosi, G. and Massart, P., Concentration Inequalities: A Nonasymptotic Theory of Independence (Oxford University Press, 2013). · Zbl 1279.60005
[14] Boyd, S., Parikh, N., Chu, E., Peleato, B. and Eckstein, J., Distributed optimization and statistical learning via the alternating direction method of multipliers, Found. Trends Mach. Learn.3(1) (2011) 1-122. · Zbl 1229.90122
[15] Brunton, S. L. and Kutz, J. N., Data-Driven Science and Engineering: Machine Learning, Dynamical Systems, and Control (Cambridge University Press, 2022). · Zbl 1487.68001
[16] M. Chen, H. Jiang, W. Liao and T. Zhao, Nonparametric regression on low-dimensional manifolds using deep relu networks, preprint (2019), arXiv:1908.01842.
[17] Chui, C. K. and Mhaskar, H., Deep nets for local manifold learning, Front. Appl. Math. Stat.4 (2018) 12.
[18] Cloninger, A. and Klock, T., A deep network construction that adapts to intrinsic dimensionality beyond the domain, Neural Netw.141 (2021) 404-419. · Zbl 07743453
[19] Colbrook, M. J., Antun, V. and Hansen, A. C., The difficulty of computing stable and accurate neural networks: On the barriers of deep learning and smale’s 18th problem, Proc. Natl. Acad. Sci. USA119(12) (2022) e2107151119.
[20] Dabov, K., Foi, A., Katkovnik, V. and Egiazarian, K., Image denoising by sparse 3-D transform-domain collaborative filtering, IEEE Trans. Image Process.16(8) (2007) 2080-2095.
[21] D. Elbrächter, P. Grohs, A. Jentzen and C. Schwab, DNN expression rate analysis of high-dimensional PDEs: Application to option pricing, preprint (2018), arXiv:1809.07669. · Zbl 1500.35009
[22] Engl, H. W., Hanke, M. and Neubauer, A., Regularization of Inverse Problems, Vol. 375 (Springer Science & Business Media, 1996). · Zbl 0859.65054
[23] Genzel, M., Macdonald, J. and März, M., Solving inverse problems with deep neural networks-robustness included, IEEE Trans. Pattern Anal. Mach. Intell.45(1) (2022) 1119-1134.
[24] Goodfellow, I., Bengio, Y. and Courville, A., Deep Learning (MIT Press, 2016), http://www.deeplearningbook.org. · Zbl 1373.68009
[25] N. M. Gottschling, V. Antun, B. Adcock and A. C. Hansen, The troublesome kernel: Why deep learning for inverse problems is typically unstable, preprint (2020), arXiv:2001.01258.
[26] Graybill, F. A. and Marsaglia, G., Idempotent matrices and quadratic forms in the general linear hypothesis, Ann. Math. Stat.28(3) (1957) 678-686. · Zbl 0080.35502
[27] Gregor, K. and LeCun, Y., Learning fast approximations of sparse coding, in Proc. 27th Int. Conf. Machine Learning (Omnipress, 2010), pp. 399-406.
[28] P. Grohs and F. Voigtlaender, Proof of the theory-to-practice gap in deep learning via sampling complexity bounds for neural network approximation spaces, preprint (2021), arXiv:2104.02746.
[29] I. Gühring, M. Raslan and G. Kutyniok, Expressivity of deep neural networks, preprint (2020), arXiv:2007.04759.
[30] Han, J., Jentzen, A. and E, W., Solving high-dimensional partial differential equations using deep learning, Proc. Natl. Acad. Sci. USA115(34) (2018) 8505-8510. · Zbl 1416.35137
[31] J. He, L. Li, J. Xu and C. Zheng, ReLU deep neural networks and linear finite elements, preprint (2018), arXiv:1807.03973. · Zbl 1463.68072
[32] He, K., Zhang, X., Ren, S. and Sun, J., Deep residual learning for image recognition, in Proc. IEEE Conf. Computer Vision and Pattern Recognition (IEEE, 2016), pp. 770-778.
[33] Hein, M., Audibert, J.-Y. and Luxburg, U. V., From graphs to manifolds-weak and strong pointwise consistency of graph Laplacians, in Int. Conf. Computational Learning Theory (Springer, 2005), pp. 470-485. · Zbl 1095.68097
[34] Hu, J., Shen, L. and Sun, G., Squeeze-and-excitation networks, in 2018 IEEE/CVF Conf. Computer Vision and Pattern Recognition (IEEE, 2018), pp. 7132-7141.
[35] Huang, Y., Würfl, T., Breininger, K., Liu, L., Lauritsch, G. and Maier, A., Some investigations on robustness of deep learning in limited angle tomography, in Int. Conf. Medical Image Computing and Computer-Assisted Intervention (Springer, 2018), pp. 145-153.
[36] Kaltenbacher, B., Neubauer, A. and Scherzer, O., Iterative Regularization Methods for Nonlinear Ill-Posed Problems (de Gruyter, 2008). · Zbl 1145.65037
[37] Kirsch, A., An Introduction to the Mathematical Theory of Inverse Problems, Vol. 120 (Springer, 2011). · Zbl 1213.35004
[38] M. Kohler, A. Krzyzak and S. Langer, Estimation of a function of low local dimensionality by deep neural networks, preprint (2019), arXiv:1908.11140. · Zbl 1505.62504
[39] Kontak, M. and Michel, V., A greedy algorithm for nonlinear inverse problems with an application to nonlinear inverse gravimetry, GEM Int. J. Geomath.9(2) (2018) 167-198. · Zbl 1415.65132
[40] Krizhevsky, A., Sutskever, I. and Hinton, G. E., Imagenet classification with deep convolutional neural networks, in Adv. Neural Inf. Process. Syst., Vol. 25 (Curran Associates, 2012), pp. 1097-1105.
[41] LeCun, Y., Bengio, Y. and Hinton, G., Deep learning, Nature521(7553) (2015) 436-444.
[42] Lee, J. M., Smooth manifolds, in Introduction to Smooth Manifolds (Springer, 2013), pp. 1-31. · Zbl 1258.53002
[43] Li, Q., Lin, T. and Shen, Z., Deep learning via dynamical systems: An approximation perspective, J. Eur. Math. Soc. (2022). https://doi.org/10.4171/JEMS/1221. · Zbl 1528.41106
[44] Marinov, T. T. and Vatsala, A. S., Inverse problem for coefficient identification in the Euler-Bernoulli equation, Comput. Math. Appl.56(2) (2008) 400-410. · Zbl 1155.34356
[45] McCulloch, W. and Pitts, W., A logical calculus of ideas immanent in nervous activity, Bull. Math. Biophys.5 (1943) 115-133. · Zbl 0063.03860
[46] McShane, E. J., Extension of range of functions, Bull. Am. Math. Soc.40(12) (1934) 837-842. · JFM 60.0973.02
[47] Mhaskar, H. N., Eignets for function approximation on manifolds, Appl. Comput. Harmon. Anal.29(1) (2010) 63-87. · Zbl 1201.41003
[48] Miller, R. K., Volterra integral equations in a Banach space, Funkcial. Ekvac.18(2) (1975) 163-193. · Zbl 0326.45007
[49] Munkres, J. R., Topology, Vol. 2 (Prentice Hall, Upper Saddle River, 2000). · Zbl 0951.54001
[50] Nakada, R. and Imaizumi, M., Adaptive approximation and generalization of deep neural network with intrinsic dimensionality, J. Mach. Learn. Res.21(174) (2020) 1-38. · Zbl 1525.68135
[51] Natterer, F., Regularisierung schlecht gestellter Probleme durch Projektionsverfahren, Numer. Math.28(3) (1977) 329-341. · Zbl 0364.65042
[52] Natterer, F., The Mathematics of Computerized Tomography (SIAM, 2001). · Zbl 0973.92020
[53] Natterer, F. and Wübbeling, F., Mathematical Methods in Image Reconstruction (SIAM, 2001). · Zbl 0974.92016
[54] Niwas, S. and Singhal, D. C., Aquifer transmissivity of porous media from resistivity data, J. Hydrol.82(1-2) (1985) 143-153.
[55] Ongie, G., Jalal, A., Metzler, C. A, Baraniuk, R. G., Dimakis, A. G. and Willett, R., Deep learning techniques for inverse problems in imaging, IEEE J. Sel. Areas Inf. Theory1(1) (2020) 39-56.
[56] Opschoor, J. A. A., Petersen, P. C. and Schwab, Ch., Deep ReLU networks and high-order finite element methods, Anal. Appl.18(05) (2020) 715-770. · Zbl 1452.65354
[57] Petersen, P. C., Neural network theory, University of Vienna (2020).
[58] Petersen, P. C. and Voigtlaender, F., Optimal approximation of piecewise smooth functions using deep ReLU neural networks, Neural Netw.180 (2018) 296-330. · Zbl 1434.68516
[59] Plato, R. and Vainikko, G., On the regularization of projection methods for solving ill-posed problems, Numer. Math.57(1) (1990) 63-79. · Zbl 0675.65053
[60] Raissi, M., Perdikaris, P. and Karniadakis, G. E., Physics-informed neural networks: A deep learning framework for solving forward and inverse problems involving nonlinear partial differential equations, J. Comput. Phys.378 (2019) 686-707. · Zbl 1415.68175
[61] Razzaghi, M. and Ordokhani, Y., Solution of nonlinear Volterra-Hammerstein integral equations via rationalized Haar functions, Math. Probl. Eng.7(2) (2001) 205-219. · Zbl 0990.65152
[62] Rondi, L., A remark on a paper by Alessandrini and Vessella, Adv. Appl. Math.36(1) (2006) 67-69. · Zbl 1158.35105
[63] J. Schmidt-Hieber, Deep ReLU network approximation of functions on a manifold, preprint (2019), arXiv:1908.00695.
[64] Sepehrian, B. and Razzaghi, M., Solution of nonlinear Volterra-Hammerstein integral equations via single-term Walsh series method, Math. Probl. Eng.2005(5) (2005) 547-554. · Zbl 1200.65114
[65] Shaham, U., Cloninger, A. and Coifman, R. R., Provable approximation properties for deep neural networks, Appl. Comput. Harmon. Anal.44(3) (2018) 537-557. · Zbl 1390.68553
[66] Singer, A., From graph to manifold Laplacian: The convergence rate, Appl. Comput. Harmon. Anal.21(1) (2006) 128-134. · Zbl 1095.68102
[67] Sirignano, J. and Spiliopoulos, K., Dgm: A deep learning algorithm for solving partial differential equations, J. Comput. Phys.375 (2018) 1339-1364. · Zbl 1416.65394
[68] Sun, J., Li, H. and Xu, Z., Deep admm-net for compressive sensing MRI, in Adv. Neural Inf. Process. Syst., Vol. 29 (Curran Associates, 2016), pp. 10-18.
[69] Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, Ł. and Polosukhin, I., Attention is all you need, in Adv. Neural Inf. Process. Syst., Vol. 30 (Curran Associates, 2017), pp. 5998-6008.
[70] Venkatakrishnan, S. V., Bouman, C. A. and Wohlberg, B., Plug-and-play priors for model based reconstruction, in 2013 IEEE Global Conf. Signal and Information Processing (IEEE, 2013), pp. 945-948.
[71] Yarotsky, D., Error bounds for approximations with deep ReLU networks, Neural Netw.94 (2017) 103-114. · Zbl 1429.68260
[72] Yeh, W. W.-G., Review of parameter identification procedures in groundwater hydrology: The inverse problem, Water Resour. Res.22(2) (1986) 95-108.
[73] Zhu, B., Liu, J. Z, Cauley, S. F, Rosen, B. R. and Rosen, M. S., Image reconstruction by domain-transform manifold learning, Nature555(7697) (2018) 487-492.
[74] Zhu, J. Y., Park, T., Isola, P. and Efros, A. A., Unpaired image-to-image translation using cycle-consistent adversarial networks, in Proc. IEEE Int. Conf. Computer Vision (IEEE, 2017), pp. 2223-2232.
This reference list is based on information provided by the publisher or from digital mathematics libraries. Its items are heuristically matched to zbMATH identifiers and may contain data conversion errors. In some cases that data have been complemented/enhanced by data from zbMATH Open. This attempts to reflect the references listed in the original paper as accurately as possible without claiming completeness or a perfect matching.