×

Adaptive two-layer ReLU neural network. I: Best least-squares approximation. (English) Zbl 1504.65275

Summary: In this paper, we introduce adaptive network enhancement (ANE) method for the best least-squares approximation using two-layer ReLU neural networks (NNs). For a given function \(f(\mathbf{x})\), the ANE method generates a two-layer ReLU NN and a numerical integration mesh such that the approximation accuracy is within the prescribed tolerance. The ANE method provides a natural process for obtaining a good initialization which is crucial for training nonlinear optimization problems. Numerical results for functions of two variables exhibiting either intersecting interface singularities or sharp interior layers demonstrate efficiency of the ANE method.

MSC:

65N99 Numerical methods for partial differential equations, boundary value problems
68T07 Artificial neural networks and deep learning

Software:

Adam; DGM
PDFBibTeX XMLCite
Full Text: DOI arXiv

References:

[1] Hebb, D. O., The Organization of Behavior: A Neuropsychological Theory (1949), Wiley: Wiley New York
[2] Rosenblatt, F., The perceptron: a probabilistic model for information storage and organization in the brain, Psychol. Rev., 65, 6, 386-408 (1958)
[3] Cybenko, G., Approximation by superpositions of a sigmoidal function, Math. Control Signals Syst., 2, 303-314 (1989) · Zbl 0679.94019
[4] Hornik, K.; Stinchcombe, M.; White, H., Multilayer feedforward networks are universal approximators, Neural Netw., 2, 359-366 (1989) · Zbl 1383.92015
[5] Petrushev, P. P., Approximation by ridge functions and neural networks, SIAM J. Math. Anal., 30, 155-189 (1998) · Zbl 0927.41006
[6] Pinkus, A., Approximation theory of the mlp model in neural networks, Acta Numer., 8, 143-195 (1999) · Zbl 0959.68109
[7] Liu, M.; Cai, Z., Adaptive two-layer ReLU neural network: II. Ritz approximation to elliptic PDEs, Comput. Math. Appl., 113, 103-116 (2022) · Zbl 1524.65939
[8] Dick, J.; Kuo, F.; Sloan, I., High-dimensional integration - the quasi-Monte Carlo way, Acta Numer., 15, 133-288 (2014) · Zbl 1296.65004
[9] Bungartz, H. J.; Griebel, M., Sparse grids, Acta Numer., 13, 1-123 (2004)
[10] Bottou, L.; Curtis, F. E.; Nocedal, J., Optimization methods for large-scale machine learning, SIAM Rev., 60, 223-311 (2018) · Zbl 1397.65085
[11] Schumaker, L., Spline Functions: Basic Theory (1981) · Zbl 0449.41004
[12] Jupp, D., Approximation to data by splines with free knots, SIAM J. Numer. Anal., 15, 6, 328-343 (1978) · Zbl 0403.65004
[13] Baker, A. J., On optimization aspects of a cfd finite element penalty algorithm, (Whiteman, J. R., The Mathematics of Finite Element and Applications V (1985)), 391-414 · Zbl 0575.76071
[14] Rice, J. R., The Approximation of Functions, vol. 2 (1969), Addison-Wesley: Addison-Wesley MA · Zbl 0185.30601
[15] Powell, M., On best l 2 spline approximations, Num. Math. Differ. Approx., 317-339 (1968) · Zbl 0176.14102
[16] Chui, C. K.; Smith, P. W.; Ward, J. D., On the smoothness of best L2 approximants from nonlinear spline manifolds, Math. Comput., 31, 17-23 (1977) · Zbl 0355.41028
[17] Dissanayake, M.; Phan-Thien, N., Neural network based approximations for solving partial differential equations, Commun. Numer. Methods Eng., 10, 3, 195-201 (1994) · Zbl 0802.65102
[18] Sirignano, J.; Spiliopoulos, K., DGM: a deep learning algorithm for solving partial differential equations, J. Comput. Phys., 375, 1139-1364 (2018) · Zbl 1416.65394
[19] Raissia, M.; Perdikarisb, P.; Karniadakisa, G., Physics-informed neural networks: a deep learning framework for solving forward and inverse problems involving nonlinear partial differential equations, J. Comput. Phys., 378, 686-707 (2019) · Zbl 1415.68175
[20] Cai, Z.; Chen, J.; Liu, M.; Liu, X., Deep least-squares methods: an unsupervised learning-based numerical method for solving elliptic pdes, J. Comput. Phys., 420, Article 109707 pp. (2020) · Zbl 07506625
[21] Temlyakov, V. N., The Marcinkiewicz-type discretization theorems, Constr. Approx., 48, 337-369 (2018) · Zbl 1404.41012
[22] Pearson, K., On lines and planes of closest fit to systems of points in space, The London, Edinburgh, and Dublin Philos. Mag. J. Sci., 2, 11, 559-572 (1901) · JFM 32.0246.07
[23] Kingma, D. P.; Ba Adam, J., A method for stochastic optimization (2014), arXiv preprint
[24] He, J.; Li, L.; Xu, J.; Zheng, C., Relu deep neural networks and linear finite elements, J. Comput. Math., 38, 3, 502-527 (2020) · Zbl 1463.68072
[25] Morin, P.; Nochetto, R. H.; Siebert, K. G., Convergence of adaptive finite element methods, SIAM Rev., 44, 4, 631-658 (2002) · Zbl 1016.65074
[26] Cai, Z.; Zhang, S., Recovery-based error estimator for interface problems: conforming linear elements, SIAM J. Numer. Anal., 47, 3, 2132-2156 (2009) · Zbl 1204.65129
[27] Cai, Z.; Chen, J.; Liu, M., Self-adaptive deep neural network: numerical approximation to functions and PDEs (2021)
[28] Allgower, E.; Georg, K., Numerical Continuation Methods (1990), Springer-Verlag: Springer-Verlag Berlin and Heidelberg · Zbl 0717.65030
This reference list is based on information provided by the publisher or from digital mathematics libraries. Its items are heuristically matched to zbMATH identifiers and may contain data conversion errors. In some cases that data have been complemented/enhanced by data from zbMATH Open. This attempts to reflect the references listed in the original paper as accurately as possible without claiming completeness or a perfect matching.