Adam swMATH ID: 22205 Software Authors: Diederik P. Kingma, Jimmy Ba Description: Adam: A Method for Stochastic Optimization. We introduce Adam, an algorithm for first-order gradient-based optimization of stochastic objective functions, based on adaptive estimates of lower-order moments. The method is straightforward to implement, is computationally efficient, has little memory requirements, is invariant to diagonal rescaling of the gradients, and is well suited for problems that are large in terms of data and/or parameters. The method is also appropriate for non-stationary objectives and problems with very noisy and/or sparse gradients. The hyper-parameters have intuitive interpretations and typically require little tuning. Some connections to related algorithms, on which Adam was inspired, are discussed. We also analyze the theoretical convergence properties of the algorithm and provide a regret bound on the convergence rate that is comparable to the best known results under the online convex optimization framework. Empirical results demonstrate that Adam works well in practice and compares favorably to other stochastic optimization methods. Finally, we discuss AdaMax, a variant of Adam based on the infinity norm. Homepage: https://arxiv.org/abs/1412.6980 Keywords: Learning; arXiv cs.LG; arXiv; stochastic optimization; AdaMax; first-order gradient-based optimization Related Software: TensorFlow; PyTorch; ImageNet; AlexNet; GitHub; DGM; Keras; AdaGrad; DiffSharp; U-Net; DeepXDE; Scikit; Python; L-BFGS; RMSprop; CIFAR; ADADELTA; MNIST; PRMLT; DeepONet Cited in: 1,388 Documents all top 5 Cited by 3,855 Authors 29 Karniadakis, George Em 15 Perdikaris, Paris G. 11 Han, Jiequn 9 E, Weinan 9 Hesthaven, Jan S. 9 Osher, Stanley Joel 9 Ray, Deep 8 Wang, Bao 7 Cai, Zhiqiang 7 Durlofsky, Louis J. 7 Jentzen, Arnulf 7 Koutsourelakis, Phaedon-Stelios 7 Liu, Min 7 Lu, Lu 7 Mishra, Siddhartha 7 Wang, Sifan 7 Warin, Xavier 7 Yang, Haizhao 7 Zabaras, Nicholas J. 6 Chen, Jingshuang 6 Haghighat, Ehsan 6 Sun, WaiChing 6 Xiu, Dongbin 5 Bengio, Yoshua 5 Doostan, Alireza 5 Efendiev, Yalchin R. 5 Germain, Maximilien 5 Huang, Jianguo 5 Lu, Jianfeng 5 Oosterlee, Cornelis Willebrordus 5 Peters, Jan 5 Wang, Jianxun 5 Ying, Lexing 4 Beck, Christian 4 Chung, Tsz Shun Eric 4 Dehghani, Hamidreza 4 Dong, Bin 4 Duraisamy, Karthik 4 Gao, Han 4 Gibou, Frédéric 4 Hao, Wenrui 4 He, Yang-Hui Evariste 4 Heinlein, Alexander 4 Jagtap, Ameya D. 4 Juanes, Ruben 4 Klawonn, Axel 4 Lee, Jaehong 4 Li, Qianxiao 4 Lin, Lin 4 Meng, Xuhui 4 Oseledets, Ivan Valer’evich 4 Prato, Marco 4 Raissi, Maziar 4 Rozza, Gianluigi 4 Situ, Haozhen 4 Sun, Wei 4 Tang, Yifa 4 Thürey, Nils 4 Trask, Nathaniel A. 4 Wan, Xiaoliang 4 Wang, Chunmei 4 Wang, Haoqin 4 Wang, Qian 4 Yan, Liang 4 Yang, Yibo 4 Zepeda-Núñez, Leonardo 4 Zhang, Dongxiao 4 Zhang, Linfeng 4 Zilian, Andreas 3 Adams, Nikolaus A. 3 Andersson, Kristoffer 3 Antonietti, Paola Francesca 3 Badías, Alberto 3 Becker, Sebastian 3 Beneddine, Samir 3 Bertozzi, Andrea Louise 3 Bezgin, Deniz A. 3 Bouklas, Nikolaos 3 Cai, Wei 3 Carlberg, Kevin T. 3 Chang, Haibin 3 Chen, Lei 3 Chen, Liwei 3 Chen, Wei 3 Cheridito, Patrick 3 Cheung, Siu Wun 3 Chinesta, Francisco 3 Cueto, Elias 3 Darbon, Jerome 3 De, Subhayan 3 Dede’, Luca 3 D’Elia, Marta 3 Dong, Suchuan 3 Elsheikh, Ahmed H. 3 Flores, Kevin B. 3 Fuhg, Jan Niklas 3 Fukagata, Koji 3 Fukami, Kai 3 Ghommem, Mehdi 3 Gu, Yiqi ...and 3,755 more Authors all top 5 Cited in 248 Serials 168 Journal of Computational Physics 104 Computer Methods in Applied Mechanics and Engineering 56 Machine Learning 49 Journal of Machine Learning Research (JMLR) 35 SIAM Journal on Scientific Computing 31 Computational Mechanics 30 Neural Computation 30 Communications in Computational Physics 22 Journal of Scientific Computing 22 European Journal of Operational Research 20 Computers and Fluids 20 Journal of Fluid Mechanics 18 Computers & Mathematics with Applications 18 Data Mining and Knowledge Discovery 16 Artificial Intelligence 16 Journal of Computational and Applied Mathematics 16 Journal of Mathematical Imaging and Vision 16 Quantitative Finance 16 SIAM Journal on Imaging Sciences 15 International Journal of Computer Vision 14 The Journal of Artificial Intelligence Research (JAIR) 14 Computational Geosciences 13 Neural Networks 13 Quantum Information Processing 13 Statistics and Computing 11 Inverse Problems 11 Information Sciences 11 Inverse Problems and Imaging 11 SIAM Journal on Mathematics of Data Science 10 Physica D 9 Mathematical Biosciences and Engineering 8 Physica A 8 SIAM Journal on Optimization 8 European Journal of Mechanics. A. Solids 8 Communications in Nonlinear Science and Numerical Simulation 7 Chaos, Solitons and Fractals 7 Computers & Operations Research 7 Journal of High Energy Physics 7 International Journal of Computational Methods 7 Journal of Statistical Mechanics: Theory and Experiment 7 Research in the Mathematical Sciences 6 Computer Physics Communications 6 Automatica 6 Computer Aided Geometric Design 6 International Journal of Approximate Reasoning 6 Applied Mathematical Modelling 6 INFORMS Journal on Computing 6 Optimization Methods & Software 5 Physics Letters. A 5 Applied Mathematics and Computation 5 Journal of the American Statistical Association 5 Journal of Global Optimization 5 Computational Statistics and Data Analysis 5 Computational Optimization and Applications 5 IEEE Transactions on Image Processing 5 International Journal of Wavelets, Multiresolution and Information Processing 5 Mathematical Geosciences 5 Journal of the Operations Research Society of China 4 Physics Letters. B 4 SIAM Journal on Numerical Analysis 4 European Journal of Applied Mathematics 4 Computational Statistics 4 ETNA. Electronic Transactions on Numerical Analysis 4 SIAM Journal on Applied Dynamical Systems 4 SIAM Journal on Financial Mathematics 4 Communications in Mathematics and Statistics 3 Biological Cybernetics 3 Journal of the Franklin Institute 3 Physics Reports 3 Psychometrika 3 Journal of Econometrics 3 Journal of Optimization Theory and Applications 3 Mathematics and Computers in Simulation 3 Insurance Mathematics & Economics 3 Journal of Economic Dynamics & Control 3 Automation and Remote Control 3 Journal of Statistical Computation and Simulation 3 Mathematical Programming. Series A. Series B 3 Journal of Computer and Systems Sciences International 3 Journal of Mathematical Sciences (New York) 3 Computational and Applied Mathematics 3 Journal of Inverse and Ill-Posed Problems 3 Annals of Mathematics and Artificial Intelligence 3 European Journal of Control 3 Multibody System Dynamics 3 Lobachevskii Journal of Mathematics 3 Scandinavian Actuarial Journal 3 ASTIN Bulletin 3 Computational & Mathematical Methods in Medicine 3 Electronic Journal of Statistics 3 The Annals of Applied Statistics 3 Numerical Mathematics: Theory, Methods and Applications 3 Journal of Computational and Graphical Statistics 3 Journal of Theoretical Biology 3 East Asian Journal on Applied Mathematics 3 SIAM/ASA Journal on Uncertainty Quantification 3 Proceedings of the Royal Society of London. A. Mathematical, Physical and Engineering Sciences 3 Communications on Applied Mathematics and Computation 3 SN Partial Differential Equations and Applications 2 The American Statistician ...and 148 more Serials all top 5 Cited in 52 Fields 873 Computer science (68-XX) 400 Numerical analysis (65-XX) 217 Statistics (62-XX) 163 Operations research, mathematical programming (90-XX) 154 Fluid mechanics (76-XX) 150 Partial differential equations (35-XX) 135 Biology and other natural sciences (92-XX) 111 Mechanics of deformable solids (74-XX) 89 Game theory, economics, finance, and other social and behavioral sciences (91-XX) 80 Probability theory and stochastic processes (60-XX) 69 Information and communication theory, circuits (94-XX) 52 Quantum theory (81-XX) 43 Systems theory; control (93-XX) 38 Statistical mechanics, structure of matter (82-XX) 34 Calculus of variations and optimal control; optimization (49-XX) 31 Geophysics (86-XX) 30 Dynamical systems and ergodic theory (37-XX) 17 Mechanics of particles and systems (70-XX) 14 Ordinary differential equations (34-XX) 10 Approximations and expansions (41-XX) 10 Operator theory (47-XX) 10 Optics, electromagnetic theory (78-XX) 8 Classical thermodynamics, heat transfer (80-XX) 7 Linear and multilinear algebra; matrix theory (15-XX) 7 Harmonic analysis on Euclidean spaces (42-XX) 7 Relativity and gravitational theory (83-XX) 6 Combinatorics (05-XX) 4 Differential geometry (53-XX) 3 General and overarching topics; collections (00-XX) 3 Mathematical logic and foundations (03-XX) 3 Number theory (11-XX) 3 Algebraic geometry (14-XX) 3 Real functions (26-XX) 3 Several complex variables and analytic spaces (32-XX) 3 Algebraic topology (55-XX) 3 Global analysis, analysis on manifolds (58-XX) 2 Integral transforms, operational calculus (44-XX) 2 Integral equations (45-XX) 2 General topology (54-XX) 2 Manifolds and cell complexes (57-XX) 1 Associative rings and algebras (16-XX) 1 Category theory; homological algebra (18-XX) 1 Group theory and generalizations (20-XX) 1 Measure and integration (28-XX) 1 Functions of a complex variable (30-XX) 1 Potential theory (31-XX) 1 Special functions (33-XX) 1 Difference and functional equations (39-XX) 1 Functional analysis (46-XX) 1 Geometry (51-XX) 1 Convex and discrete geometry (52-XX) 1 Astronomy and astrophysics (85-XX) Citations by Year