swMATH ID: 22205
Software Authors: Diederik P. Kingma, Jimmy Ba
Description: Adam: A Method for Stochastic Optimization. We introduce Adam, an algorithm for first-order gradient-based optimization of stochastic objective functions, based on adaptive estimates of lower-order moments. The method is straightforward to implement, is computationally efficient, has little memory requirements, is invariant to diagonal rescaling of the gradients, and is well suited for problems that are large in terms of data and/or parameters. The method is also appropriate for non-stationary objectives and problems with very noisy and/or sparse gradients. The hyper-parameters have intuitive interpretations and typically require little tuning. Some connections to related algorithms, on which Adam was inspired, are discussed. We also analyze the theoretical convergence properties of the algorithm and provide a regret bound on the convergence rate that is comparable to the best known results under the online convex optimization framework. Empirical results demonstrate that Adam works well in practice and compares favorably to other stochastic optimization methods. Finally, we discuss AdaMax, a variant of Adam based on the infinity norm.
Homepage: https://arxiv.org/abs/1412.6980
Keywords: Learning; arXiv cs.LG; arXiv; stochastic optimization; AdaMax; first-order gradient-based optimization
Related Software: TensorFlow; PyTorch; ImageNet; AlexNet; GitHub; DGM; Keras; AdaGrad; DiffSharp; U-Net; DeepXDE; Scikit; Python; L-BFGS; RMSprop; CIFAR; ADADELTA; MNIST; PRMLT; DeepONet
Cited in: 1,388 Documents
all top 5

Cited by 3,855 Authors

29 Karniadakis, George Em
15 Perdikaris, Paris G.
11 Han, Jiequn
9 E, Weinan
9 Hesthaven, Jan S.
9 Osher, Stanley Joel
9 Ray, Deep
8 Wang, Bao
7 Cai, Zhiqiang
7 Durlofsky, Louis J.
7 Jentzen, Arnulf
7 Koutsourelakis, Phaedon-Stelios
7 Liu, Min
7 Lu, Lu
7 Mishra, Siddhartha
7 Wang, Sifan
7 Warin, Xavier
7 Yang, Haizhao
7 Zabaras, Nicholas J.
6 Chen, Jingshuang
6 Haghighat, Ehsan
6 Sun, WaiChing
6 Xiu, Dongbin
5 Bengio, Yoshua
5 Doostan, Alireza
5 Efendiev, Yalchin R.
5 Germain, Maximilien
5 Huang, Jianguo
5 Lu, Jianfeng
5 Oosterlee, Cornelis Willebrordus
5 Peters, Jan
5 Wang, Jianxun
5 Ying, Lexing
4 Beck, Christian
4 Chung, Tsz Shun Eric
4 Dehghani, Hamidreza
4 Dong, Bin
4 Duraisamy, Karthik
4 Gao, Han
4 Gibou, Frédéric
4 Hao, Wenrui
4 He, Yang-Hui Evariste
4 Heinlein, Alexander
4 Jagtap, Ameya D.
4 Juanes, Ruben
4 Klawonn, Axel
4 Lee, Jaehong
4 Li, Qianxiao
4 Lin, Lin
4 Meng, Xuhui
4 Oseledets, Ivan Valer’evich
4 Prato, Marco
4 Raissi, Maziar
4 Rozza, Gianluigi
4 Situ, Haozhen
4 Sun, Wei
4 Tang, Yifa
4 Thürey, Nils
4 Trask, Nathaniel A.
4 Wan, Xiaoliang
4 Wang, Chunmei
4 Wang, Haoqin
4 Wang, Qian
4 Yan, Liang
4 Yang, Yibo
4 Zepeda-Núñez, Leonardo
4 Zhang, Dongxiao
4 Zhang, Linfeng
4 Zilian, Andreas
3 Adams, Nikolaus A.
3 Andersson, Kristoffer
3 Antonietti, Paola Francesca
3 Badías, Alberto
3 Becker, Sebastian
3 Beneddine, Samir
3 Bertozzi, Andrea Louise
3 Bezgin, Deniz A.
3 Bouklas, Nikolaos
3 Cai, Wei
3 Carlberg, Kevin T.
3 Chang, Haibin
3 Chen, Lei
3 Chen, Liwei
3 Chen, Wei
3 Cheridito, Patrick
3 Cheung, Siu Wun
3 Chinesta, Francisco
3 Cueto, Elias
3 Darbon, Jerome
3 De, Subhayan
3 Dede’, Luca
3 D’Elia, Marta
3 Dong, Suchuan
3 Elsheikh, Ahmed H.
3 Flores, Kevin B.
3 Fuhg, Jan Niklas
3 Fukagata, Koji
3 Fukami, Kai
3 Ghommem, Mehdi
3 Gu, Yiqi
...and 3,755 more Authors
all top 5

Cited in 248 Serials

168 Journal of Computational Physics
104 Computer Methods in Applied Mechanics and Engineering
56 Machine Learning
49 Journal of Machine Learning Research (JMLR)
35 SIAM Journal on Scientific Computing
31 Computational Mechanics
30 Neural Computation
30 Communications in Computational Physics
22 Journal of Scientific Computing
22 European Journal of Operational Research
20 Computers and Fluids
20 Journal of Fluid Mechanics
18 Computers & Mathematics with Applications
18 Data Mining and Knowledge Discovery
16 Artificial Intelligence
16 Journal of Computational and Applied Mathematics
16 Journal of Mathematical Imaging and Vision
16 Quantitative Finance
16 SIAM Journal on Imaging Sciences
15 International Journal of Computer Vision
14 The Journal of Artificial Intelligence Research (JAIR)
14 Computational Geosciences
13 Neural Networks
13 Quantum Information Processing
13 Statistics and Computing
11 Inverse Problems
11 Information Sciences
11 Inverse Problems and Imaging
11 SIAM Journal on Mathematics of Data Science
10 Physica D
9 Mathematical Biosciences and Engineering
8 Physica A
8 SIAM Journal on Optimization
8 European Journal of Mechanics. A. Solids
8 Communications in Nonlinear Science and Numerical Simulation
7 Chaos, Solitons and Fractals
7 Computers & Operations Research
7 Journal of High Energy Physics
7 International Journal of Computational Methods
7 Journal of Statistical Mechanics: Theory and Experiment
7 Research in the Mathematical Sciences
6 Computer Physics Communications
6 Automatica
6 Computer Aided Geometric Design
6 International Journal of Approximate Reasoning
6 Applied Mathematical Modelling
6 INFORMS Journal on Computing
6 Optimization Methods & Software
5 Physics Letters. A
5 Applied Mathematics and Computation
5 Journal of the American Statistical Association
5 Journal of Global Optimization
5 Computational Statistics and Data Analysis
5 Computational Optimization and Applications
5 IEEE Transactions on Image Processing
5 International Journal of Wavelets, Multiresolution and Information Processing
5 Mathematical Geosciences
5 Journal of the Operations Research Society of China
4 Physics Letters. B
4 SIAM Journal on Numerical Analysis
4 European Journal of Applied Mathematics
4 Computational Statistics
4 ETNA. Electronic Transactions on Numerical Analysis
4 SIAM Journal on Applied Dynamical Systems
4 SIAM Journal on Financial Mathematics
4 Communications in Mathematics and Statistics
3 Biological Cybernetics
3 Journal of the Franklin Institute
3 Physics Reports
3 Psychometrika
3 Journal of Econometrics
3 Journal of Optimization Theory and Applications
3 Mathematics and Computers in Simulation
3 Insurance Mathematics & Economics
3 Journal of Economic Dynamics & Control
3 Automation and Remote Control
3 Journal of Statistical Computation and Simulation
3 Mathematical Programming. Series A. Series B
3 Journal of Computer and Systems Sciences International
3 Journal of Mathematical Sciences (New York)
3 Computational and Applied Mathematics
3 Journal of Inverse and Ill-Posed Problems
3 Annals of Mathematics and Artificial Intelligence
3 European Journal of Control
3 Multibody System Dynamics
3 Lobachevskii Journal of Mathematics
3 Scandinavian Actuarial Journal
3 ASTIN Bulletin
3 Computational & Mathematical Methods in Medicine
3 Electronic Journal of Statistics
3 The Annals of Applied Statistics
3 Numerical Mathematics: Theory, Methods and Applications
3 Journal of Computational and Graphical Statistics
3 Journal of Theoretical Biology
3 East Asian Journal on Applied Mathematics
3 SIAM/ASA Journal on Uncertainty Quantification
3 Proceedings of the Royal Society of London. A. Mathematical, Physical and Engineering Sciences
3 Communications on Applied Mathematics and Computation
3 SN Partial Differential Equations and Applications
2 The American Statistician
...and 148 more Serials
all top 5

Cited in 52 Fields

873 Computer science (68-XX)
400 Numerical analysis (65-XX)
217 Statistics (62-XX)
163 Operations research, mathematical programming (90-XX)
154 Fluid mechanics (76-XX)
150 Partial differential equations (35-XX)
135 Biology and other natural sciences (92-XX)
111 Mechanics of deformable solids (74-XX)
89 Game theory, economics, finance, and other social and behavioral sciences (91-XX)
80 Probability theory and stochastic processes (60-XX)
69 Information and communication theory, circuits (94-XX)
52 Quantum theory (81-XX)
43 Systems theory; control (93-XX)
38 Statistical mechanics, structure of matter (82-XX)
34 Calculus of variations and optimal control; optimization (49-XX)
31 Geophysics (86-XX)
30 Dynamical systems and ergodic theory (37-XX)
17 Mechanics of particles and systems (70-XX)
14 Ordinary differential equations (34-XX)
10 Approximations and expansions (41-XX)
10 Operator theory (47-XX)
10 Optics, electromagnetic theory (78-XX)
8 Classical thermodynamics, heat transfer (80-XX)
7 Linear and multilinear algebra; matrix theory (15-XX)
7 Harmonic analysis on Euclidean spaces (42-XX)
7 Relativity and gravitational theory (83-XX)
6 Combinatorics (05-XX)
4 Differential geometry (53-XX)
3 General and overarching topics; collections (00-XX)
3 Mathematical logic and foundations (03-XX)
3 Number theory (11-XX)
3 Algebraic geometry (14-XX)
3 Real functions (26-XX)
3 Several complex variables and analytic spaces (32-XX)
3 Algebraic topology (55-XX)
3 Global analysis, analysis on manifolds (58-XX)
2 Integral transforms, operational calculus (44-XX)
2 Integral equations (45-XX)
2 General topology (54-XX)
2 Manifolds and cell complexes (57-XX)
1 Associative rings and algebras (16-XX)
1 Category theory; homological algebra (18-XX)
1 Group theory and generalizations (20-XX)
1 Measure and integration (28-XX)
1 Functions of a complex variable (30-XX)
1 Potential theory (31-XX)
1 Special functions (33-XX)
1 Difference and functional equations (39-XX)
1 Functional analysis (46-XX)
1 Geometry (51-XX)
1 Convex and discrete geometry (52-XX)
1 Astronomy and astrophysics (85-XX)

Citations by Year