swMATH ID: 22205
Software Authors: Diederik P. Kingma, Jimmy Ba
Description: Adam: A Method for Stochastic Optimization. We introduce Adam, an algorithm for first-order gradient-based optimization of stochastic objective functions, based on adaptive estimates of lower-order moments. The method is straightforward to implement, is computationally efficient, has little memory requirements, is invariant to diagonal rescaling of the gradients, and is well suited for problems that are large in terms of data and/or parameters. The method is also appropriate for non-stationary objectives and problems with very noisy and/or sparse gradients. The hyper-parameters have intuitive interpretations and typically require little tuning. Some connections to related algorithms, on which Adam was inspired, are discussed. We also analyze the theoretical convergence properties of the algorithm and provide a regret bound on the convergence rate that is comparable to the best known results under the online convex optimization framework. Empirical results demonstrate that Adam works well in practice and compares favorably to other stochastic optimization methods. Finally, we discuss AdaMax, a variant of Adam based on the infinity norm.
Homepage: https://arxiv.org/abs/1412.6980
Keywords: Learning; arXiv cs.LG; arXiv; stochastic optimization; AdaMax; first-order gradient-based optimization
Related Software: TensorFlow; PyTorch; ImageNet; AlexNet; GitHub; DGM; AdaGrad; Keras; DiffSharp; U-Net; DeepXDE; CIFAR; L-BFGS; DeepONet; RMSprop; Scikit; PRMLT; Tensor2Tensor; Python; ADADELTA
Cited in: 1,935 Documents
all top 5

Cited by 5,308 Authors

38 Karniadakis, George Em
16 Perdikaris, Paris G.
12 Ray, Deep
11 Han, Jiequn
9 E, Weinan
9 Hesthaven, Jan S.
9 Osher, Stanley Joel
8 Cai, Zhiqiang
8 Durlofsky, Louis J.
8 Jentzen, Arnulf
8 Liu, Min
8 Lu, Lu
8 Mishra, Siddhartha
8 Wang, Bao
8 Wang, Sifan
8 Yang, Haizhao
7 Koutsourelakis, Phaedon-Stelios
7 Wang, Jianxun
7 Warin, Xavier
7 Zabaras, Nicholas J.
6 Chen, Jingshuang
6 Doostan, Alireza
6 Haghighat, Ehsan
6 Huang, Jianguo
6 Lu, Jianfeng
6 Meng, Xuhui
6 Pardo, David
6 Rozza, Gianluigi
6 Sun, WaiChing
6 Xiu, Dongbin
5 Bengio, Yoshua
5 Darbon, Jerome
5 Efendiev, Yalchin R.
5 Germain, Maximilien
5 Gibou, Frédéric
5 Hao, Wenrui
5 He, Yang-Hui Evariste
5 Jagtap, Ameya D.
5 Klawonn, Axel
5 Li, Qianxiao
5 Manzoni, Andrea
5 Oosterlee, Cornelis Willebrordus
5 Peters, Jan
5 Situ, Haozhen
5 Trask, Nathaniel A.
5 Wan, Xiaoliang
5 Yan, Liang
5 Yan, Zhenya
5 Ying, Lexing
4 Antonietti, Paola Francesca
4 Beck, Christian
4 Becker, Sebastian
4 Cai, Wei
4 Chung, Tsz Shun Eric
4 Dehghani, Hamidreza
4 Dong, Bin
4 Dong, Suchuan
4 Duraisamy, Karthik
4 Gao, Han
4 Gasnikov, Aleksandr V.
4 Ghommem, Mehdi
4 Gong, Maoguo
4 Gu, Yiqi
4 Heinlein, Alexander
4 Hertrich, Johannes
4 Hu, Xiaozhe
4 Iwata, Tomoharu
4 Juanes, Ruben
4 Kawaguchi, Kenji
4 Lanser, Martin
4 Lee, Jaehong
4 Li, Yu
4 Lin, Guang
4 Lin, Lin
4 Liu, Wing Kam
4 Liu, Yang
4 Maulik, Romit
4 Oseledets, Ivan Valer’evich
4 Piga, Dario
4 Puzyrev, Vladimir
4 Raissi, Maziar
4 Schiavazzi, Daniele E.
4 Shin, Yeonjong
4 Son, Hwijae
4 Sun, Hao
4 Takáč, Martin
4 Tang, Yifa
4 Thürey, Nils
4 Wang, Chunmei
4 Wang, Haoqin
4 Wang, Qian
4 Weber, Janine
4 Wen, Zaiwen
4 Yang, Chao
4 Yang, Yibo
4 Zepeda-Núñez, Leonardo
4 Zhang, Dongxiao
4 Zhang, Kai
4 Zhang, Linfeng
4 Zhang, Zhen
...and 5,208 more Authors
all top 5

Cited in 304 Serials

214 Journal of Computational Physics
161 Computer Methods in Applied Mechanics and Engineering
72 Neural Networks
69 Machine Learning
49 SIAM Journal on Scientific Computing
49 Journal of Machine Learning Research (JMLR)
44 Information Sciences
34 Computational Mechanics
34 Communications in Computational Physics
30 Neural Computation
29 Computers & Mathematics with Applications
29 Journal of Scientific Computing
26 Artificial Intelligence
26 Computers and Fluids
25 European Journal of Operational Research
24 Journal of Fluid Mechanics
23 Journal of Computational and Applied Mathematics
20 Quantum Information Processing
20 SIAM Journal on Imaging Sciences
19 Data Mining and Knowledge Discovery
19 Quantitative Finance
19 Statistics and Computing
17 Inverse Problems
16 Journal of Mathematical Imaging and Vision
15 International Journal of Computer Vision
15 Computational Geosciences
14 Physica D
14 SIAM Journal on Mathematics of Data Science
13 The Journal of Artificial Intelligence Research (JAIR)
12 Computer Physics Communications
12 Automatica
12 Mathematical Finance
12 Communications in Nonlinear Science and Numerical Simulation
11 SIAM Journal on Optimization
11 Inverse Problems and Imaging
9 Journal of the Franklin Institute
9 Journal of the American Statistical Association
9 Computers & Operations Research
9 Journal of Statistical Mechanics: Theory and Experiment
8 Physica A
8 Applied Mathematics and Computation
8 European Journal of Mechanics. A. Solids
8 Journal of High Energy Physics
8 International Journal of Computational Methods
8 Mathematical Biosciences and Engineering
8 Journal of Computational and Graphical Statistics
7 Chaos, Solitons and Fractals
7 Computer Aided Geometric Design
7 Applied Mathematical Modelling
7 Computational Optimization and Applications
7 Optimization Methods & Software
7 The Annals of Applied Statistics
7 Research in the Mathematical Sciences
7 Stat
6 Physics Letters. B
6 International Journal of Approximate Reasoning
6 Journal of Global Optimization
6 INFORMS Journal on Computing
6 Journal of Physics A: Mathematical and Theoretical
6 Mathematical Geosciences
6 SIAM Journal on Financial Mathematics
6 Journal of the Operations Research Society of China
5 Physics Letters. A
5 Journal of Econometrics
5 Journal of Optimization Theory and Applications
5 Mathematics and Computers in Simulation
5 Statistical Science
5 Computational Statistics
5 Computational Statistics and Data Analysis
5 IEEE Transactions on Image Processing
5 International Journal of Wavelets, Multiresolution and Information Processing
5 Numerical Mathematics: Theory, Methods and Applications
5 East Asian Journal on Applied Mathematics
5 SN Partial Differential Equations and Applications
4 Bulletin of Mathematical Biology
4 SIAM Journal on Numerical Analysis
4 Journal of Economic Dynamics & Control
4 European Journal of Applied Mathematics
4 Mathematical Programming. Series A. Series B
4 Computational and Applied Mathematics
4 ETNA. Electronic Transactions on Numerical Analysis
4 Journal of Systems Science and Complexity
4 SIAM Journal on Applied Dynamical Systems
4 Electronic Journal of Statistics
4 Bayesian Analysis
4 Communications in Mathematics and Statistics
4 SIAM/ASA Journal on Uncertainty Quantification
4 Sampling Theory, Signal Processing, and Data Analysis
3 Biological Cybernetics
3 Physics Reports
3 Psychometrika
3 Theoretical Computer Science
3 Insurance Mathematics & Economics
3 Operations Research Letters
3 Journal of Computational Mathematics
3 SIAM Journal on Matrix Analysis and Applications
3 Numerical Algorithms
3 Automation and Remote Control
3 Journal of Statistical Computation and Simulation
3 SIAM Review
...and 204 more Serials
all top 5

Cited in 55 Fields

1,233 Computer science (68-XX)
597 Numerical analysis (65-XX)
337 Statistics (62-XX)
231 Operations research, mathematical programming (90-XX)
212 Partial differential equations (35-XX)
199 Fluid mechanics (76-XX)
167 Biology and other natural sciences (92-XX)
161 Mechanics of deformable solids (74-XX)
126 Game theory, economics, finance, and other social and behavioral sciences (91-XX)
109 Probability theory and stochastic processes (60-XX)
81 Information and communication theory, circuits (94-XX)
68 Quantum theory (81-XX)
65 Systems theory; control (93-XX)
47 Calculus of variations and optimal control; optimization (49-XX)
46 Statistical mechanics, structure of matter (82-XX)
41 Dynamical systems and ergodic theory (37-XX)
39 Geophysics (86-XX)
23 Mechanics of particles and systems (70-XX)
19 Approximations and expansions (41-XX)
17 Ordinary differential equations (34-XX)
17 Optics, electromagnetic theory (78-XX)
14 Linear and multilinear algebra; matrix theory (15-XX)
14 Operator theory (47-XX)
9 Combinatorics (05-XX)
9 Classical thermodynamics, heat transfer (80-XX)
9 Relativity and gravitational theory (83-XX)
7 Harmonic analysis on Euclidean spaces (42-XX)
6 Mathematical logic and foundations (03-XX)
5 General and overarching topics; collections (00-XX)
5 Real functions (26-XX)
5 Differential geometry (53-XX)
5 Algebraic topology (55-XX)
4 Number theory (11-XX)
4 Algebraic geometry (14-XX)
4 Several complex variables and analytic spaces (32-XX)
3 Integral equations (45-XX)
3 Functional analysis (46-XX)
3 Global analysis, analysis on manifolds (58-XX)
2 Category theory; homological algebra (18-XX)
2 Topological groups, Lie groups (22-XX)
2 Measure and integration (28-XX)
2 Special functions (33-XX)
2 Integral transforms, operational calculus (44-XX)
2 Geometry (51-XX)
2 Convex and discrete geometry (52-XX)
2 Manifolds and cell complexes (57-XX)
1 Order, lattices, ordered algebraic structures (06-XX)
1 Associative rings and algebras (16-XX)
1 Group theory and generalizations (20-XX)
1 Functions of a complex variable (30-XX)
1 Potential theory (31-XX)
1 Difference and functional equations (39-XX)
1 General topology (54-XX)
1 Astronomy and astrophysics (85-XX)
1 Mathematics education (97-XX)

Citations by Year