ADADELTA swMATH ID: 39429 Software Authors: Matthew D. Zeiler Description: ADADELTA: An Adaptive Learning Rate Method. We present a novel per-dimension learning rate method for gradient descent called ADADELTA. The method dynamically adapts over time using only first order information and has minimal computational overhead beyond vanilla stochastic gradient descent. The method requires no manual tuning of a learning rate and appears robust to noisy gradient information, different model architecture choices, various data modalities and selection of hyperparameters. We show promising results compared to other methods on the MNIST digit classification task using a single machine and on a large scale voice dataset in a distributed cluster environment. Homepage: https://paperswithcode.com/paper/adadelta-an-adaptive-learning-rate-method Related Software: Adam; AdaGrad; RMSprop; ImageNet; AlexNet; TensorFlow; PRMLT; PyTorch; GitHub; MNIST; darch; L-BFGS; Keras; CIFAR; ADVI; Matlab; ElemStatLearn; Saga; R; DiffSharp Cited in: 80 Publications all top 5 Cited by 227 Authors 6 Nott, David John 4 Smith, Michael Stanley 3 Loaiza-Maya, Rubén 3 Tran, Minh Ngoc 3 Yan, Liang 3 Zhou, Tao 2 Bilionis, Ilias 2 Bottou, Léon 2 Fedkiw, Ronald P. 2 Lee, Jaehong 2 Ong, Victor M. H. 2 Osher, Stanley Joel 2 Roosta, Fred 2 Tripathy, Rohit K. 2 Wang, Bao 1 Abdessalem, Talel 1 Babenko, Mikhail 1 Bai, Jinshuai 1 Balu, Aditya 1 Bao, Michael 1 Baraniuk, Richard G. 1 Barekat, Farzin 1 Basir, Shamsulhaq 1 Belotto da Silva, André Ricardo 1 Bernardo, Felipe 1 Berrada, Leonard 1 Bertozzi, Andrea Louise 1 Bifet, Albert 1 Bolmier, Geoffrey 1 Borges, Marcio R. 1 Bouzerdoum, Abdesselam 1 Brandman, David M. 1 Burkhart, Michael C. 1 Chen, Jingrun 1 Chen, Tom 1 Chung, Julianne M. 1 Chung, Matthias 1 Clausen, Johan Bjerre Bach 1 Cortés-Mendoza, Jorge M. 1 Curtis, Frank E. 1 Danaher, Peter J. 1 De Loera, Jesús A. 1 De, Subhayan 1 Dingreville, Rémi 1 Do, Dieu T. T. 1 Do, Xuan Anh 1 Doostan, Alireza 1 Drovandi, Christopher C. 1 Du, Qiang 1 Erway, Jennifer B. 1 Fang, Chih-Hao 1 Feng, Miria 1 Feng, Wenying 1 Forestier, Germain 1 Franco, Brian 1 Gao, Junbin 1 Gazeau, Maxime 1 Geete, Kanu 1 Geng, Zhenglin 1 Gerlach, Richard H. 1 Ghadai, Sambit 1 Giraldi, Gilson Antonio 1 Gomes, Heitor Murilo 1 Göttlich, Simone 1 Grama, Ananth Y. 1 Griffin, Joshua D. 1 Gu, Yuantong 1 Guo, Jun 1 Ha, Jingwan 1 Haddock, Jamie 1 Halford, Max 1 Han, Xianpei 1 Harandi, Ali 1 Harrison, Matthew T. 1 Hebishima, Hana 1 Henderson, Donna 1 Hochberg, Leigh R. 1 Huck, Nicolas 1 Hyde, David A. B. 1 Idoumghar, Lhassane 1 Inage, Sin-ichi 1 Ismail Fawaz, Hassan 1 Jeong, Hyogu 1 Jeong, Jong-Hyeon 1 Jia, Yichen 1 Jo, Seongil 1 Kangal, Evrim Ersin 1 Kantas, Nikolas 1 Karniadakis, George Em 1 Karumuri, Sharmila 1 Kejzlar, Vojtech 1 Kim, Minseok 1 Kim, Sehwan 1 Kim, Sundong 1 Knapp, Stephan 1 Kohn, Robert J. 1 Koike, Atsushi 1 Kojima, Ryosuke 1 Krauss, Christopher 1 Krishnamurthy, Adarsh ...and 127 more Authors all top 5 Cited in 49 Serials 7 Journal of Machine Learning Research (JMLR) 6 Journal of Computational Physics 6 Journal of Computational and Graphical Statistics 4 Computer Methods in Applied Mechanics and Engineering 4 Communications in Computational Physics 3 Neural Computation 3 Statistics and Computing 2 Machine Learning 2 Computational Statistics 2 Journal of Statistical Computation and Simulation 1 Inverse Problems 1 Physics Reports 1 Psychometrika 1 Journal of Computational and Applied Mathematics 1 Journal of Econometrics 1 Mathematics and Computers in Simulation 1 Programming and Computer Software 1 Computer Aided Geometric Design 1 Journal of Computational Mathematics 1 Computers & Operations Research 1 Computational Mechanics 1 International Journal of Approximate Reasoning 1 SIAM Journal on Matrix Analysis and Applications 1 Journal of Scientific Computing 1 Annals of Physics 1 Applied Mathematical Modelling 1 European Journal of Operational Research 1 SIAM Review 1 Stochastic Processes and their Applications 1 Computational Statistics and Data Analysis 1 SIAM Journal on Optimization 1 The Journal of Artificial Intelligence Research (JAIR) 1 Annals of Mathematics and Artificial Intelligence 1 Optimization Methods & Software 1 Data Mining and Knowledge Discovery 1 Interdisciplinary Information Sciences (IIS) 1 Computational Geosciences 1 Probability in the Engineering and Informational Sciences 1 Scandinavian Actuarial Journal 1 Archives of Computational Methods in Engineering 1 International Journal of Computational Methods 1 SIAM Journal on Imaging Sciences 1 Journal of Control Science and Engineering 1 Mathematical Programming Computation 1 Journal of the Operations Research Society of China 1 SIAM/ASA Journal on Uncertainty Quantification 1 Research in the Mathematical Sciences 1 Annals of Applied Mathematics 1 Mathematical Foundations of Computing all top 5 Cited in 21 Fields 43 Computer science (68-XX) 28 Statistics (62-XX) 27 Numerical analysis (65-XX) 20 Operations research, mathematical programming (90-XX) 6 Partial differential equations (35-XX) 6 Mechanics of deformable solids (74-XX) 5 Probability theory and stochastic processes (60-XX) 4 Calculus of variations and optimal control; optimization (49-XX) 3 Game theory, economics, finance, and other social and behavioral sciences (91-XX) 2 General and overarching topics; collections (00-XX) 2 Fluid mechanics (76-XX) 2 Quantum theory (81-XX) 2 Systems theory; control (93-XX) 1 Linear and multilinear algebra; matrix theory (15-XX) 1 Harmonic analysis on Euclidean spaces (42-XX) 1 Algebraic topology (55-XX) 1 Optics, electromagnetic theory (78-XX) 1 Statistical mechanics, structure of matter (82-XX) 1 Astronomy and astrophysics (85-XX) 1 Biology and other natural sciences (92-XX) 1 Information and communication theory, circuits (94-XX) Citations by Year