AdaGrad swMATH ID: 22202 Software Authors: Duchi, John; Hazan, Elad; Singer, Yoram Description: ADAGRAD: adaptive gradient algorithm; Adaptive subgradient methods for online learning and stochastic optimization. We present a new family of subgradient methods that dynamically incorporate knowledge of the geometry of the data observed in earlier iterations to perform more informative gradient-based learning. Metaphorically, the adaptation allows us to find needles in haystacks in the form of very predictive but rarely seen features. Our paradigm stems from recent advances in stochastic optimization and online learning which employ proximal functions to control the gradient steps of the algorithm. We describe and analyze an apparatus for adaptively modifying the proximal function, which significantly simplifies setting a learning rate and results in regret guarantees that are provably as good as the best proximal function that can be chosen in hindsight. We give several efficient algorithms for empirical risk minimization problems with common and important regularization functions and domain constraints. We experimentally study our theoretical analysis and show that adaptive subgradient methods outperform state-of-the-art, yet non-adaptive, subgradient algorithms. Homepage: http://www.jmlr.org/papers/volume12/duchi11a/duchi11a.pdf Keywords: subgradient methods; adaptivity; online learning; stochastic convex optimization Related Software: Adam; RMSprop; ADADELTA; ImageNet; Saga; AlexNet; TensorFlow; PyTorch; CIFAR; PRMLT; MNIST; L-BFGS; darch; LIBSVM; SGD-QN; ElemStatLearn; GitHub; UCI-ml; DiffSharp; Theano Cited in: 206 Documents Standard Articles 1 Publication describing the Software, including 1 Publication in zbMATH Year Adaptive subgradient methods for online learning and stochastic optimization. Zbl 1280.68164Duchi, John; Hazan, Elad; Singer, Yoram 2011 all top 5 Cited by 561 Authors 5 Wen, Zaiwen 4 Crammer, Koby 4 Duchi, John C. 3 Curtis, Frank E. 3 Milzarek, Andre 3 Orabona, Francesco 3 Yang, Haizhao 3 Yuan, Ya-xiang 2 Barakat, Anas 2 Bianchi, Pascal 2 Chaudhari, Pratik 2 Crisan, Dan O. 2 Dvurechensky, Pavel E. 2 Fedkiw, Ronald P. 2 Gasnikov, Aleksandr V. 2 Goda, Takashi 2 Gu, Yiqi 2 Guo, Tiande 2 Haghighat, Ehsan 2 Hazan, Elad 2 Hu, Jiang 2 Hu, Yaohua 2 Iiduka, Hideaki 2 Jiang, Bo 2 Jordan, Michael Irwin 2 Juanes, Ruben 2 Karniadakis, George Em 2 Kitade, Wataru 2 Larochelle, Hugo 2 Lee, Jaehong 2 Lei, Lihua 2 Liu, Hailiang 2 Martínez, Ana M. 2 Moroshko, Edward 2 Nguyen, Lam M. 2 Nocedal, Jorge 2 Oberman, Adam M. 2 Osher, Stanley Joel 2 Pál, Dávid 2 Pardalos, Panos M. 2 Park, Seonho 2 Pauwels, Edouard Jean Robert 2 Perdikaris, Paris G. 2 Phan, Dzung T. 2 Richtárik, Peter 2 Roosta, Fred 2 Ruan, Feng 2 Ruthotto, Lars 2 Scheinberg, Katya 2 Soatto, Stefano 2 Srebro, Nathan 2 Sun, Tao 2 Takáč, Martin 2 Wang, Xiao 2 Xu, Yangyang 2 Yan, Yonggui 2 Yang, Xiaoqi 2 Yu, Carisa Kwok Wai 1 Abate, Alessandro 1 Achab, Massil 1 Adosoglou, George 1 Agarwal, Alekh 1 Agarwal, Naman 1 Aggarwal, Charu C. 1 Airoldi, Edoardo M. 1 Ajakan, Hana 1 Akyildiz, Ömer Deniz 1 Alonso-Mora, Javier 1 Ammar, Haitham Bou 1 Artikis, Alexander 1 Ashbrock, Jonathan 1 Bach, Francis R. 1 Bacry, Emmanuel 1 Bai, Jinshuai 1 Baldassi, Carlo 1 Bao, Gang 1 Bao, Michael 1 Baraniuk, Richard G. 1 Barrio, Roberto 1 Baydin, Atılım Güneş 1 Bekar, Ali Can 1 Belhouchette, Kenza 1 Berahas, Albert S. 1 Bertozzi, Andrea Louise 1 Beskos, Alexandros 1 Bilionis, Ilias 1 Blei, David M. 1 Boffi, Nicholas M. 1 Böhmer, Wendelin 1 Bonettini, Silvia 1 Borgs, Christian 1 Bottou, Léon 1 Bouchard, Guillaume 1 Bouchard, Kristofer E. 1 Brandman, David M. 1 Bu, Kaifeng 1 Bu, Yuheng 1 Bullins, Brian 1 Burkhart, Michael C. 1 Byrd, Richard H. ...and 461 more Authors all top 5 Cited in 85 Serials 21 Journal of Machine Learning Research (JMLR) 13 SIAM Journal on Optimization 10 Journal of Computational Physics 10 Machine Learning 8 Computer Methods in Applied Mechanics and Engineering 8 Neural Computation 7 Optimization Methods & Software 6 SIAM Journal on Scientific Computing 6 Statistics and Computing 5 Mathematical Programming. Series A. Series B 4 Theoretical Computer Science 4 Journal of Global Optimization 4 Journal of the Operations Research Society of China 3 Information Sciences 3 Mathematics and Computers in Simulation 3 Neural Networks 3 Communications in Computational Physics 3 SIAM Journal on Imaging Sciences 2 The Annals of Statistics 2 Journal of Computational and Applied Mathematics 2 Journal of Optimization Theory and Applications 2 International Journal of Approximate Reasoning 2 Journal of Scientific Computing 2 Numerical Algorithms 2 European Journal of Operational Research 2 Computational Statistics and Data Analysis 2 Computational Optimization and Applications 2 Annals of Mathematics and Artificial Intelligence 1 Artificial Intelligence 1 Computers & Mathematics with Applications 1 Inverse Problems 1 Journal of the Franklin Institute 1 Physics Reports 1 Mathematics of Computation 1 Automatica 1 Journal of Differential Equations 1 Mathematics of Operations Research 1 Operations Research Letters 1 Journal of Computational Mathematics 1 Optimization 1 Sequential Analysis 1 Statistical Science 1 Computers & Operations Research 1 Computational Mechanics 1 SIAM Journal on Matrix Analysis and Applications 1 Multidimensional Systems and Signal Processing 1 Computational Mathematics and Mathematical Physics 1 Applied Mathematical Modelling 1 Journal of Statistical Computation and Simulation 1 SIAM Review 1 Journal of Mathematical Imaging and Vision 1 International Journal of Computer Vision 1 Journal of Inverse and Ill-Posed Problems 1 Engineering Analysis with Boundary Elements 1 INFORMS Journal on Computing 1 Data Mining and Knowledge Discovery 1 Computational Geosciences 1 CEJOR. Central European Journal of Operations Research 1 Journal of High Energy Physics 1 Archives of Computational Methods in Engineering 1 Journal of Systems Science and Complexity 1 Quantum Information Processing 1 International Journal of Wavelets, Multiresolution and Information Processing 1 International Journal of Computational Methods 1 Mediterranean Journal of Mathematics 1 Journal of Statistical Mechanics: Theory and Experiment 1 Optimization Letters 1 Inverse Problems and Imaging 1 Electronic Journal of Statistics 1 Mathematical Geosciences 1 Discrete Mathematics, Algorithms and Applications 1 Mathematical Programming Computation 1 Science China. Information Sciences 1 Journal of Computational and Graphical Statistics 1 Stochastic and Partial Differential Equations. Analysis and Computations 1 SIAM/ASA Journal on Uncertainty Quantification 1 Research in the Mathematical Sciences 1 Open Mathematics 1 Annals of Applied Mathematics 1 Advances in Data Science and Adaptive Analysis 1 Mathematical Foundations of Computing 1 Japanese Journal of Statistics and Data Science 1 SIAM Journal on Mathematics of Data Science 1 Sampling Theory, Signal Processing, and Data Analysis 1 Fixed Point Theory and Algorithms for Sciences and Engineering all top 5 Cited in 28 Fields 104 Computer science (68-XX) 93 Operations research, mathematical programming (90-XX) 74 Numerical analysis (65-XX) 54 Statistics (62-XX) 15 Calculus of variations and optimal control; optimization (49-XX) 9 Partial differential equations (35-XX) 9 Mechanics of deformable solids (74-XX) 9 Systems theory; control (93-XX) 8 Biology and other natural sciences (92-XX) 7 Probability theory and stochastic processes (60-XX) 7 Information and communication theory, circuits (94-XX) 5 Fluid mechanics (76-XX) 4 Dynamical systems and ergodic theory (37-XX) 3 Linear and multilinear algebra; matrix theory (15-XX) 3 Ordinary differential equations (34-XX) 3 Quantum theory (81-XX) 3 Game theory, economics, finance, and other social and behavioral sciences (91-XX) 2 Operator theory (47-XX) 2 Statistical mechanics, structure of matter (82-XX) 2 Geophysics (86-XX) 1 General and overarching topics; collections (00-XX) 1 Number theory (11-XX) 1 Real functions (26-XX) 1 Functions of a complex variable (30-XX) 1 Approximations and expansions (41-XX) 1 Convex and discrete geometry (52-XX) 1 Algebraic topology (55-XX) 1 Mechanics of particles and systems (70-XX) Citations by Year