AdaGrad swMATH ID: 22202 Software Authors: Duchi, John; Hazan, Elad; Singer, Yoram Description: ADAGRAD: adaptive gradient algorithm; Adaptive subgradient methods for online learning and stochastic optimization. We present a new family of subgradient methods that dynamically incorporate knowledge of the geometry of the data observed in earlier iterations to perform more informative gradient-based learning. Metaphorically, the adaptation allows us to find needles in haystacks in the form of very predictive but rarely seen features. Our paradigm stems from recent advances in stochastic optimization and online learning which employ proximal functions to control the gradient steps of the algorithm. We describe and analyze an apparatus for adaptively modifying the proximal function, which significantly simplifies setting a learning rate and results in regret guarantees that are provably as good as the best proximal function that can be chosen in hindsight. We give several efficient algorithms for empirical risk minimization problems with common and important regularization functions and domain constraints. We experimentally study our theoretical analysis and show that adaptive subgradient methods outperform state-of-the-art, yet non-adaptive, subgradient algorithms. Homepage: http://www.jmlr.org/papers/volume12/duchi11a/duchi11a.pdf Keywords: subgradient methods; adaptivity; online learning; stochastic convex optimization Related Software: Adam; ADADELTA; RMSprop; ImageNet; Saga; AlexNet; CIFAR; PyTorch; TensorFlow; L-BFGS; MNIST; PRMLT; darch; LIBSVM; GitHub; DiffSharp; DGM; SGD-QN; ElemStatLearn; Fashion-MNIST Cited in: 322 Documents Standard Articles 1 Publication describing the Software, including 1 Publication in zbMATH Year Adaptive subgradient methods for online learning and stochastic optimization. Zbl 1280.68164Duchi, John; Hazan, Elad; Singer, Yoram 2011 all top 5 Cited by 862 Authors 6 Gasnikov, Aleksandr V. 6 Wen, Zaiwen 5 Takáč, Martin 4 Crammer, Koby 4 Duchi, John C. 4 Milzarek, Andre 3 Beznosikov, Aleksandr 3 Curtis, Frank E. 3 Dvurechensky, Pavel E. 3 Fedkiw, Ronald P. 3 Gratton, Serge 3 Guo, Tiande 3 Hu, Jiang 3 Iiduka, Hideaki 3 Kamzolov, Dmitry 3 Liu, Hailiang 3 Maiti, Tapabrata 3 Orabona, Francesco 3 Pauwels, Edouard Jean Robert 3 Soatto, Stefano 3 Toint, Philippe Louis 3 Wang, Hansheng 3 Wang, Xiao 3 Xu, Dongpo 3 Xu, Yangyang 3 Yan, Yonggui 3 Yang, Haizhao 3 Yuan, Ya-xiang 2 Bach, Francis R. 2 Barakat, Anas 2 Bhattacharya, Shrijita 2 Bianchi, Pascal 2 Bu, Kaifeng 2 Burkhart, Michael C. 2 Cai, Li 2 Chaudhari, Pratik 2 Chen, Jie 2 Chen, Yaping 2 Crisan, Dan O. 2 Duruisseaux, Valentin 2 Gao, Yuan 2 Garcia, Roy J. 2 Goda, Takashi 2 Godichon-Baggioni, Antoine 2 Gorbunov, Eduard 2 Gu, Yiqi 2 Haghighat, Ehsan 2 Han, Congying 2 Han, Deren 2 Hazan, Elad 2 Hu, Yaohua 2 Huang, DanYang 2 Jaffe, Arthur Michael 2 Jiang, Bo 2 Jordan, Michael Irwin 2 Juanes, Ruben 2 Karniadakis, George Em 2 Kejzlar, Vojtech 2 Kitade, Wataru 2 Krejić, Nataša 2 Krklec Jerinkić, Nataša 2 Larochelle, Hugo 2 Lee, Jaehong 2 Lei, Lihua 2 Leok, Melvin 2 Li, Xian 2 Liu, Jingchen 2 Liu, Jinlan 2 Liu, Yan 2 Liu, Youqiong 2 Martínez, Ana M. 2 Mobasher, Mostafa E. 2 Moroshko, Edward 2 Nguyen-Thoi, Trung 2 Nguyen, Lam M. 2 Nocedal, Jorge 2 Oberman, Adam M. 2 Osher, Stanley Joel 2 Pál, Dávid 2 Palagi, Laura 2 Pantidis, Panos 2 Pardalos, Panos M. 2 Park, Seonho 2 Perdikaris, Paris G. 2 Phan, Dzung T. 2 Phoha, Shashi P. 2 Pinson, Pierre 2 Qi, Haobo 2 Ray, Asok 2 Richtárik, Peter 2 Roosta, Fred 2 Ruan, Feng 2 Ruthotto, Lars 2 Saab, Samer jun. 2 Scheinberg, Katya 2 Schmidt, Mark 2 Srebro, Nathan 2 Sun, Tao 2 Tang, Xueying 2 Tian, Xuping ...and 762 more Authors all top 5 Cited in 119 Serials 21 Journal of Machine Learning Research (JMLR) 17 SIAM Journal on Optimization 16 Neural Networks 14 Machine Learning 13 Journal of Computational Physics 11 Statistics and Computing 10 Computer Methods in Applied Mechanics and Engineering 8 Neural Computation 8 Computational Optimization and Applications 7 Journal of Global Optimization 7 SIAM Journal on Scientific Computing 7 Optimization Methods & Software 6 Information Sciences 6 Mathematical Programming. Series A. Series B 5 Journal of the Operations Research Society of China 4 Journal of Optimization Theory and Applications 4 Theoretical Computer Science 4 Journal of Scientific Computing 4 Numerical Algorithms 3 Computers & Mathematics with Applications 3 Journal of the American Statistical Association 3 Journal of Computational and Applied Mathematics 3 Mathematics and Computers in Simulation 3 SIAM Journal on Matrix Analysis and Applications 3 Computational Mathematics and Mathematical Physics 3 Computational Statistics and Data Analysis 3 Communications in Computational Physics 3 SIAM Journal on Imaging Sciences 2 The Annals of Statistics 2 Automatica 2 British Journal of Mathematical & Statistical Psychology 2 Journal of Differential Equations 2 International Journal of Approximate Reasoning 2 European Journal of Operational Research 2 Annals of Mathematics and Artificial Intelligence 2 Computational Geosciences 2 Journal of High Energy Physics 2 Optimization and Engineering 2 Quantum Information Processing 2 International Journal of Computational Methods 2 Computational Management Science 2 Journal of Industrial and Management Optimization 2 Optimization Letters 2 Mathematical Geosciences 2 Mathematical Programming Computation 2 Journal of Computational and Graphical Statistics 2 SIAM/ASA Journal on Uncertainty Quantification 1 The American Statistician 1 Artificial Intelligence 1 The Canadian Journal of Statistics 1 General Relativity and Gravitation 1 Inverse Problems 1 Journal of the Franklin Institute 1 Physics Reports 1 Mathematics of Computation 1 Biometrics 1 Computing 1 International Statistical Review 1 Journal of Econometrics 1 Mathematics of Operations Research 1 Meccanica 1 Technometrics 1 Operations Research Letters 1 Chinese Annals of Mathematics. Series B 1 Journal of Computational Mathematics 1 Statistics 1 Optimization 1 Sequential Analysis 1 Statistical Science 1 Revista Matemática Iberoamericana 1 Computers & Operations Research 1 Computational Mechanics 1 Annals of Operations Research 1 Multidimensional Systems and Signal Processing 1 Applied Mathematical Modelling 1 Journal of Statistical Computation and Simulation 1 SIAM Review 1 Journal of Mathematical Imaging and Vision 1 International Journal of Computer Vision 1 Computational and Applied Mathematics 1 Journal of Inverse and Ill-Posed Problems 1 Top 1 Engineering Analysis with Boundary Elements 1 ACM Transactions on Modeling and Computer Simulation 1 INFORMS Journal on Computing 1 Mathematical Finance 1 Data Mining and Knowledge Discovery 1 Journal of the European Mathematical Society (JEMS) 1 CEJOR. Central European Journal of Operations Research 1 Lobachevskii Journal of Mathematics 1 Revista de Matemática: Teoría y Aplicaciones 1 Archives of Computational Methods in Engineering 1 Journal of Systems Science and Complexity 1 Natural Computing 1 International Journal of Wavelets, Multiresolution and Information Processing 1 Mediterranean Journal of Mathematics 1 Journal of Statistical Mechanics: Theory and Experiment 1 Mathematical Biosciences and Engineering 1 Inverse Problems and Imaging 1 Electronic Journal of Statistics ...and 19 more Serials all top 5 Cited in 32 Fields 162 Computer science (68-XX) 149 Operations research, mathematical programming (90-XX) 105 Numerical analysis (65-XX) 82 Statistics (62-XX) 23 Calculus of variations and optimal control; optimization (49-XX) 14 Partial differential equations (35-XX) 13 Mechanics of deformable solids (74-XX) 13 Systems theory; control (93-XX) 10 Biology and other natural sciences (92-XX) 7 Probability theory and stochastic processes (60-XX) 7 Fluid mechanics (76-XX) 7 Game theory, economics, finance, and other social and behavioral sciences (91-XX) 6 Dynamical systems and ergodic theory (37-XX) 6 Information and communication theory, circuits (94-XX) 5 Linear and multilinear algebra; matrix theory (15-XX) 5 Quantum theory (81-XX) 4 Ordinary differential equations (34-XX) 3 Mechanics of particles and systems (70-XX) 3 Geophysics (86-XX) 2 Operator theory (47-XX) 2 Statistical mechanics, structure of matter (82-XX) 2 Relativity and gravitational theory (83-XX) 1 General and overarching topics; collections (00-XX) 1 Number theory (11-XX) 1 Category theory; homological algebra (18-XX) 1 Topological groups, Lie groups (22-XX) 1 Real functions (26-XX) 1 Functions of a complex variable (30-XX) 1 Approximations and expansions (41-XX) 1 Convex and discrete geometry (52-XX) 1 Algebraic topology (55-XX) 1 Astronomy and astrophysics (85-XX) Citations by Year