swMATH ID: 22202
Software Authors: Duchi, John; Hazan, Elad; Singer, Yoram
Description: ADAGRAD: adaptive gradient algorithm; Adaptive subgradient methods for online learning and stochastic optimization. We present a new family of subgradient methods that dynamically incorporate knowledge of the geometry of the data observed in earlier iterations to perform more informative gradient-based learning. Metaphorically, the adaptation allows us to find needles in haystacks in the form of very predictive but rarely seen features. Our paradigm stems from recent advances in stochastic optimization and online learning which employ proximal functions to control the gradient steps of the algorithm. We describe and analyze an apparatus for adaptively modifying the proximal function, which significantly simplifies setting a learning rate and results in regret guarantees that are provably as good as the best proximal function that can be chosen in hindsight. We give several efficient algorithms for empirical risk minimization problems with common and important regularization functions and domain constraints. We experimentally study our theoretical analysis and show that adaptive subgradient methods outperform state-of-the-art, yet non-adaptive, subgradient algorithms.
Homepage: http://www.jmlr.org/papers/volume12/duchi11a/duchi11a.pdf
Keywords: subgradient methods; adaptivity; online learning; stochastic convex optimization
Related Software: Adam; RMSprop; ADADELTA; ImageNet; Saga; AlexNet; TensorFlow; PyTorch; PRMLT; CIFAR; MNIST; L-BFGS; darch; ElemStatLearn; GitHub; SGD-QN; Theano; LIBSVM; Keras; DiffSharp
Cited in: 186 Publications
all top 5

Cited by 516 Authors

5 Wen, Zaiwen
4 Crammer, Koby
4 Duchi, John C.
3 Milzarek, Andre
3 Orabona, Francesco
3 Yang, Haizhao
3 Yuan, Ya-xiang
2 Barakat, Anas
2 Bianchi, Pascal
2 Chaudhari, Pratik
2 Curtis, Frank E.
2 Fedkiw, Ronald P.
2 Gu, Yiqi
2 Guo, Tiande
2 Haghighat, Ehsan
2 Hazan, Elad
2 Hu, Jiang
2 Hu, Yaohua
2 Iiduka, Hideaki
2 Jordan, Michael Irwin
2 Juanes, Ruben
2 Karniadakis, George Em
2 Larochelle, Hugo
2 Lee, Jaehong
2 Lei, Lihua
2 Liu, Hailiang
2 Martínez, Ana M.
2 Moroshko, Edward
2 Nguyen, Lam M.
2 Nocedal, Jorge
2 Oberman, Adam M.
2 Osher, Stanley Joel
2 Pál, Dávid
2 Pardalos, Panos M.
2 Park, Seonho
2 Pauwels, Edouard Jean Robert
2 Perdikaris, Paris G.
2 Phan, Dzung T.
2 Roosta, Fred
2 Ruan, Feng
2 Ruthotto, Lars
2 Scheinberg, Katya
2 Soatto, Stefano
2 Srebro, Nathan
2 Sun, Tao
2 Wang, Xiao
2 Xu, Yangyang
2 Yan, Yonggui
2 Yang, Xiaoqi
2 Yu, Carisa Kwok Wai
1 Abate, Alessandro
1 Achab, Massil
1 Adosoglou, George
1 Agarwal, Alekh
1 Agarwal, Naman
1 Aggarwal, Charu C.
1 Airoldi, Edoardo M.
1 Ajakan, Hana
1 Akyildiz, Ömer Deniz
1 Ammar, Haitham Bou
1 Artikis, Alexander
1 Ashbrock, Jonathan
1 Bach, Francis R.
1 Bacry, Emmanuel
1 Bai, Jinshuai
1 Baldassi, Carlo
1 Bao, Gang
1 Bao, Michael
1 Baraniuk, Richard G.
1 Barrio, Roberto
1 Baydin, Atılım Güneş
1 Bekar, Ali Can
1 Bertozzi, Andrea Louise
1 Beskos, Alexandros
1 Bilionis, Ilias
1 Blei, David M.
1 Boffi, Nicholas M.
1 Bonettini, Silvia
1 Borgs, Christian
1 Bottou, Léon
1 Bouchard, Guillaume
1 Bouchard, Kristofer E.
1 Brandman, David M.
1 Bu, Yuheng
1 Bullins, Brian
1 Burkhart, Michael C.
1 Byrd, Richard H.
1 Cai, Difeng
1 Carlberg, Kevin T.
1 Carlier, Guillaume
1 Cen, Shicong
1 Cesa-Bianchi, Nicolò
1 Chada, Neil K.
1 Chae, Kum Ju
1 Chakradhar, Srimat T.
1 Chang, Hongyu
1 Chayes, Jennifer Tour
1 Chen, Jie
1 Chen, Jingrun
1 Chen, Jiun-Shyan
...and 416 more Authors
all top 5

Cited in 78 Serials

21 Journal of Machine Learning Research (JMLR)
13 SIAM Journal on Optimization
10 Journal of Computational Physics
10 Machine Learning
8 Neural Computation
7 Computer Methods in Applied Mechanics and Engineering
6 SIAM Journal on Scientific Computing
5 Mathematical Programming. Series A. Series B
4 Theoretical Computer Science
4 Optimization Methods & Software
4 Statistics and Computing
4 Journal of the Operations Research Society of China
3 Information Sciences
3 Neural Networks
3 Journal of Global Optimization
3 Communications in Computational Physics
3 SIAM Journal on Imaging Sciences
2 The Annals of Statistics
2 Mathematics and Computers in Simulation
2 International Journal of Approximate Reasoning
2 Journal of Scientific Computing
2 Numerical Algorithms
2 European Journal of Operational Research
2 Computational Statistics and Data Analysis
2 Computational Optimization and Applications
2 Annals of Mathematics and Artificial Intelligence
2 Engineering Analysis with Boundary Elements
1 Artificial Intelligence
1 Computers & Mathematics with Applications
1 Inverse Problems
1 Journal of the Franklin Institute
1 Physics Reports
1 Mathematics of Computation
1 Automatica
1 Journal of Differential Equations
1 Journal of Optimization Theory and Applications
1 Mathematics of Operations Research
1 Operations Research Letters
1 Journal of Computational Mathematics
1 Optimization
1 Sequential Analysis
1 Statistical Science
1 Computers & Operations Research
1 Computational Mechanics
1 SIAM Journal on Matrix Analysis and Applications
1 Multidimensional Systems and Signal Processing
1 Computational Mathematics and Mathematical Physics
1 Applied Mathematical Modelling
1 Journal of Statistical Computation and Simulation
1 SIAM Review
1 International Journal of Computer Vision
1 Journal of Inverse and Ill-Posed Problems
1 INFORMS Journal on Computing
1 Data Mining and Knowledge Discovery
1 Computational Geosciences
1 CEJOR. Central European Journal of Operations Research
1 Archives of Computational Methods in Engineering
1 Journal of Systems Science and Complexity
1 International Journal of Wavelets, Multiresolution and Information Processing
1 International Journal of Computational Methods
1 Mediterranean Journal of Mathematics
1 Journal of Statistical Mechanics: Theory and Experiment
1 Optimization Letters
1 Inverse Problems and Imaging
1 Electronic Journal of Statistics
1 Mathematical Geosciences
1 Discrete Mathematics, Algorithms and Applications
1 Mathematical Programming Computation
1 Science China. Information Sciences
1 Journal of Computational and Graphical Statistics
1 SIAM/ASA Journal on Uncertainty Quantification
1 Research in the Mathematical Sciences
1 Annals of Applied Mathematics
1 Mathematical Foundations of Computing
1 Japanese Journal of Statistics and Data Science
1 SIAM Journal on Mathematics of Data Science
1 Sampling Theory, Signal Processing, and Data Analysis
1 Fixed Point Theory and Algorithms for Sciences and Engineering

Citations by Year