Saga swMATH ID: 39677 Software Authors: Aaron Defazio, Francis Bach, Simon Lacoste-Julien Description: SAGA: A Fast Incremental Gradient Method With Support for Non-Strongly Convex Composite Objectives. In this work we introduce a new optimisation method called SAGA in the spirit of SAG, SDCA, MISO and SVRG, a set of recently proposed incremental gradient algorithms with fast linear convergence rates. SAGA improves on the theory behind SAG and SVRG, with better theoretical convergence rates, and has support for composite objectives where a proximal operator is used on the regulariser. Unlike SDCA, SAGA supports non-strongly convex problems directly, and is adaptive to any inherent strong convexity of the problem. We give experimental results showing the effectiveness of our method. Homepage: https://paperswithcode.com/paper/saga-a-fast-incremental-gradient-method-with Related Software: AdaGrad; LIBSVM; Finito; Adam; ProxSARAH; HOGWILD; MNIST; ElemStatLearn; RMSprop; AlexNet; ImageNet; Pegasos; L-BFGS; UNLocBoX; CIFAR; SGD-QN; LIBLINEAR; ADADELTA; QUIC; iPiano Cited in: 129 Publications all top 5 Cited by 288 Authors 9 Richtárik, Peter 5 Gürbüzbalaban, Mert 5 Schönlieb, Carola-Bibiane 4 Mairal, Julien 4 Mokhtari, Aryan 4 Nguyen, Lam M. 4 Ozdaglar, Asuman E. 4 Tran Dinh Quoc 3 Driggs, Derek 3 Duchi, John C. 3 Ehrhardt, Matthias Joachim 3 Liang, Jingwei 3 Lin, Qihang 3 Parrilo, Pablo A. 3 Phan, Dzung T. 3 Ribeiro, Alejandro R. 3 Takáč, Martin 3 Wen, Zaiwen 3 Yang, Tianbao 2 Bach, Francis R. 2 Benning, Martin 2 Cevher, Volkan 2 Dai, Yu-Hong 2 Davis, Damek 2 Deng, Cheng 2 Gu, Bin 2 Gu, Quanquan 2 Harchaoui, Zaid 2 Huang, Heng 2 Iutzeler, Franck 2 Jordan, Michael Irwin 2 Lan, Guanghui 2 Latafat, Puya 2 Lei, Lihua 2 Li, Zhize 2 Lin, Hongzhou 2 Liu, Xinwei 2 Malick, Jérôme 2 Milzarek, Andre 2 Mishchenko, Konstantin 2 Nedić, Angelia 2 Nocedal, Jorge 2 Osher, Stanley Joel 2 Palagi, Laura 2 Patrinos, Panagiotis 2 Pham, Nhan H. 2 Scheinberg, Katya 2 Sidford, Aaron 2 Sun, Jie 2 Themelis, Andreas 2 Vũ, Bằng Công 2 Xiao, Lin 2 Xiao, Xiantao 2 Xu, Pan 2 Yu, Tengteng 2 Zhang, Xiaoqun 2 Zhou, Dongruo 2 Zhou, Yi 1 Agarwal, Naman 1 Ahookhosh, Masoud 1 Alacaoglu, Ahmet 1 An, Nguyen Thai 1 Aravkin, Aleksandr Y. 1 Barekat, Farzin 1 Barrio, Roberto 1 Başar, Tamer 1 Belomestny, Denis 1 Betcke, Marta M. 1 Bian, Fengmiao 1 Bianchi, Pascal 1 Blanchet, Jose H. 1 Blei, David M. 1 Boffi, Nicholas M. 1 Bollapragada, Raghu 1 Borovykh, Anastasia 1 Bottou, Léon 1 Briceño-Arias, Luis M. 1 Bullins, Brian 1 Burger, Martin 1 Byrd, Richard H. 1 Carlier, Guillaume 1 Carmon, Yair 1 Cen, Shicong 1 Chambolle, Antonin 1 Chaudhari, Pratik 1 Chehreghani, Morteza Haghir 1 Chen, Chenxi 1 Chen, Yunmei 1 Chen, Zengping 1 Cheng, Lizhi 1 Cheng, Shuyu 1 Cheung, Yiuming 1 Chi, Yuejie 1 Clémençon, Stéphan 1 Condat, Laurent 1 Csiba, Dominik 1 Cui, Shisheng 1 Curtis, Frank E. 1 Davies, Mike E. 1 Drori, Yoel ...and 188 more Authors all top 5 Cited in 39 Serials 22 SIAM Journal on Optimization 21 Journal of Machine Learning Research (JMLR) 13 Mathematical Programming. Series A. Series B 9 Computational Optimization and Applications 4 Journal of Optimization Theory and Applications 4 Mathematics of Operations Research 4 Machine Learning 4 Optimization Letters 3 Optimization Methods & Software 3 SIAM Journal on Imaging Sciences 3 SIAM Journal on Mathematics of Data Science 2 Inverse Problems 2 Neural Computation 2 Journal of Global Optimization 2 Numerical Algorithms 2 SIAM Journal on Scientific Computing 2 Analysis and Applications (Singapore) 2 Set-Valued and Variational Analysis 2 SIAM/ASA Journal on Uncertainty Quantification 2 Research in the Mathematical Sciences 1 Applicable Analysis 1 Journal of the Franklin Institute 1 The Annals of Statistics 1 Applied Mathematics and Optimization 1 Automatica 1 Information Sciences 1 Theoretical Computer Science 1 Journal of Computational Mathematics 1 Physica D 1 Statistical Science 1 SIAM Journal on Matrix Analysis and Applications 1 Journal of Scientific Computing 1 Neural Networks 1 SIAM Review 1 International Journal of Wavelets, Multiresolution and Information Processing 1 Acta Numerica 1 Journal of Industrial and Management Optimization 1 Journal of the Operations Research Society of China 1 Annals of Applied Mathematics all top 5 Cited in 16 Fields 101 Operations research, mathematical programming (90-XX) 42 Computer science (68-XX) 39 Numerical analysis (65-XX) 19 Statistics (62-XX) 10 Calculus of variations and optimal control; optimization (49-XX) 5 Biology and other natural sciences (92-XX) 5 Systems theory; control (93-XX) 4 Operator theory (47-XX) 3 Partial differential equations (35-XX) 3 Probability theory and stochastic processes (60-XX) 3 Information and communication theory, circuits (94-XX) 2 Linear and multilinear algebra; matrix theory (15-XX) 1 Real functions (26-XX) 1 Convex and discrete geometry (52-XX) 1 Mechanics of deformable solids (74-XX) 1 Statistical mechanics, structure of matter (82-XX) Citations by Year