SGDR swMATH ID: 30752 Software Authors: Loshchilov, Ilya; Hutter, Frank Description: SGDR: Stochastic Gradient Descent with Warm Restarts. Restart techniques are common in gradient-free optimization to deal with multimodal functions. Partial warm restarts are also gaining popularity in gradient-based optimization to improve the rate of convergence in accelerated gradient schemes to deal with ill-conditioned functions. In this paper, we propose a simple warm restart technique for stochastic gradient descent to improve its anytime performance when training deep neural networks. We empirically study its performance on the CIFAR-10 and CIFAR-100 datasets, where we demonstrate new state-of-the-art results at 3.14 Homepage: https://arxiv.org/abs/1608.03983 Source Code: https://github.com/loshchil/SGDR Related Software: Adam; PyTorch; ImageNet; CIFAR; Python; U-Net; Tensor2Tensor; RMSprop; DeepONet; MixMatch; AlexNet; Fashion-MNIST; BERT; GitHub; AdaGrad; Xception; TensorFlow; GELUs; SciPy; RandAugment Cited in: 38 Documents all top 5 Cited by 148 Authors 3 Böhlke, Thomas 3 Gajek, Sebastian 3 Schneider, Matti 2 Yeo, Kyongmin 1 Adler, Jonas 1 Ainsworth, Mark 1 Andrade-Campos, António 1 Arridge, Simon R. 1 Banert, Sebastian 1 Bao, Chenglong 1 Baraniuk, Richard G. 1 Barbano, Riccardo 1 Berrada, Leonard 1 Bertin, Michaël 1 Bertozzi, Andrea Louise 1 Boning, Duane S. 1 Bonneel, Nicolas 1 Chatigny, Philippe 1 Chen, Songcan 1 Cheng, Guang 1 Chung, Kiryong 1 Clouteau, Didier 1 Courty, Nicolas 1 Cueto, Elias 1 Danilova, Marina 1 Deo, Indu Kant 1 Digne, Julie 1 Dong, Miao 1 Du, Weitao 1 Dvurechensky, Pavel E. 1 Egger, Jan 1 Favero, Alessandro 1 Fei, Fei 1 Figurnov, Michael 1 Gao, Rui 1 Gao, Yiming 1 Gasnikov, Aleksandr V. 1 Gatti, Filippo 1 Geiger, Mario 1 Georgieva, Petia 1 Gifford, Wesley M. 1 Givoli, Dan 1 Gong, Shiqi 1 Gorbunov, Eduard 1 Gorji, Mohammad Hossein 1 Grullon, Dylan E. C. 1 Guminov, Sergey 1 Guo, Hongwei 1 Hauptmann, Andreas S. 1 He, Juncai 1 He, Junjie 1 Holzapfel, Gerhard Alfred 1 Huang, Guangbin 1 Huang, Jinshu 1 Huang, Shengjun 1 Ikegawa, Shin-ichi 1 Jaiman, Rajeev Kumar 1 Jendersie, Robert 1 Jenny, Patrick 1 Jin, Bangti 1 Kahana, Adar 1 Kalagnanam, Jayant R. 1 Kamzolov, Dmitry 1 Kanamori, Takafumi 1 Karlsson, Johan 1 Karniadakis, George Em 1 Kereta, Željko 1 Kim, Chongam 1 Kim, Juhyun 1 Kumar, M. Pawan 1 Lacombe, Julien 1 Lehmann, Fanny 1 Lessig, Christian 1 Li, Congqiao 1 Li, Shao-Yuan 1 Li, Wenjie 1 Li, Zan 1 Li, Zhengyi 1 Liang, Zenglong 1 Liao, Qifeng 1 Lin, Shan 1 Lin, Zhouchen 1 Liu, Hongsheng 1 Liu, Tie-Yan 1 Lourenço, Rúben 1 Ma, Zhi-Ming 1 Margenberg, Nils 1 Meng, Qi 1 Meng, Qingyan 1 Mnih, Andriy 1 Mohamed, Shakir 1 Nakagawa, Takumi 1 Nguyen, Tan Mai 1 Öktem, Ozan 1 Osher, Stanley Joel 1 Ovadia, Oded 1 Paren, Alasdair 1 Patenaude, Jean-Marc 1 Pepe, Antonio 1 Petrini, Leonardo ...and 48 more Authors all top 5 Cited in 19 Serials 6 Computer Methods in Applied Mechanics and Engineering 5 SIAM Journal on Scientific Computing 4 Neural Networks 3 Journal of Computational Physics 2 Inverse Problems 2 Machine Learning 2 Journal of Machine Learning Research (JMLR) 1 Journal of the Mechanics and Physics of Solids 1 Computational Mechanics 1 International Journal of Approximate Reasoning 1 SIAM Journal on Optimization 1 Journal of Mathematical Imaging and Vision 1 Engineering Analysis with Boundary Elements 1 Journal of High Energy Physics 1 Journal of Statistical Mechanics: Theory and Experiment 1 SIAM Journal on Imaging Sciences 1 Discrete and Continuous Dynamical Systems. Series S 1 Journal of the Operations Research Society of China 1 Research in the Mathematical Sciences all top 5 Cited in 19 Fields 26 Computer science (68-XX) 11 Numerical analysis (65-XX) 6 Statistics (62-XX) 6 Mechanics of deformable solids (74-XX) 6 Operations research, mathematical programming (90-XX) 5 Fluid mechanics (76-XX) 3 Partial differential equations (35-XX) 2 Calculus of variations and optimal control; optimization (49-XX) 2 Probability theory and stochastic processes (60-XX) 2 Statistical mechanics, structure of matter (82-XX) 2 Game theory, economics, finance, and other social and behavioral sciences (91-XX) 2 Biology and other natural sciences (92-XX) 2 Systems theory; control (93-XX) 2 Information and communication theory, circuits (94-XX) 1 Linear and multilinear algebra; matrix theory (15-XX) 1 Dynamical systems and ergodic theory (37-XX) 1 Operator theory (47-XX) 1 Quantum theory (81-XX) 1 Geophysics (86-XX) Citations by Year