SGDR swMATH ID: 30752 Software Authors: Ilya Loshchilov, Frank Hutter Description: SGDR: Stochastic Gradient Descent with Warm Restarts. Restart techniques are common in gradient-free optimization to deal with multimodal functions. Partial warm restarts are also gaining popularity in gradient-based optimization to improve the rate of convergence in accelerated gradient schemes to deal with ill-conditioned functions. In this paper, we propose a simple warm restart technique for stochastic gradient descent to improve its anytime performance when training deep neural networks. We empirically study its performance on the CIFAR-10 and CIFAR-100 datasets, where we demonstrate new state-of-the-art results at 3.14 Homepage: https://arxiv.org/abs/1608.03983 Source Code: https://github.com/loshchil/SGDR Related Software: Adam; PyTorch; ImageNet; CIFAR; Python; MixMatch; mixup; RMSprop; RandAugment; ReMixMatch; FixMatch; MobileNetV2; DARTS; EfficientNet; TensorFlow; DeepONet; EnKF; SeqGAN; ADADELTA; GitHub Cited in: 16 Publications all top 5 Cited by 49 Authors 2 Böhlke, Thomas 2 Cheng, Ran 2 Gajek, Sebastian 2 Schneider, Matti 2 Yeo, Kyongmin 1 Adler, Jonas 1 Ahn, Changwook 1 Ainsworth, Mark 1 Bakhtin, Anton 1 Banert, Sebastian 1 Baraniuk, Richard G. 1 Bertozzi, Andrea Louise 1 Boning, Duane S. 1 Chatigny, Philippe 1 Chen, Songcan 1 Chen, Yiming 1 Deng, Yuntian 1 Figurnov, Michael 1 Gifford, Wesley M. 1 Gross, Sam 1 Grullon, Dylan E. C. 1 Huang, Shengjun 1 Kalagnanam, Jayant R. 1 Kang, Dongseok 1 Karlsson, Johan 1 Li, Shao-Yuan 1 Li, Zan 1 Mnih, Andriy 1 Mohamed, Shakir 1 Öktem, Ozan 1 Osher, Stanley Joel 1 Ott, Myle 1 Pan, Tianci 1 Patenaude, Jean-Marc 1 Ranzato, Marc’Aurelio 1 Rawson, Michael 1 Reger, Giles 1 Ringh, Axel 1 Rosca, Mihaela 1 Shi, Ye 1 Shin, Yeonjong 1 Sun, Fan-Keng 1 Sun, Ruoyu 1 Sun, Tao 1 Szlam, Arthur D. 1 Tan, Hao 1 Tang, Dexuan 1 Wang, Bao 1 Wang, Shengrui all top 5 Cited in 9 Serials 3 SIAM Journal on Scientific Computing 2 Journal of Machine Learning Research (JMLR) 1 Computer Methods in Applied Mechanics and Engineering 1 Computational Mechanics 1 International Journal of Approximate Reasoning 1 Machine Learning 1 SIAM Journal on Optimization 1 SIAM Journal on Imaging Sciences 1 Journal of the Operations Research Society of China all top 5 Cited in 8 Fields 13 Computer science (68-XX) 4 Numerical analysis (65-XX) 4 Operations research, mathematical programming (90-XX) 3 Statistics (62-XX) 2 Mechanics of deformable solids (74-XX) 1 Dynamical systems and ergodic theory (37-XX) 1 Operator theory (47-XX) 1 Systems theory; control (93-XX) Citations by Year