Hyperband swMATH ID: 41120 Software Authors: L. Li, K. Jamieson, G. DeSalvo, A. Rostamizadeh, A. Talwalkar Description: Hyperband: a novel bandit-based approach to hyperparameter optimization. Performance of machine learning algorithms depends critically on identifying a good set of hyperparameters. While recent approaches use Bayesian optimization to adaptively select configurations, we focus on speeding up random search through adaptive resource allocation and early-stopping. We formulate hyperparameter optimization as a pure-exploration nonstochastic infinite-armed bandit problem where a predefined resource like iterations, data samples, or features is allocated to randomly sampled configurations. We introduce a novel algorithm, Hyperband, for this framework and analyze its theoretical properties, providing several desirable guarantees. Furthermore, we compare Hyperband with popular Bayesian optimization methods on a suite of hyperparameter optimization problems. We observe that Hyperband can provide over an order-of-magnitude speedup over our competitor set on a variety of deep-learning and kernel-based learning problems. Homepage: https://arxiv.org/abs/1603.06560 Source Code: https://github.com/thuijskens/scikit-hyperband Related Software: Spearmint; SMAC; Hyperopt; BOHB; Adam; GitHub; Scikit; PyTorch; BoTorch; AlexNet; ImageNet; auto-sklearn; TensorFlow; NOMAD; R; CIFAR; MNIST; RMSprop; Python; RoBO Cited in: 21 Documents all top 5 Cited by 75 Authors 2 Ammar, Haitham Bou 2 Cowen-Rivers, Alexander I. 2 Griffiths, Ryan-Rhys 2 Grosnit, Antoine 2 Lakhmiri, Dounia 2 Le Digabel, Sébastien 2 Tutunov, Rasul 2 Wang, Jun 1 Altschuler, Jason M. 1 Archetti, Francesco 1 Baldi, Pierre 1 Berg, Carl Fredrik 1 Biedenkapp, André 1 Binder, Martin 1 Bischl, Bernd 1 Brunel, Victor-Emmanuel 1 Calandra, Roberto 1 Candelieri, Antonio 1 Cartis, Coralia 1 Chamakh, Linda 1 Chawshin, Kurdistan 1 Chen, Xiaoli 1 De Loera, Jesús A. 1 DeSalvo, Giulia 1 Duan, Jinqiao 1 Eimer, Theresa 1 Elsheikh, Ahmed H. 1 Faust, Aleksandra 1 Gillen, Daniel L. 1 Gobet, Emmanuel 1 Haddock, Jamie 1 Hertel, Lars 1 Huber, Marco F. 1 Hutter, Frank 1 Jamieson, Kevin 1 Jianye, Hao 1 Karniadakis, George Em 1 Kotthoff, Lars 1 Lang, Michel 1 Li, Lisha 1 Lindauer, Marius 1 Lopez, Olivier 1 Lyu, Wenlong 1 Ma, Anna 1 Malek, Alan 1 Maraval, Alexandre Max 1 Miao, Yingjie 1 Moniz, Nuno 1 Needell, Deanna 1 Nomura, Masahiro 1 Onishi, Masaki 1 Ozaki, Yoshihiko 1 Parker-Holder, Jack 1 Perego, Riccardo 1 Peters, Jan 1 Pfisterer, Florian 1 Rajan, Raghu 1 Ribeiro, Rita P. 1 Roberts, Lindon 1 Rostamizadeh, Afshin 1 Schneider, Lennart 1 Sheridan-Methven, Oliver 1 Song, Xingyou 1 Talwalkar, Ameet 1 Tanigaki, Yuki 1 Tribes, Christophe 1 Varagnolo, Damiano 1 Wang, Zhi 1 Watanabe, Shuhei 1 Yang, Zebin 1 Zabaras, Nicholas J. 1 Zhang, Aijun 1 Zhang, Baohe 1 Zhu, Yinhao 1 Zöller, Marc-André all top 5 Cited in 12 Serials 6 Journal of Machine Learning Research (JMLR) 4 The Journal of Artificial Intelligence Research (JAIR) 2 Journal of Computational Physics 1 ACM Transactions on Mathematical Software 1 Optimization 1 European Journal of Applied Mathematics 1 Machine Learning 1 Journal of Global Optimization 1 Annals of Mathematics and Artificial Intelligence 1 Computational Geosciences 1 Journal of Computational and Graphical Statistics 1 SN Operations Research Forum all top 5 Cited in 7 Fields 17 Computer science (68-XX) 5 Statistics (62-XX) 5 Operations research, mathematical programming (90-XX) 4 Numerical analysis (65-XX) 2 Partial differential equations (35-XX) 2 Fluid mechanics (76-XX) 1 Geophysics (86-XX) Citations by Year