swMATH ID: 42958
Software Authors: Cowen-Rivers, Alexander I.; Lyu, Wenlong; Tutunov, Rasul; Wang, Zhi; Grosnit, Antoine; Griffiths, Ryan Rhys; Maraval, Alexandre Max; Jianye, Hao; Wang, Jun; Peters, Jan; Bou-Ammar, Haitham
Description: HEBO: Pushing the limits of sample-efficient hyper-parameter optimisation. In this work we rigorously analyse assumptions inherent to black-box optimisation hyper-parameter tuning tasks. Our results on the Bayesmark benchmark indicate that heteroscedasticity and non-stationarity pose significant challenges for black-box optimisers. Based on these findings, we propose a Heteroscedastic and Evolutionary Bayesian Optimisation solver (HEBO). HEBO performs non-linear input and output warping, admits exact marginal log-likelihood optimisation and is robust to the values of learned parameters. We demonstrate HEBO’s empirical efficacy on the NeurIPS 2020 Black-Box Optimisation challenge, where HEBO placed first. Upon further analysis, we observe that HEBO significantly outperforms existing black-box optimisers on 108 machine learning hyperparameter tuning tasks comprising the Bayesmark benchmark. Our findings indicate that the majority of hyper-parameter tuning tasks exhibit heteroscedasticity and non-stationarity, multiobjective acquisition ensembles with Pareto front solutions improve queried configurations, and robust acquisition maximisers afford empirical advantages relative to their non-robust counterparts. We hope these findings may serve as guiding principles for practitioners of Bayesian optimisation
Homepage: https://arxiv.org/abs/2012.03826
Source Code: https://github.com/huawei-noah/hebo
Dependencies: Python
Keywords: hyperparameter tuning; Bayesian optimization; automated reasoning; uncertainty
Related Software: BOHB; Hyperopt; Hyperband; BoTorch; pymoo; RoBO; GPflowOpt; Nevergrad; BOCK; OpenTuner; SMAC; pySOT; Adam; Spearmint; TensorFlow; Scikit; UCI-ml
Cited in: 1 Publication

Citations by Year