×

Efficient benchmarking of algorithm configurators via model-based surrogates. (English) Zbl 1457.68341

Summary: The optimization of algorithm (hyper-)parameters is crucial for achieving peak performance across a wide range of domains, ranging from deep neural networks to solvers for hard combinatorial problems. However, the proper evaluation of new algorithm configuration (AC) procedures (or configurators) is hindered by two key hurdles. First, AC scenarios are hard to set up, including the target algorithm to be optimized and the problem instances to be solved. Second, and even more significantly, they are computationally expensive: a single configurator run involves many costly runs of the target algorithm. Here, we propose a benchmarking approach that uses surrogate scenarios, which are computationally cheap while remaining close to the original AC scenarios. These surrogate scenarios approximate the response surface corresponding to true target algorithm performance using a regression model. In our experiments, we construct and evaluate surrogate scenarios for hyperparameter optimization as well as for AC problems that involve performance optimization of solvers for hard combinatorial problems. We generalize previous work by building surrogates for AC scenarios with multiple problem instances, stochastic target algorithms and censored running time observations. We show that our surrogate scenarios capture overall important characteristics of the original AC scenarios from which they were derived, while being much easier to use and orders of magnitude cheaper to evaluate.

MSC:

68W40 Analysis of algorithms
PDFBibTeX XMLCite
Full Text: DOI

References:

[1] Ahmadizadeh, K., Dilkina, B., Gomes, C., & Sabharwal, A. (2010). An empirical study of optimization for maximizing diffusion in networks. In D. Cohen (Ed.) Proceedings of the international conference on principles and practice of constraint programming (CP’10), Lecture Notes in Computer Science (Vol. 6308, pp. 514-521). Berlin: Springer.
[2] Ansel, J., Kamil, S., Veeramachaneni, K., Ragan-Kelley, J., Bosboom, J., O’Reilly, U., & Amarasinghe, S. (2014). Opentuner: An extensible framework for program autotuning. In J. Amaral, & J. Torrellas (Eds.) International conference on parallel architectures and compilation (pp. 303-316). New York: ACM.
[3] Ansótegui, C., Sellmann, M., & Tierney, K. (2009). A gender-based genetic algorithm for the automatic configuration of algorithms. In I. Gent (Ed.) Proceedings of the fifteenth international conference on principles and practice of constraint programming (CP’09) Lecture Notes in Computer Science (Vol. 5732, pp. 142-157). Berlin: Springer.
[4] Ansótegui, C., Malitsky, Y., Sellmann, M., & Tierney, K. (2015). Model-based genetic algorithms for algorithm configuration. In Q. Yang, & M. Wooldridge (Eds.) Proceedings of the 25th international joint conference on artificial intelligence (IJCAI’15) (pp. 733-739).
[5] Arbelaez, A., Truchet, C., & O’Sullivan, B. (2016). Learning sequential and parallel runtime distributions for randomized algorithms. In Proceedings of the international conference on tools with artificial intelligence (ICTAI).
[6] Balint, A., & Schöning, U. (2012). Choosing probability distributions for stochastic local search and the role of make versus break. In A. Cimatti, & R. Sebastiani (Eds.) Proceedings of the fifteenth international conference on theory and applications of satisfiability testing (SAT’12), Lecture Notes in Computer Science (Vol. 7317, pp. 16-29). Berlin:Springer. · Zbl 1273.68333
[7] Bardenet, R., Brendel, M., Kégl, B., & Sebag, M. (2014). Collaborative hyperparameter tuning. In S. Dasgupta, & D. McAllester (Eds.) Proceedings of the 30th international conference on machine learning (ICML’13) (pp. 199-207). Madison: Omnipress.
[8] Bensusan, H., & Kalousis, A. (2001). Estimating the predictive accuracy of a classifier. In Proceedings of the 12th European conference on machine learning (ECML) (pp. 25-36). Berlin: Springer. · Zbl 1007.68539
[9] Bergstra, J., Yamins, D., & Cox, D. (2014). Making a science of model search: Hyperparameter optimization in hundreds of dimensions for vision architectures. In S. Dasgupta, & D. McAllester (Eds.) Proceedings of the 30th international conference on machine learning (ICML’13) (pp. 115-123). Madison: Omnipress.
[10] Biere, A. (2013). Lingeling, plingeling and treengeling entering the sat competition 2013. In A. Balint, A. Belov, M. Heule, & M. Järvisalo (Eds.) Proceedings of SAT competition 2013: Solver and benchmark descriptions (Vol. B-2013-1, pp. 51-52). Helsinki: University of Helsinki, Department of Computer Science Series of Publications B.
[11] Biere, A. (2014). Yet another local search solver and Lingeling and friends entering the SAT competition 2014. In A. Belov, D. Diepold, M. Heule, & M. Järvisalo (Eds.) Proceedings of SAT competition 2014: Solver and benchmark descriptions (Vol. B-2014-2, pp. 39-40). Helsinki: University of Helsinki, Department of Computer Science Series of Publications B.
[12] Birattari, M., Stützle, T., Paquete, L., & Varrentrapp, K. (2002). A racing algorithm for configuring metaheuristics. In W. Langdon, E. Cantu-Paz, K. Mathias, R. Roy, D. Davis, R. Poli, K. Balakrishnan, V. Honavar, G. Rudolph, J. Wegener, L. Bull, M. Potter, A. Schultz, J. Miller, E. Burke, & N. Jonoska (Eds.) Proceedings of the genetic and evolutionary computation conference (GECCO’02) (pp. 11-18). Burlington: Morgan Kaufmann Publishers.
[13] Bischl, B., Kerschke, P., Kotthoff, L., Lindauer, M., Malitsky, Y., Frechétte, A., et al. (2016). ASlib: A benchmark library for algorithm selection. Artificial Intelligence, 237, 41-58. · Zbl 1357.68202 · doi:10.1016/j.artint.2016.04.003
[14] Blundell, C., Cornebise, J., Kavukcuoglu, K., & Wierstra, D. (2015). Weight uncertainty in neural network. In F. Bach, & D. Blei (Eds.) Proceedings of the 32nd international conference on machine learning (ICML’15) (Vol. 37, pp. 1613-1622). Madison: Omnipress.
[15] Brazdil, P., Giraud-Carrier, C., Soares, C., & Vilalta, R. (2008). Metalearning: Applications to data mining (1st ed.). Berlin: Springer Publishing Company. · Zbl 1173.68625
[16] Breimann, L. (2001). Random forests. Machine Learning Journal, 45, 5-32. · Zbl 1007.68152 · doi:10.1023/A:1010933404324
[17] Brochu, E., Cora, V., & de Freitas, N. (2010). A tutorial on Bayesian optimization of expensive cost functions, with application to active user modeling and hierarchical reinforcement learning. Computing Research Repository (CoRR) abs/1012.2599.
[18] Brummayer, R., Lonsing, F., & Biere, A. (2012). Automated testing and debugging of SAT and QBF solvers. In A. Cimatti, & R. Sebastiani (Eds.) Proceedings of the fifteenth international conference on theory and applications of satisfiability testing (SAT’12), Lecture Notes in Computer Science (Vol. 7317, pp. 44-57). Berlin: Springer. · Zbl 1306.68155
[19] Chen, T., & Guestrin, C. (2016). Xgboost: A scalable tree boosting system. In B. Krishnapuram, M. Shah, A. Smola, C. Aggarwal, D. Shen, & R. Rastogi (Eds.) Proceedings of the 22nd ACM SIGKDD international conference on knowledge discovery and data mining (KDD) (pp. 785-794). New York: ACM.
[20] Collobert, R., Bengio, S., & Bengio, Y. (2002). A parallel mixture of svms for very large scale problems. Neural Computation, 14(5), 1105-1114. · Zbl 1003.68135 · doi:10.1162/089976602753633402
[21] Cortes, C., & Vapnik, V. (1995). Support-vector networks. Machine Learning, 20(3), 273-297. · Zbl 0831.68098
[22] Dixon, L., & Szegö, G. (1978). The global optimization problem: An introduction. Towards global optimization, 2, 1-15.
[23] Domhan, T., Springenberg, J. T., & Hutter, F. (2015). Speeding up automatic hyperparameter optimization of deep neural networks by extrapolation of learning curves. In Q. Yang, & M. Wooldridge (Eds.) Proceedings of the 25th international joint conference on artificial intelligence (IJCAI’15) (pp 3460-3468).
[24] Eén, N., & Sörensson, N. (2003). An extensible sat-solver. In E. Giunchiglia, & A. Tacchella (Eds.) Proceedings of the conference on theory and applications of satisfiability testing (SAT) Lecture Notes in Computer Science (Vol. 2919, pp. 502-518). Berlin: Springer. · Zbl 1204.68191
[25] Eggensperger, K., Feurer, M., Hutter, F., Bergstra, J., Snoek, J., Hoos, H., & Leyton-Brown, K. (2013). Towards an empirical foundation for assessing Bayesian optimization of hyperparameters. In NIPS workshop on Bayesian optimization in theory and practice (BayesOpt’13).
[26] Eggensperger, K., Hutter, F., Hoos, H., & Leyton-Brown, K. (2015). Efficient benchmarking of hyperparameter optimizers via surrogates. In B. Bonet, & S. Koenig (Eds.) Proceedings of the twenty-nineth national conference on artificial intelligence (AAAI’15) (pp. 1114-1120). AAAI Press.
[27] Fawcett, C., & Hoos, H. (2016). Analysing differences between algorithm configurations through ablation. Journal of Heuristics, 22(4), 431-458. · doi:10.1007/s10732-014-9275-9
[28] Fawcett, C., Vallati, M., Hutter, F., Hoffmann, J., Hoos, H., & Leyton-Brown, K. (2014). Improved features for runtime prediction of domain-independent planners. In S. Chien, D. Minh, A. Fern, & W. Ruml (Eds.) Proceedings of the twenty-fourth international conference on automated planning and scheduling (ICAPS-14), AAAI.
[29] Feurer, M., Klein, A., Eggensperger, K., Springenberg, J. T., & Blum, M., Hutter, F. (2015a). Efficient and robust automated machine learning. In C. Cortes, N. Lawrence, D. Lee, M. Sugiyama, & R. Garnett (Eds.) Proceedings of the 29th international conference on advances in neural information processing systems (NIPS’15) (pp. 2962-2970).
[30] Feurer, M., Springenberg, T., & Hutter, F. (2015b). Initializing Bayesian hyperparameter optimization via meta-learning. In B. Bonet, & S. Koenig (Eds.) Proceedings of the twenty-nineth national conference on artificial intelligence (AAAI’15) (pp. 1128-1135). AAAI Press.
[31] Gama, J., & Brazdil, P. (1995). Characterization of classification algorithms. In Proceedings of the 7th Portuguese conference on artificial intelligence (pp. 189-200). Berlin: Springer.
[32] Gebser, M., Kaminski, R., Kaufmann, B., Schaub, T., Schneider, M., & Ziller, S. (2011). A portfolio solver for answer set programming: Preliminary report. In J. Delgrande, & W. Faber (Eds.) Proceedings of the eleventh international conference on logic programming and nonmonotonic reasoning (LPNMR’11), Lecture Notes in Computer Science (Vol. 6645, pp. 352-357). Berlin: Springer.
[33] Gebser, M., Kaufmann, B., & Schaub, T. (2012). Conflict-driven answer set solving: From theory to practice. Artificial Intelligence, 187-188, 52-89. · Zbl 1251.68060 · doi:10.1016/j.artint.2012.04.001
[34] Gelbart, M., Snoek, J., & Adams, R. (2014). Bayesian optimization with unknown constraints. In N. Zhang, & J. Tian (Eds.) Proceedings of the 30th conference on uncertainty in artificial intelligence (UAI’14), AUAI Press.
[35] Gerevini, A., & Serina, I. (2002). LPG: A planner based on local search for planning graphs with action costs. In M. Ghallab, J. Hertzberg, & P. Traverso (Eds.) Proceedings of the sixth international conference on artificial intelligence (pp. 13-22). Cambridge: The MIT Press.
[36] Gorissen, D., Couckuyt, I., Demeester, P., Dhaene, T., & Crombecq, K. (2010). A surrogate modeling and adaptive sampling toolbox for computer based design. Journal of Machine Learning Research, 11, 2051-2055.
[37] Guerra, S., Prudêncio, R., & Ludermir, T. (2008). Predicting the performance of learning algorithms using support vector machines as meta-regressors. In V. Kurkova-Pohlova, & J. Koutnik (Eds) International conference on artificial neural networks (ICANN’08) (vol. 18, pp. 523-532). Berlin: Springer.
[38] Hoos, H. (2017). Empirical algorithmics. Cambridge: Cambridge University Press. to appear.
[39] Hoos, H., & Stützle, T. (2004). Stochastic local search: Foundations and applications. Burlington: Morgan Kaufmann Publishers Inc. · Zbl 1126.68032
[40] Hoos, H., Lindauer, M., & Schaub, T. (2014). claspfolio 2: Advances in algorithm selection for answer set programming. Theory and Practice of Logic Programming, 14, 569-585. · Zbl 1307.68016 · doi:10.1017/S1471068414000210
[41] Hutter, F., Babić, D., Hoos, H., & Hu, A. (2007). Boosting verification by automatic tuning of decision procedures. In L. O’Conner (Ed.) Formal methods in computer aided design (FMCAD’07) (pp. 27-34). Washington, DC: IEEE Computer Society Press.
[42] Hutter, F., Hoos, H., Leyton-Brown, K., & Stützle, T. (2009). ParamILS: An automatic algorithm configuration framework. Journal of Artificial Intelligence Research, 36, 267-306. · Zbl 1192.68831
[43] Hutter, F., Hoos, H., & Leyton-Brown, K. (2010). Automated configuration of mixed integer programming solvers. In A. Lodi, M. Milano, & P. Toth (Eds.) Proceedings of the seventh international conference on integration of AI and OR techniques in constraint programming (CPAIOR’10) Lecture Notes in Computer Science (Vol. 6140, pp. 186-202). Berlin: Springer.
[44] Hutter, F., Hoos, H., & Leyton-Brown, K. (2011a). Bayesian optimization with censored response data. In NIPS workshop on Bayesian optimization, sequential experimental design, and bandits (BayesOpt’11).
[45] Hutter, F., Hoos, H., & Leyton-Brown, K, (2011b). Sequential model-based optimization for general algorithm configuration. In C. Coello (Ed.) Proceedings of the fifth international conference on learning and intelligent optimization (LION’11) Lecture Notes in Computer Science (Vol. 6683, pp. 507-523). Belin: Springer.
[46] Hutter, F., López-Ibánez, M., Fawcett, C., Lindauer, M., Hoos, H., Leyton-Brown, K., & Stützle, T. (2014a). Aclib: A benchmark library for algorithm configuration. In P. Pardalos, & M. Resende (Eds.) Proceedings of the eighth international conference on learning and intelligent optimization (LION’14) Lecture Notes in Computer Science. Berlin: Springer.
[47] Hutter, F., Xu, L., Hoos, H., & Leyton-Brown, K. (2014b). Algorithm runtime prediction: Methods and evaluation. Artificial Intelligence, 206, 79-111. · Zbl 1334.68185 · doi:10.1016/j.artint.2013.10.003
[48] Hutter, F., Lindauer, M., Balint, A., Bayless, S., Hoos, H., & Leyton-Brown, K. (2017). The configurable SAT solver challenge (CSSC). Artificial Intelligence, 243, 1-25. · Zbl 1402.68161 · doi:10.1016/j.artint.2016.09.006
[49] Kadioglu, S., Malitsky, Y., Sellmann, M., & Tierney, K. (2010). ISAC-instance-specific algorithm configuration. In H. Coelho, R. Studer, & M. Wooldridge (Eds.) Proceedings of the nineteenth european conference on artificial intelligence (ECAI’10) (pp. 751-756). Amsterdam: IOS Press.
[50] Koenker, R. (2005). Quantile regression. Cambridge: Cambridge University Press. · Zbl 1111.62037 · doi:10.1017/CBO9780511754098
[51] Kotthoff, L. (2014). Algorithm selection for combinatorial search problems: A survey. AI Magazine, 35, 48-60. · doi:10.1609/aimag.v35i3.2460
[52] Köpf, C., Taylor, C., & Keller, J. (2000). Meta-analysis: From data characterisation for meta-learning to meta-regression. In Proceedings of the PKDD-00 workshop on data mining, decision support, meta-learning and ILP.
[53] Krizhevsky, A., Sutskever, I., & Hinton, G. (2012). ImageNet classification with deep convolutional neural networks. In P. Bartlett, F. Pereira, C. Burges, L. Bottou, & K. Weinberger (Eds.) Proceedings of the 26th international conference on advances in neural information processing systems (NIPS’12) (pp 1097-1105).
[54] Lang, M., Kotthaus, H., Marwedel, P., Weihs, C., Rahnenführer, J., & Bischl, B. (2015). Automatic model selection for high-dimensional survival analysis. Journal of Statistical Computation and Simulation, 85, 62-76. · Zbl 1457.62350 · doi:10.1080/00949655.2014.929131
[55] Leite, R., & Brazdil, P. (2010). Active testing strategy to predict the best classification algorithm via sampling and metalearning. In H. Coelho, R. Studer, & M. Wooldridge (Eds.) Proceedings of the nineteenth European conference on artificial intelligence (ECAI’10) (pp. 309-314). Amsterdam: IOS Press.
[56] Leite, R., Brazdil, P., & Vanschoren, J. (2012). Selecting classification algorithms with active testing. In P. Perner (Ed.), Machine learning and data mining in pattern recognition Lecture Notes in Computer Science (pp. 117-131). Berlin: Springer.
[57] Leyton-Brown, K., Pearson, M., & Shoham, Y. (2000). Towards a universal test suite for combinatorial auction algorithms. In Proceedings of the international conference on economics and computation (pp. 66-76).
[58] Leyton-Brown, K., Nudelman, E., & Shoham, Y. (2009). Empirical hardness models: Methodology and a case study on combinatorial auctions. Journal of the ACM, 56(4), 22. · Zbl 1325.68110 · doi:10.1145/1538902.1538906
[59] Lierler, Y., & Schüller, P. (2012). Parsing combinatory categorial grammar via planning in answer set programming. In Lecture Notes in Computer Science (Vol. 7265, pp. 436-453). Berlin: Springer. · Zbl 1357.68256
[60] Lindauer, M., Hoos, H., Hutter, F., & Schaub, T. (2015). Autofolio: An automatically configured algorithm selector. Journal of Artificial Intelligence Research, 53, 745-778.
[61] Long, D., & Fox, M. (2003). The 3rd international planning competition: Results and analysis. Journal of Artificial Intelligence Research (JAIR), 20, 1-59. · Zbl 1036.68097 · doi:10.1023/A:1026044009832
[62] López-Ibáñez, M., Dubois-Lacoste, J., Caceres, L. P., Birattari, M., & Stützle, T. (2016). The irace package: Iterated racing for automatic algorithm configuration. Operations Research Perspectives, 3, 43-58. · doi:10.1016/j.orp.2016.09.002
[63] Loreggia, A., Malitsky, Y., Samulowitz, H., & Saraswat, V. (2016). Deep learning for algorithm portfolios. In D. Schuurmans, & M. Wellman (Eds.) Proceedings of the thirtieth national conference on artificial intelligence (AAAI’16) (pp. 1280-1286). AAAI Press.
[64] Malitsky, Y., Sabharwal, A., Samulowitz, H., & Sellmann, M. (2013). Algorithm portfolios based on cost-sensitive hierarchical clustering. In F. Rossi (Ed.) Proceedings of the 23rd international joint conference on artificial intelligence (IJCAI’13) (pp. 608-614).
[65] Manthey, N., & Lindauer, M. (2016). Spybug: Automated bug detection in the configuration space of SAT solvers. In Proceedings of the international conference on theory and applications of satisfiability testing (SAT) (pp. 554-561). · Zbl 1475.68443
[66] Manthey, N., & Steinke, P. (2014). Too many rooks. In A. Belov, D. Diepold, M. Heule, & M. Järvisalo (Eds.) Proceedings of SAT competition 2014: Solver and benchmark descriptions (Vol. B-2014-2, pp. 97-98). University of Helsinki, Department of Computer Science Series of Publications B.
[67] Maratea, M., Pulina, L., & Ricca, F. (2014). A multi-engine approach to answer-set programming. Theory and Practice of Logic Programming, 14, 841-868. · doi:10.1017/S1471068413000094
[68] Maron, O., & Moore, A. (1994). Hoeffding races: Accelerating model selection search for classification and function approximation. In Proceedings of the 6th international conference on advances in neural information processing systems (NIPS’94) (pp. 59-66). Burlington: Morgan Kaufmann Publishers.
[69] Meinshausen, N. (2006). Quantile regression forests. Journal of Machine Learning Research, 7, 983-999. · Zbl 1222.68262
[70] Neal, R. (1995). Bayesian learning for neural networks. PhD thesis, University of Toronto, Toronto, Canada. · Zbl 0888.62021
[71] Nudelman, E., Leyton-Brown, K., Andrew, G., Gomes, C., McFadden, J., Selman, B., & Shoham, Y. (2003). Satzilla 0.9, solver description. International SAT Competition.
[72] Nudelman, E., Leyton-Brown, K., Devkar, A., Shoham, Y., & Hoos, H. (2004). Understanding random SAT: Beyond the clauses-to-variables ratio. In International conference on principles and practice of constraint programming (CP’04) (pp. 438-452). · Zbl 1152.68569
[73] Oh, C. (2014). Minisat hack 999ed, minisat hack 1430ed and swdia5by. In A. Belov, D. Diepold, M. Heule, & M. Järvisalo (Eds.) Proceedings of SAT competition 2014: solver and benchmark descriptions (Vol. B-2014-2, p. 46). University of Helsinki, Department of Computer Science Series of Publications B.
[74] Penberthy, J., & Weld, D. (1994). Temporal planning with continuous change. In B. Hayes-Roth, & R. Korf (Eds.) Proceedings of the 12th national conference on artificial intelligence (pp. 1010-1015). Cambridge: The MIT Press.
[75] Rasmussen, C., & Williams, C. (2006). Gaussian processes for machine learning. Cambridge: The MIT Press. · Zbl 1177.68165
[76] Reif, M., Shafait, F., Goldstein, M., Breuel, T., & Dengel, A. (2014). Automatic classifier selection for non-experts. Pattern Analysis and Applications, 17(1), 83-96. · doi:10.1007/s10044-012-0280-z
[77] Rice, J. (1976). The algorithm selection problem. Advances in Computers, 15, 65-118. · doi:10.1016/S0065-2458(08)60520-3
[78] Sacks, J., Welch, W., Welch, T., & Wynn, H. (1989). Design and analysis of computer experiments. Statistical Science, 4(4), 409-423. · Zbl 0955.62619 · doi:10.1214/ss/1177012413
[79] Santner, T., Williams, B., & Notz, W. (2003). The design and analysis of computer experiments. Berlin: Springer. · Zbl 1041.62068 · doi:10.1007/978-1-4757-3799-8
[80] Sarkar, A., Guo, J., Siegmund, N., Apel, S., & Czarnecki, K. (2015). Cost-efficient sampling for performance prediction of configurable systems. In M. Cohen, L. Grunske, & M. Whalen (Eds.) 30th IEEE/ACM International Conference on Automated Software Engineering (pp. 342-352). IEEE.
[81] Schilling, N., Wistuba, M., Drumond, L., & Schmidt-Thieme, L. (2015). Hyperparameter optimization with factorized multilayer perceptrons. In Machine learning and knowledge discovery in databases (pp. 87-103). Berlin: Springer.
[82] Schmee, J., & Hahn, G. (1979). A simple method for regression analysis with censored data. Technometrics, 21, 417-432. · doi:10.1080/00401706.1979.10489811
[83] Shahriari, B., Swersky, K., Wang, Z., Adams, R., & de Freitas, N. (2016). Taking the human out of the loop: A review of Bayesian optimization. Proceedings of the IEEE, 104(1), 148-175. · doi:10.1109/JPROC.2015.2494218
[84] Silverthorn, B., Lierler, Y., & Schneider, M. (2012). Surviving solver sensitivity: An ASP practitioner’s guide. In A. Dovier, & V. Santos Costa (Eds.) Technical communications of the twenty-eighth international conference on logic programming (ICLP’12), Leibniz international proceedings in informatics (LIPIcs) (Vol. 17, pp. 164-175).
[85] Snoek, J., Larochelle, H., & Adams, RP. (2012). Practical Bayesian optimization of machine learning algorithms. In P. Bartlett, F. Pereira, C. Burges, L. Bottou, & K. Weinberger (Eds.) Proceedings of the 26th international conference on advances in neural information processing systems (NIPS’12) (pp. 2960-2968).
[86] Snoek J, Rippel O, Swersky K, Kiros R, Satish N, Sundaram N, Patwary M, Prabhat, Adams R (2015). Scalable Bayesian optimization using deep neural networks. In F. Bach, & D. Blei (Eds.) Proceedings of the 32nd international conference on machine learning (ICML’15) (Vol. 37, pp. 2171-2180). Madison: Omnipress.
[87] Soares, C., & Brazdil, P. (2004). A meta-learning method to select the kernel width in support vector regression. Machine Learning Journal, 54, 195-209. · Zbl 1101.68083 · doi:10.1023/B:MACH.0000015879.28004.9b
[88] Spearman, C. (1904). The proof and measurement of association between two things. American Journal of Psychology, 15, 71-101.
[89] Springenberg, J., Klein, A., Falkner, S., & Hutter, F. (2016). Bayesian optimization with robust Bayesian neural networks. In Proceedings of the international conference on advances in neural information processing systems (NIPS’16).
[90] Takeuchi, I., Le, Q., Sears, T., & Smola, A. (2006). Nonparametric quantile estimation. Journal of Machine Learning Research, 7, 1231-1264. · Zbl 1222.68316
[91] Thornton, C., Hutter, F., Hoos, H., & Leyton-Brown, K. (2013). Auto-WEKA: Combined selection and hyperparameter optimization of classification algorithms. In I. Dhillon, Y. Koren, R. Ghani, T. Senator, P. Bradley, R. Parekh, J. He, R. Grossman, & R. Uthurusamy (Eds.) The 19th ACM SIGKDD international conference on knowledge discovery and data mining (KDD’13) (pp 847-855). New York: ACM Press
[92] Vallati, M., Fawcett, C., Gerevini, A., Hoos, H., & Saetti, A. (2013). Automatic generation of efficient domain-optimized planners from generic parametrized planners. In M. Helmert, & G. Röger (Eds.) Proceedings of the sixth annual symposium on combinatorial search (SOCS’14), AAAI Press.
[93] Wistuba, M., Schilling, N., & Schmidt-Thieme, L. (2015). Learning hyperparameter optimization initializations. In Proceedings of the international conference on data science and advanced analytics (DSAA) (pp. 1-10). IEEE
[94] Xu, L., Hutter, F., Hoos, H., & Leyton-Brown, K. (2008). SATzilla: Portfolio-based algorithm selection for SAT. Journal of Artificial Intelligence Research, 32, 565-606. · Zbl 1182.68272
[95] Xu, L., Hutter, F., Hoos, H., & Leyton-Brown, K. (2011). Hydra-MIP: Automated algorithm configuration and selection for mixed integer programming. In RCRA workshop on Experimental Evaluation of Algorithms for Solving Problems with Combinatorial Explosion at the International Joint Conference on Artificial Intelligence (IJCAI).
This reference list is based on information provided by the publisher or from digital mathematics libraries. Its items are heuristically matched to zbMATH identifiers and may contain data conversion errors. In some cases that data have been complemented/enhanced by data from zbMATH Open. This attempts to reflect the references listed in the original paper as accurately as possible without claiming completeness or a perfect matching.