Beiranvand, Vahid; Hare, Warren; Lucet, Yves Best practices for comparing optimization algorithms. (English) Zbl 1390.90601 Optim. Eng. 18, No. 4, 815-848 (2017). Summary: Comparing, or benchmarking, of optimization algorithms is a complicated task that involves many subtle considerations to yield a fair and unbiased evaluation. In this paper, we systematically review the benchmarking process of optimization algorithms, and discuss the challenges of fair comparison. We provide suggestions for each step of the comparison process and highlight the pitfalls to avoid when evaluating the performance of optimization algorithms. We also discuss various methods of reporting the benchmarking results. Finally, some suggestions for future research are presented to improve the current benchmarking process. Cited in 17 Documents MSC: 90C60 Abstract computational complexity for mathematical programming problems 68W40 Analysis of algorithms 90-08 Computational methods for problems pertaining to operations research and mathematical programming Keywords:benchmarking; algorithm comparison; guidelines; performance; software; testing; metric; timing; optimization algorithms Software:CUTEst; CUTE; Optimization Test Environment; OPAL; Algorithm 709; MIPLIB; Benchmarks for Optimization Software; PSwarm; CUTEr; COPS; AMPL; LANCELOT; LIBOPT; GraphBase; MINPACK-2; CONDOR; GAMS; COCONUT; testenv; PAVER; Paramils; L-BFGS; MLMSRBF; CPLEX; MultiMin; SifDec; minpack; pyMDO; Algorithm 829 PDFBibTeX XMLCite \textit{V. Beiranvand} et al., Optim. Eng. 18, No. 4, 815--848 (2017; Zbl 1390.90601) Full Text: DOI arXiv References: [1] Addis, B; Locatelli, M, A new class of test functions for global optimization, J Glob Optim, 38, 479-501, (2007) · Zbl 1180.90302 [2] Ali, MM; Khompatraporn, C; Zabinsky, ZB, A numerical evaluation of several stochastic algorithms on selected continuous global optimization test problems, J Glob Optim, 31, 635-672, (2005) · Zbl 1093.90028 [3] Andrei, N, An unconstrained optimization test functions collection, Adv Model Optim, 10, 147-161, (2008) · Zbl 1161.90486 [4] Asaadi, J, A computational comparison of some non-linear programs, Math Program, 4, 144-154, (1973) · Zbl 0259.90044 [5] Audet, C; Orban, D, Finding optimal algorithmic parameters using derivative-free optimization, SIAM J Optim, 17, 642-664, (2006) · Zbl 1128.90060 [6] Audet, C; Dang, CK; Orban, D; Ken, N (ed.); Keita, T (ed.); John, C (ed.); Reiji, S (ed.), Algorithmic parameter optimization of the DFO method with the OPAL framework, 255-274, (2010), New York [7] Audet, C; Dang, K-C; Orban, D, Optimization of algorithms with OPAL, Math Program Comput, 6, 233-254, (2014) · Zbl 1323.65063 [8] Audet C, Le Digabel S, Peyrega M (2014b) Linear equalities in blackbox optimization. Technical report, Les Cahiers du GERAD · Zbl 1311.90185 [9] Averick BM, Carter RG, Moré JJ (1991) The MINPACK-2 test problem collection. Technical report, Argonne National Laboratory, Argonne [10] Balint, A; Gall, D; Kapler, G; Retz, R, Experiment design and administration for computer clusters for SAT-solvers (EDACC), system description, J Satisf Boolean Model Comput, 7, 77-82, (2010) [11] Bard, Y, Comparison of gradient methods for the solution of nonlinear parameter estimation problems, SIAM J Numer Anal, 7, 157-186, (1970) · Zbl 0202.16904 [12] Barr, RS; Hickman, BL, Reporting computational experiments with parallel algorithms: issues, measures, and experts’ opinions, INFORMS J Comput, 5, 2-18, (1993) · Zbl 0775.65029 [13] Barr, RS; Golden, BL; Kelly, JP; Resende, MGC; Stewart, WR, Designing and reporting on computational experiments with heuristic methods, J Heuristics, 1, 9-32, (1995) · Zbl 0853.68154 [14] Barton RR (1987) Testing strategies for simulation optimization. In Proceedings of the 19th conference on winter simulation, WSC’87, New York, NY, USA. ACM, pp 391-401 · Zbl 1387.90196 [15] Bartz-Beielstein, T; Preuss, M; Borenstein, Y (ed.); Moraglio, A (ed.), Experimental analysis of optimization algorithms: tuning and beyond, 205-245, (2014), Berlin [16] Bartz-Beielstein T, Lasarczyk CWG, Preuss M (2005) Sequential parameter optimization. In: The 2005 IEEE congress on evolutionary computation, vol 1, pp 773-780 · Zbl 1208.90166 [17] Baz M, Hunsaker B, Brooks P, Gosavi A (2007) Automated tuning of optimization software parameters. Technical report, University of Pittsburgh, Department of Industrial Engineering [18] Beiranvand V, Hare W, Lucet Y, Hossain S (2015) Multi-haul quasi network flow model for vertical alignment optimization. Technical report, Computer Science, University of British Columbia, Kelowna, BC, Canada · Zbl 1329.90112 [19] Beltrami, EJ; Beckmann, M (ed.); Künzi, HP (ed.), A comparison of some recent iterative methods for the numerical solution of nonlinear programs, No. 14, 20-29, (1969), Berlin [20] Benson, HY; Shanno, DF; Vanderbei, RJ; Pillo, G (ed.); Murli, A (ed.), A comparative study of large-scale nonlinear optimization algorithms, No. 82, 95-127, (2003), New York [21] Benson, HY; Shanno, DF; Vanderbei, RJ, Interior-point methods for nonconvex nonlinear programming: jamming and numerical testing, Math Progr, 99, 35-48, (2004) · Zbl 1055.90068 [22] Berthold, T, Measuring the impact of primal heuristics, Oper Res Lett, 41, 611-614, (2013) · Zbl 1287.90037 [23] Billups, SC; Dirkse, SP; Ferris, MC, A comparison of large scale mixed complementarity problem solvers, Comput Optim Appl, 7, 3-25, (1997) · Zbl 0883.90116 [24] Birattari M (2009) Tuning metaheuristics: a machine learning perspective. Springer, Berlin (1st ed. 2005. 2nd printing edition) · Zbl 1183.68464 [25] Bondarenko AS, Bortz DM, Moré JJ (1999) COPS: large-scale nonlinearly constrained optimization problems. Technical report, Mathematics and Computer Science Division, Argonne National Laboratory. Technical report ANL/MCS-TM-237 [26] Bongartz, I; Conn, AR; Gould, N; Toint, PL, CUTE: constrained and unconstrained testing environment, ACM Trans Math Softw, 21, 123-160, (1995) · Zbl 0886.65058 [27] Bongartz I, Conn AR, Gould NIM, Saunders MA, Toint PL (1997) A numerical comparison between the LANCELOT and MINOS packages for large scale constrained optimization. Technical report, SCAN-9711063 [28] Box, MJ, A comparison of several current optimization methods, and the use of transformations in constrained problems, Comput J, 9, 67-77, (1966) · Zbl 0146.13304 [29] Buckley, AG, Algorithm 709: testing algorithm implementations, ACM Trans Math Softw, 18, 375-391, (1992) · Zbl 0892.65034 [30] Bussieck, MR; Drud, AS; Meeraus, A; Pruessner, A; Bliek, C (ed.); Jermann, C (ed.); Neumaier, A (ed.), Quality assurance and global optimization, No. 2861, 223-238, (2003), Berlin [31] Bussieck, MR; Dirkse, SP; Vigerske, S, PAVER 2.0: an open source environment for automated performance analysis of benchmarking data, J Glob Optim, 59, 259-275, (2014) · Zbl 1300.90003 [32] CPLEX’s automatic tuning tool. Technical report, IBM Corporation, 2014 [33] Colville AR (1968) A comparative study of nonlinear programming codes. Technical report 320-2949, IBM Scientific Center, New York · Zbl 1241.90192 [34] Conn, AR; Gould, N; Toint, PL, Numerical experiments with the LANCELOT package (release A) for large-scale nonlinear optimization, Math Program, 73, 73-110, (1996) · Zbl 0848.90109 [35] Crowder, H; Dembo, RS; Mulvey, JM, On reporting computational experiments with mathematical software, ACM Trans Math Softw, 5, 193-203, (1979) [36] Dannenbring, DG, Procedures for estimating optimal solution values for large combinatorial problems, Manag Sci, 23, 1273-1283, (1977) · Zbl 0377.90051 [37] Dembo, RS, A set of geometric programming test problems and their solutions, Math Program, 10, 192-213, (1976) · Zbl 0349.90066 [38] Dembo, RS, Current state of the art of algorithms and computer software for geometric programming, J Optim Theory Appl, 26, 149-183, (1978) · Zbl 0369.90121 [39] Derigs, U, Using confidence limits for the global optimum in combinatorial optimization, Oper Res, 33, 1024-1049, (1985) · Zbl 0587.90070 [40] Dixon LCW, Szegö GP (1978) Towards global optimisation 2. North-Holland, Amsterdam · Zbl 0385.00011 [41] Dolan ED, Moré JJ (2000) Benchmarking optimization software with COPS. Technical report, Argonne National Laboratory research report [42] Dolan, ED; Moré, JJ, Benchmarking optimization software with performance profiles, Math Program, 91, 201-213, (2002) · Zbl 1049.90004 [43] Dolan ED, Moré JJ (2004) Benchmarking optimization software with COPS 3.0. Argonne National Laboratory research report · Zbl 1348.90010 [44] Domes, F; Fuchs, M; Schichl, H; Neumaier, A, The optimization test environment, Optim Eng, 15, 443-468, (2014) · Zbl 1364.90006 [45] Eason, ED; Mulvey, JM (ed.), Evidence of fundamental difficulties in nonlinear optimization code comparisons, No. 199, 60-71, (1982), Berlin [46] Eason, ED; Fenton, RG, A comparison of numerical optimization methods for engineering design, J Manuf Sci Eng, 96, 196-200, (1974) [47] Elam, JJ; Klingman, D; Mulvey, JM (ed.), NETGEN-II: a system for generating structured network-based mathematical programming test problems, No. 199, 16-23, (1982), Berlin [48] Evtushenko YG (1985) Numerical optimization techniques. Translation series in mathematics and engineering. Optimization Software, Inc., Publications Division, New York (distributed by Springer, New York, Translated from the Russian, Translation edited and with a foreword by J. Stoer) [49] Famularo, D; Pugliese, P; Sergeyev, YD; Dzemyda, G (ed.); Šaltenis, V (ed.); Žilinskas, A (ed.), Test problems for Lipschitz univariate global optimization with multiextremal constraints, No. 59, 93-109, (2002), Dordrecht · Zbl 1211.90180 [50] Floudas CA, Pardalos PM (1990) A collection of test problems for constrained global optimization algorithms, vol 455. Springer, Berlin · Zbl 0718.90054 [51] Floudas CA, Pardalos PM, Adjiman CS, Esposito WR, Gëmës ZH, Harding ST, Klepeis JL, Meyer CA, Schweiger CA (1999) Handbook of test problems in local and global optimization. Springer, New York · Zbl 0943.90001 [52] Fourer R, Gay DM, Kernighan BW (2002) AMPL: a modeling language for mathematical programming, 2nd edn. Duxbury Press, Belmont · Zbl 0701.90062 [53] Fowler, KR; Reese, JP; Kees, CE; Dennis, JE; Kelley, CT; Miller, CT; Audet, C; Booker, AJ; Couture, G; Darwin, RW; Farthing, MW; Finkel, DE; Gablonsky, JM; Gray, G; Kolda, TG, Comparison of derivative-free optimization methods for groundwater supply and hydraulic capture community problems, Adv Water Resour, 31, 743-757, (2008) [54] Gaviano, M; Kvasov, DE; Lera, D; Sergeyev, YD, Algorithm 829: software for generation of classes of test functions with known local and global minima for global optimization, ACM Trans Math Softw, 29, 469-480, (2003) · Zbl 1068.90600 [55] Gilbert JC, Jonsson X (2009) LIBOPT an environment for testing solvers on heterogeneous collections of problems the manual, version 2.1. Technical report, INRIA, Le Chesnay [56] Gillard, JW; Kvasov, DE, Lipschitz optimization methods for Fitting a sum of damped sinusoids to a series of observations, Stat Interface, 10, 59-70, (2017) · Zbl 1387.90196 [57] Golden, BL; Stewart, WR; Lawler, EL (ed.); Lenstra, V (ed.); Rinnooy Kan, AHG (ed.); Shmoys, DB (ed.), Empirical analysis of heuristics, 207-249, (1985), New York [58] Gould, N; Scott, J, A note on performance profiles for benchmarking software, ACM Trans Math Softw, 43, 15:1-15:5, (2016) · Zbl 1369.65202 [59] Gould, N; Orban, D; Toint, PL, Cuter and sifdec: a constrained and unconstrained testing environment, revisited, ACM Trans Math Softw, 29, 373-394, (2003) · Zbl 1068.90526 [60] Gould, N; Orban, D; Toint, PL, Cutest: a constrained and unconstrained testing environment with safe threads for mathematical optimization, Comput Optim Appl, 60, 545-557, (2015) · Zbl 1325.90004 [61] Grishagin, VA; Floudas, CA (ed.); Pardalos, PM (ed.), Operating characteristics of some global search algorithms, No. 7, 198-206, (2009), Riga [62] Grundel, D; Jeffcoat, D; Floudas, CA (ed.); Pardalos, PM (ed.), Combinatorial test problems and problem generators, 391-394, (2009), New York [63] Hansen, P; Jaumard, B; Lu, SH, Global optimization of univariate Lipschitz functions: II. new algorithms and computational comparison, Math Program, 55, 273-292, (1992) · Zbl 0825.90756 [64] Hare, W; Planiden, C, The NC-proximal average for multiple functions, Optim Lett, 8, 849-860, (2014) · Zbl 1317.90277 [65] Hare, W; Sagastizábal, C, Benchmark of some nonsmooth optimization solvers for computing nonconvex proximal points, Pac J Optim, 2, 545-573, (2006) · Zbl 1124.90024 [66] Hare, W; Sagastizábal, C, A redistributed proximal bundle method for nonconvex optimization, SIAM J Optim, 20, 2442-2473, (2010) · Zbl 1211.90183 [67] Hare WL, Wang Y (2010) Fairer benchmarking of optimization algorithms via derivative free optimization. Technical report, optimization-online [68] Hare, WL; Koch, VR; Lucet, Y, Models and algorithms to improve earthwork operations in road design using mixed integer linear programming, Eur J Oper Res, 215, 470-480, (2011) · Zbl 1237.90276 [69] Hillstrom, KE, A simulation test approach to the evaluation of nonlinear optimization algorithms, ACM Trans Math Softw, 3, 305-315, (1977) [70] Hiriart-Urruty J-B, Lemaréchal C (1993) Convex analysis and minimization algorithms. II. Grundlehren der Mathematischen Wissenschaften [Fundamental principles of mathematical sciences]. Advanced theory and bundle methods, vol 306. Springer, Berlin · Zbl 0795.49002 [71] Hock W, Schittkowski K (1981) Test examples for nonlinear programming codes. Lecture notes in economics and mathematical systems. Springer, Berlin · Zbl 0452.90038 [72] Hock, W; Schittkowski, K, A comparative performance evaluation of 27 nonlinear programming codes, Computing, 30, 335-358, (1983) · Zbl 0497.90057 [73] Hoffman, KL; Jackson, RHF; Mulvey, JM (ed.), In pursuit of a methodology for testing mathematical programming software, No. 199, 177-199, (1982), Berlin [74] Hoffman, A; Mannos, M; Sokolowsky, D; Wiegmann, N, Computational experience in solving linear programs, J Soc Ind Appl Math, 1, 17-33, (1953) · Zbl 0053.41805 [75] Hough, P; Kolda, T; Torczon, V, Asynchronous parallel pattern search for nonlinear optimization, SIAM J Sci Comput, 23, 134-156, (2001) · Zbl 0990.65067 [76] Houstis, EN; Rice, JR; Christara, CC; Vavalis, EA; Rice, JR (ed.), Performance of scientific software, No. 14, 123-155, (1988), New York [77] Huang, HY; Levy, AV, Numerical experiments on quadratically convergent algorithms for function minimization, J Optim Theory Appl, 6, 269-282, (1970) · Zbl 0187.40401 [78] Hutter, F; Hoos, HH; Leyton-Brown, K; Stützle, T, Paramils: an automatic algorithm configuration framework, J Artif Intell Res, 36, 267-306, (2009) · Zbl 1192.68831 [79] Hutter, F; Hoos, HH; Leyton-Brown, K; Lodi, A (ed.); Milano, M (ed.); Toth, P (ed.), Automated configuration of mixed integer programming solvers, No. 6140, 186-202, (2010), Berlin [80] Jackson, RHF; Boggs, PT; Nash, SG; Powell, S, Guidelines for reporting results of computational experiments. report of the ad hoc committee, Math Program, 49, 413-425, (1990) [81] Jamil, M; Yang, XS, A literature survey of benchmark functions for global optimisation problems, Int J Math Model Numer Optim, 4, 150-194, (2013) · Zbl 1280.65053 [82] Johnson DS, McGeoch LA, Rothberg EE (1996) Asymptotic experimental analysis for the Held-Karp traveling salesman bound. In: Proceedings of the seventh annual ACM-SIAM symposium on discrete algorithms, SODA’96, Philadelphia, PA, USA. Society for Industrial and Applied Mathematics, pp 341-350 · Zbl 0845.90123 [83] Khompatraporn, C; Pinter, JD; Zabinsky, ZB, Comparative assessment of algorithms and software for global optimization, J Glob Optim, 31, 613-633, (2005) · Zbl 1093.90043 [84] Knuth DE (1994) The Stanford GraphBase: a platform for combinatorial computing, vol 37. Addison-Wesley Publishing Company, Boston · Zbl 0824.68040 [85] Koch, T; Achterberg, T; Andersen, E; Bastert, O; Berthold, T; Bixby, RE; Danna, E; Gamrath, G; Gleixner, AM; Heinz, S; Lodi, A; Mittelmann, H; Ralphs, T; Salvagnin, D; Steffy, DE; Wolter, K, Miplib 2010, Math Program Comput, 3, 103-163, (2011) [86] Kolda, TG; Lewis, RM; Torczon, V, Optimization by direct search: new perspectives on some classical and modern methods, SIAM Rev, 45, 385-482, (2003) · Zbl 1059.90146 [87] Kortelainen, M; Lesinski, T; Moré, J; Nazarewicz, W; Sarich, J; Schunck, N; Stoitsov, MV; Wild, S, Nuclear energy density optimization, Phys Rev C, 82, 024313, (2010) [88] Kvasov, DE; Mukhametzhanov, MS, One-dimensional global search: nature-inspired vs. Lipschitz methods, AIP Conf Proc, 1738, 400012, (2016) [89] Kvasov DE, Mukhametzhanov MS (2017) Metaheuristic vs. deterministic global optimization algorithms: the univariate case. Appl Math Comput · Zbl 1325.90004 [90] Kvasov, DE; Sergeyev, YD, Deterministic approaches for solving practical black-box global optimization problems, Adv Eng Softw, 80, 58-66, (2015) [91] LaMarca, A; Ladner, R, The influence of caches on the performance of heaps, J Exp Algorithmics, 1, 4, (1996) · Zbl 1073.68892 [92] Lenard, ML; Minkoff, M, Randomly generated test problems for positive definite quadratic programming, ACM Trans Math Softw, 10, 86-96, (1984) · Zbl 0545.90081 [93] Liu, D; Zhang, XS, Test problem generator by neural network for algorithms that try solving nonlinear programming problems globally, J Glob Optim, 16, 229-243, (2000) · Zbl 0962.90037 [94] McGeoch, CC, Toward an experimental method for algorithm simulation, INFORMS J Comput, 8, 1-15, (1996) · Zbl 0854.68038 [95] McGeoch, CC; Pardalos, PM (ed.); Romeijn, HE (ed.), Experimental analysis of algorithms, No. 62, 489-513, (2002), New York [96] Miele A, Tietze JL, Levy AV (1972) Comparison of several gradient algorithms for mathematical programming problems. Aero-astronautics report no. 94, Rice University, Houston · Zbl 0276.65034 [97] Mittelmann, HD, An independent benchmarking of SDP and SOCP solvers, Math Program, 95, 407-430, (2003) · Zbl 1030.90080 [98] Mittelmann, HD; Pruessner, A, A server for automated performance analysis of benchmarking data, Optim Methods Softw, 21, 105-120, (2006) · Zbl 1181.90309 [99] Moré, JJ; Wild, S, Benchmarking derivative-free optimization algorithms, SIAM J Optim, 20, 172-191, (2009) · Zbl 1187.90319 [100] Moré, JJ; Garbow, BS; Hillstrom, KE, Testing unconstrained optimization software, ACM Trans Math Softw, 7, 17-41, (1981) · Zbl 0454.65049 [101] Mulvey JM (ed) (1982) Evaluating mathematical programming techniques, vol 199. Springer, Berlin · Zbl 0481.90053 [102] Nannen V, Eiben AE (2006) A method for parameter calibration and relevance estimation in evolutionary algorithms. In: Proceedings of the 8th annual conference on genetic and evolutionary computation, GECCO’06, New York, NY, USA. ACM, pp 183-190 [103] Nash, S; Nocedal, J, A numerical study of the limited memory BFGS method and the truncated-Newton method for large scale optimization, SIAM J Optim, 1, 358-372, (1991) · Zbl 0756.65091 [104] Nell, C; Fawcett, C; Hoos, HH; Leyton-Brown, K; Coello, CAC (ed.), HAL: a framework for the automated analysis and design of high-performance algorithms, No. 6683, 600-615, (2011), Berlin [105] Netlib: Netlib linear programming library. http://netlib.org/ · Zbl 1099.65051 [106] Neumaier, A; Shcherbina, O; Huyer, W; Vinkó, T, A comparison of complete global optimization solvers, Math Program, 103, 335-356, (2005) · Zbl 1099.90001 [107] Ng, C-K; Li, D, Test problem generator for unconstrained global optimization, Comput Oper Res, 51, 338-349, (2014) · Zbl 1348.90010 [108] Nocedal J, Wright S (2006) Numerical optimization. Springer series in operations research and financial engineering. Springer, New York · Zbl 0369.90121 [109] Opara, K; Arabas, J, Benchmarking procedures for continuous optimization algorithms, J Telecommun Inf Technol, 4, 73-80, (2011) [110] Parejo, JA; Ruiz-Cortés, A; Lozano, S; Fernandez, P, Metaheuristic optimization frameworks: a survey and benchmarking, Soft Comput, 16, 527-561, (2012) [111] Paulavičius, R; Sergeyev, YD; Kvasov, DE; Žilinskas, J, Globally-biased disimpl algorithm for expensive global optimization, J Glob Optim, 59, 545-567, (2014) · Zbl 1297.90130 [112] Pintér, JD; Pardalos, PM (ed.); Romeijn, HE (ed.), Global optimization: software, test problems, and applications, No. 62, 515-569, (2002), New York [113] Pintér, JD, Nonlinear optimization with GAMS /LGO, J Glob Optim, 38, 79-101, (2007) · Zbl 1179.90311 [114] Pintér, JD; Kampas, FJ, Benchmarking nonlinear optimization software in technical computing environments, Top, 21, 133-162, (2013) · Zbl 1263.65056 [115] Ramsin, H; Wedin, P, A comparison of some algorithms for the nonlinear least squares problem, BIT Numer Math, 17, 72-90, (1977) · Zbl 0356.65054 [116] Rardin, RL; Uzsoy, R, Experimental evaluation of heuristic optimization algorithms: a tutorial, J Heuristics, 7, 261-304, (2001) · Zbl 0972.68634 [117] Regis, RG; Shoemaker, CA, A stochastic radial basis function method for the global optimization of expensive functions, INFORMS J Comput, 19, 497-509, (2007) · Zbl 1241.90192 [118] Regis, RG; Wild, SM, Onorbit: constrained optimization by radial basis function interpolation in trust regions, Optim Methods Softw, 32, 1-29, (2017) [119] Ridge E (2007) Design of experiments for the tuning of optimisation algorithms. Department of Computer Science, University of York, Heslington · Zbl 1144.68305 [120] Ridge, E; Kudenko, D; Bartz-Beielstein, T (ed.); Chiarandini, M (ed.); Paquete, L (ed.); Preuss, M (ed.), Tuning an algorithm using design of experiments, 265-286, (2010), Berlin [121] Rijckaert, MJ; Martens, XM, Comparison of generalized geometric programming algorithms, J Optim Theory Appl, 26, 205-242, (1978) · Zbl 0369.90112 [122] Rios, L; Sahinidis, NV, Derivative-free optimization: a review of algorithms and comparison of software implementations, J Glob Optim, 56, 1247-1293, (2013) · Zbl 1272.90116 [123] Romesis M, Xie M, Minkovich K, Cong J (2003) Optimality study project. Technical report, UCLA Computer Science Department. http://cadlab.cs.ucla.edu/ pubbench/ [124] Rosenthal RE (2014) GAMS—a user’s guide. Technical report, GAMS Development Corporation [125] Sandgren, E; Ragsdell, KM, The utility of nonlinear programming algorithms: a comparative study, part 1, J Mech Des, 102, 540-546, (1980) [126] Sandgren, E; Ragsdell, KM, The utility of nonlinear programming algorithms: a comparative study, part 2, J Mech Des, 102, 547-551, (1980) [127] Schichl, H; Markót, MC, Algorithmic differentiation techniques for global optimization in the COCONUT environment, Optim Methods Softw, 27, 359-372, (2012) · Zbl 1242.65047 [128] Schittkowski K (1980) Nonlinear programming codes: information, tests, performance. Lecture notes in economics and mathematical systems. Springer, Berlin · Zbl 0435.90063 [129] Schittkowski K (2008) An updated set of 306 test problems for nonlinear programming with validated optimal solutions—user’s guide. Technical report, Department of Computer Science, University of Bayreuth [130] Schittkowski, K; Stoer, J, A factorization method for the solution of constrained linear least squares problems allowing subsequent data changes, Numer Math, 31, 431-463, (1978) · Zbl 0378.65026 [131] Schoen, F, A wide class of test functions for global optimization, J Glob Optim, 3, 133-137, (1993) · Zbl 0772.90072 [132] Sergeyev, YD; Kvasov, DE, Global search based on efficient diagonal partitions and a set of Lipschitz constants, SIAM J Optim, 16, 910-937, (2006) · Zbl 1097.65068 [133] Sergeyev, YD; Kvasov, DE, A deterministic global optimization using smooth diagonal auxiliary functions, Commun Nonlinear Sci Numer Simul, 21, 99-111, (2015) · Zbl 1329.90112 [134] Sergeyev YD, Strongin RG, Lera D (2013) Introduction to global optimization exploiting space-filling curves. Springer briefs in optimization. Springer, New York · Zbl 1278.90005 [135] Sergeyev, YD; Kvasov, DE; Mukhametzhanov, MS, Operational zones for comparing metaheuristic and deterministic one-dimensional global optimization algorithms, Math Comput Simul, 141, 96-109, (2016) [136] Shcherbina, O; Neumaier, A; Sam-Haroud, D; Vu, XH; Nguyen, TV; Bliek, C (ed.); Jermann, C (ed.); Neumaier, A (ed.), Benchmarking global optimization and constraint satisfaction codes, No. 6683, 211-222, (2003), Berlin · Zbl 1296.90004 [137] Strongin RG, Sergeyev YD (2000) Global optimization with non-convex constraints: sequential and parallel algorithms. Springer, New York · Zbl 0987.90068 [138] Tabak, D, Comparative study of various minimization techniques used in mathematical programming, IEEE Trans Autom Control, 14, 572-572, (1969) [139] Tedford, NP; Martins, JRRA, Benchmarking multidisciplinary design optimization algorithms, Optim Eng, 11, 159-183, (2010) · Zbl 1273.65090 [140] Törn A, Žilinskas A (1989) Global optimization. Lecture notes in computer science, vol 350. Springer, Berlin · Zbl 0752.90075 [141] Törn, A; Ali, MM; Viitanen, S, Stochastic global optimization: problem classes and solution techniques, J Glob Optim, 14, 437-447, (1999) · Zbl 0952.90030 [142] Tufte ER, Graves-Morris PR (1983) The visual display of quantitative information, vol 2. Graphics Press, Cheshire [143] Tukey JW (1977) Exploratory data analysis. Pearson, Reading · Zbl 0409.62003 [144] Vanden Berghen, F; Bersini, H, CONDOR, a new parallel, constrained extension of powell’s UOBYQA algorithm: experimental results and comparison with the DFO algorithm, J Comput Appl Math, 181, 157-175, (2005) · Zbl 1072.65088 [145] Vanderbei, RJ; Shanno, DF, An interior-point algorithm for nonconvex nonlinear programming, Comput Optim Appl, 13, 231-252, (1999) · Zbl 1040.90564 [146] Vaz, AIF; Vicente, LN, A particle swarm pattern search method for bound constrained global optimization, J Glob Optim, 39, 197-219, (2007) · Zbl 1180.90252 [147] Yeniay, O, A comparative study on optimization methods for the constrained nonlinear programming problems, Math Probl Eng, 2005, 165-173, (2005) · Zbl 1116.90098 [148] Zhang, Z, Sobolev seminorm of quadratic functions with applications to derivative-free optimization, Math Program, 146, 77-96, (2014) · Zbl 1315.90063 [149] Zhigljavsky A, Žilinskas A (2008) Stochastic global optimization. Springer optimization and its applications, vol 9. Springer, New York · Zbl 1136.90003 This reference list is based on information provided by the publisher or from digital mathematics libraries. Its items are heuristically matched to zbMATH identifiers and may contain data conversion errors. In some cases that data have been complemented/enhanced by data from zbMATH Open. This attempts to reflect the references listed in the original paper as accurately as possible without claiming completeness or a perfect matching.