Derivative-free optimization: a review of algorithms and comparison of software implementations. (English) Zbl 1272.90116

Rios, Luis Miguel; Sahinidis, Nikolaos V.

J. Glob. Optim. 56, No. 3, 1247-1293 (2013).

Summary: This paper addresses the solution of bound-constrained optimization problems using algorithms that require only the availability of objective function values but no derivative information. We refer to these algorithms as derivative-free algorithms. Fueled by a growing number of applications in science and engineering, the development of derivative-free optimization algorithms has long been studied, and it has found renewed interest in recent time. Along with many derivative-free algorithms, many software implementations have also appeared. The paper presents a review of derivative-free algorithms, followed by a systematic comparison of 22 related implementations using a test set of 502 problems. The test bed includes convex and nonconvex problems, smooth as well as nonsmooth problems. The algorithms are tested under the same conditions and ranked under several criteria, including their ability to find near-global solutions for nonconvex problems, improve a given starting point, and refine a near-optimal solution. A total of 112,448 problem instances are solved. We find that the ability of all these solvers to obtain good solutions diminishes with increasing problem size. For the problems used in this study, TOMLAB/MULTIMIN, TOMLAB/GLCCLUSTER, MCS and TOMLAB/LGO are better, on average, than other derivative-free solvers in terms of solution quality within 2,500 function evaluations. These global solvers outperform local solvers even for convex problems. Finally, TOMLAB/OQNLP, NEWUOA, and TOMLAB/MULTIMIN show superior performance in terms of refining a near-optimal solution.

MSC:

90C56	Derivative-free methods and methods using generalized derivatives
90C30	Nonlinear programming

Keywords:

derivative-free algorithms; direct search methods; surrogate models

Software:

BOBYQA; MultiMin; IMFIL; glcCluster; Ipopt; LINDOGlobal; NEWUOA; TOMLAB; MCS; HOPSPACK; UOBYQA; Spacemap; CMA-ES; fminsearch; SNOBFIT; LINDO; DFO; SPACE; EGO; NOMADm; MCS; COIN-OR; BARON; DAKOTA; OrthoMADS; PSwarm; Global Optimization Toolbox For Maple; LGO; NOMAD; KELLEY; ORBIT; BRENT; OQNLP

References:

[1]	Aarts, E.H.L.; Laarhoven, P.J.M., Statistical cooling: a general approach to combinatorial optimization problems, Phillips J. Res., 40, 193-226, (1985)
[2]	Abramson, M.A.: Pattern Search Algorithms for Mixed Variable General Constrained Optimization Problems. PhD thesis, Department of Computational and Applied Mathematics, Rice University, Houston (2002, Aug) · Zbl 0752.90076
[3]	Abramson, M.A.: NOMADm Version 4.5 User’s Guide. Air Force Institute of Technology, Wright-Patterson AFB, OH (2007) · Zbl 0121.35603
[4]	Abramson, M.A.; Asaki, T.J.; Dennis, J.E.; O’Reilly, K.R.; Pingel, R.L., Quantitative object reconstruction via Abel-based X-ray tomography and mixed variable optimization, SIAM J. Imaging Sci., 1, 322-342, (2008) · Zbl 1177.65200
[5]	Abramson, M.A.; Audet, C., Convergence of mesh adaptive direct search to second-order stationary points, SIAM J. Optim., 17, 606-609, (2006) · Zbl 1174.90877
[6]	Abramson, M.A., Audet, C., Couture, G., Dennis, J.E. Jr., LeDigabel, S.: The Nomad project. http://www.gerad.ca/nomad/ · Zbl 0251.65052
[7]	Abramson, M.A.; Audet, C.; Dennis, J.E., Filter pattern search algorithms for mixed variable constrained optimization problems, Pac. J. Optim., 3, 477-500, (2007) · Zbl 1138.65043
[8]	Abramson, M.A.; Audet, C.; Dennis, J.E.; Le Digabel, S., Orthomads: a deterministic MADS instance with orthogonal directions, SIAM J. Optim., 20, 948-966, (2009) · Zbl 1189.90202
[9]	Audet, C., Convergence results for generalized pattern search algorithms are tight, Optim. Eng., 5, 101-122, (2004) · Zbl 1085.90053
[10]	Audet, C.; Béchard, V.; Chaouki, J., Spent potliner treatment process optimization using a MADS algorithm, Optim. Eng., 9, 143-160, (2008) · Zbl 1167.92039
[11]	Audet, C.; Dennis, J.E., Mesh adaptive direct search algorithms for constrained optimization, SIAM J. Optim., 17, 188-217, (2006) · Zbl 1112.90078
[12]	Audet, C.; Dennis, J.E., A progressive barrier for derivative-free nonlinear programming, SIAM J. Optim., 20, 445-472, (2009) · Zbl 1187.90266
[13]	Awasthi, S.: Molecular Docking by Derivative-Free Optimization Solver. Master’s thesis, Department of Chemical Engineering, Carnegie Mellon University, Pittsburgh (2008) · Zbl 0884.65053
[14]	Barros, P.A.; Kirby, M.R.; Mavris, D.N., Impact of sampling techniques selection on the creation of response surface models, SAE Trans. J. Aerosp., 113, 1682-1693, (2004)
[15]	Bartholomew-Biggs, M.C.; Parkhurst, S.C.; Wilson, S.P., Using DIRECT to solve an aircraft routing problem, Comput. Optim. Appl., 21, 311-323, (2002) · Zbl 1017.90133
[16]	Barton, R.R.: Metamodeling: A state of the art review. In: Proceedings of the 1994 Winter Simulation Conference, pp. 237-244 (1994) · Zbl 1099.90047
[17]	Bélisle, C.J.; Romeijn, H.E.; Smith, R.L., Hit-and-run algorithms for generating multivariate distributions, Math. Oper. Res., 18, 255-266, (1993) · Zbl 0771.60052
[18]	Bethke, J.D.: Genetic Algorithms as Function Optimizers. PhD thesis, Department of Computer and Communication Sciences, University of Michigan, Ann Arbor (1980) · Zbl 1180.90252
[19]	Björkman , M.; Holmström, K., Global optimization of costly nonconvex functions using radial basis functions, Optim. Eng., 1, 373-397, (2000) · Zbl 1035.90061
[20]	Boender, C.G.E.; Rinnooy Kan, A.H.G.; Timmer, G.T., A stochastic method for global optimization, Math. Program., 22, 125-140, (1982) · Zbl 0525.90076
[21]	Boneh, A., Golan, A.: Constraints’ redundancy and feasible region boundedness by random feasible point generator (RFPG). In: 3rd European Congress on Operations Research (EURO III), Amsterdam (1979) · Zbl 1154.65049
[22]	Booker, A.J., Dennis, J.E., Jr., Frank, P.D., Serafini, D.B., Torczon, V.J., Trosset, M.W.: A rigorous framework for optimization of expensive functions by surrogates. In: ICASE Report, pp. 1-24 (1998)
[23]	Booker, A.J.; Dennis, J.E.; Frank, P.D.; Serafini, D.B.; Torczon, V.J.; Trosset, M.W., A rigorous framework for optimization of expensive functions by surrogates, Struct. Optim., 17, 1-13, (1999)
[24]	Booker, A.J.; Meckesheimer, M.; Torng, T., Reliability based design optimization using design explorer, Optim. Eng., 5, 179-205, (2004) · Zbl 1085.90014
[25]	Brent R.P.: Algorithms for Minimization without Derivatives. Prentice-Hall, Englewood Cliffs (1973) · Zbl 0245.65032
[26]	Chang, K.-F.: Modeling and Optimization of Polymerase Chain Reaction Using Derivative-Free Optimization. Master’s thesis, Department of Chemical Engineering, Carnegie Mellon University, Pittsburgh (2011)
[27]	Chiang, T.; Chow, Y., A limit theorem for a class of inhomogeneous Markov processes, Ann. Probab., 17, 1483-1502, (1989) · Zbl 0687.60070
[28]	COIN-OR Project. Derivative Free Optimization. http://projects.coin-or.org/Dfo
[29]	COIN-OR Project. IPOPT 2.3.x A software package for large-scale nonlinear optimization. http://www.coin-or.org/Ipopt/ipopt-fortran.html · Zbl 0809.90117
[30]	Conn, A.R.; Gould, N.; Lescrenier, M.; Toint, Ph.L.; Gomez, S. (ed.); Hennart, J.-P. (ed.), Performance of a multifrontal scheme for partially separable optimization, 79-96, (1994), Dordrecht · Zbl 0809.90117
[31]	Conn, A.R.; Scheinberg, K.; Toint, P.L.; Buhmann, M.D. (ed.); Iserles, A. (ed.), On the convergence of derivative-free methods for unconstrained optimization, 83-108, (1996), Cambridge
[32]	Conn, A.R.; Scheinberg, K.; Toint, P.L., Recent progress in unconstrained nonlinear optimization without derivatives, Math. Program., 79, 397-414, (1997) · Zbl 0887.90154
[33]	Conn, A.R., Scheinberg, K., Toint, P.L.: A derivative free optimization algorithm in practice. In: Proceedings of AIAA St Louis Conference, pp. 1-11 (1998) · Zbl 1099.90001
[34]	Conn, A.R.; Scheinberg, K.; Vicente, L.N., Global convergence of general derivative-free trust-region algorithms to first and second order critical points, SIAM J. Optim., 20, 387-415, (2009) · Zbl 1187.65062
[35]	Conn A.R., Scheinberg K., Vicente L.N.: Introduction to derivative-free optimization. SIAM, Philadelphia (2009) · Zbl 1163.49001
[36]	Cox, D.D., John, S.: SDO: A statistical method for global optimization. In: Multidisciplinary Design Optimization (Hampton, VA, 1995), pp. 315-329. SIAM, Philadelphia (1997)
[37]	Csendes, T.; Pál, L.; Sendín, J.O.H.; Banga, J.R., The GLOBAL optimization method revisited, Optim. Lett., 2, 445-454, (2008) · Zbl 1160.90660
[38]	Custódio, A.L.; Dennis, J.E.; Vicente, L.N., Using simplex gradients of nonsmooth functions in direct search methods, IMA J. Numer. Anal., 28, 770-784, (2008) · Zbl 1156.65059
[39]	Custódio, A.L., Rocha, H., Vicente, L.N.: Incorporating minimum Frobenius norm models in direct search. Comput. Optim. Appl. (to appear) · Zbl 0962.65049
[40]	Custódio, A.L., Vicente, L.N.: Using sampling and simplex derivatives in pattern search methods. SIAM J. Optim. 18, 537-555 (2007) · Zbl 1144.65039
[41]	Custódio, A.L., Vicente, L.N.: SID-PSM: A Pattern Search Method Guided by Simplex Derivatives for Use in Derivative-Free Optimization. Departamento de Matemática, Universidade de Coimbra, Coimbra (2008) · Zbl 1400.90226
[42]	Deming, S.N.; Parker, L.R.; Denton, M.B., A review of simplex optimization in analytical chemistry, Crit. Rev. Anal. Chem., 7, 187-202, (1974)
[43]	Desai, R.: A Comparison of Algorithms for Optimizing the Omega Function. Master’s thesis, Department of Chemical Engineering, Carnegie Mellon University, Pittsburgh (2010) · Zbl 1116.90417
[44]	Eberhart, R.,Kennedy, J.: A new optimizer using particle swarm theory. In: Proceedings of the 6th International Symposium on Micro Machine and Human Science, Nagoya, pp. 39-43 (1995) · Zbl 1194.76296
[45]	Eldred, M.S., Adams, B.M., Gay, D.M., Swiler, L.P., Haskell, K., Bohnhoff, W.J., Eddy, J.P., Hart, W.E., Watson, J-P, Hough, P.D., Kolda, T.G., Williams, P.J., Martinez-Canales, M.L., DAKOTA, A.: Multilevel Parallel Object-Oriented Framework for Design Optimization, Parameter Estimation, Uncertainty Quantification, and Sensitivity Analysis: Version 4.2 User’s Manual. Sandia National Laboratories, Albuquerque (2008) · Zbl 1085.90053
[46]	Fan, S.S.; Zahara, E., A hybrid simplex search and particle swarm optimization for unconstrained optimization, Eur. J. Oper. Res., 181, 527-548, (2007) · Zbl 1121.90116
[47]	Finkel, D.E.; Kelley, C.T., Additive scaling and the DIRECT algorithm, J. Glob. Optim., 36, 597-608, (2006) · Zbl 1142.90488
[48]	Fowler, K.R.; Reese, J.P.; Kees, C.E.; Dennis, J.E.; Kelley, C.T.; Miller, C.T.; Audet, C.; Booker, A.J.; Couture, G.; Darwin, R.W.; Farthing, M.W.; Finkel, D.E.; Gablonsky, J.M.; Gray, G.; Kolda, T.G., A comparison of derivative-free optimization methods for groundwater supply and hydraulic capture community problems, Adv. Water Resour., 31, 743-757, (2008)
[49]	Gablonsky, J.M.: Modifications of the DIRECT Algorithm. PhD thesis, Department of Mathematics, North Carolina State University, Raleigh (2001) · Zbl 1039.90049
[50]	Gilmore, P.; Kelley, C.T., An implicit filtering algorithm for optimization of functions with many local minima, SIAM J. Optim., 5, 269-285, (1995) · Zbl 0828.65064
[51]	GLOBAL Library. http://www.gamsworld.org/global/globallib.htm · Zbl 1031.90047
[52]	Gray, G.; Kolda, T.; Sale, K.; Young, M., Optimizing an empirical scoring function for transmembrane protein structure determination, INFORMS J. Comput., 16, 406-418, (2004) · Zbl 1239.90115
[53]	Gutmann, H.-M., A radial basis function method for global optimization, J. Glob. Optim., 19, 201-227, (2001) · Zbl 0972.90055
[54]	Han, J.; Kokkolaras, M.; Papalambros, P.Y., Optimal design of hybrid fuel cell vehicles, J. Fuel Cell Sci. Technol., 5, 041014, (2008)
[55]	Hansen, N.: The CMA Evolution Strategy: A tutorial. http://www.lri.fr/hansen/cmaesintro.html · Zbl 1225.90162
[56]	Hansen, N.; Lozano, J.A. (ed.); Larranaga, P. (ed.); Inza, I. (ed.); Bengoetxea, E. (ed.), The CMA evolution strategy: a comparing review, 75-102, (2006), Berlin
[57]	Hayes, R.E.; Bertrand, F.H.; Audet, C.; Kolaczkowski, S.T., Catalytic combustion kinetics: using a direct search algorithm to evaluate kinetic parameters from light-off curves, Can. J. Chem. Eng., 81, 1192-1199, (2003)
[58]	Holland J.H.: Adaptation in Natural and Artificial Systems. The University of Michigan Press, Ann Arbor (1975)
[59]	Holmström, K.: Private Communication (2009) · Zbl 1172.90492
[60]	Holmström, K., Göran, A.O., Edvall, M.M.: User’s Guide for TOMLAB 7. Tomlab Optimization. http://tomopt.com · Zbl 0796.49032
[61]	Holmström, K., Göran, A.O., Edvall, M.M.: User’s Guide for TOMLAB/CGO. Tomlab Optimization (2007). http://tomopt.com
[62]	Holmström, K., Göran, A.O., Edvall, M.M.: User’s Guide for TOMLAB/OQNLP. Tomlab Optimization (2007). http://tomopt.com
[63]	Holmström , K.; Quttineh, N.-H.; Edvall, M.M., An adaptive radial basis algorithm (ARBF) for expensive black-box mixed-integer constrained global optimization, Optim. Eng., 9, 311-339, (2008) · Zbl 1400.90226
[64]	Hooke, R.; Jeeves, T.A., Direct search solution of numerical and statistical problems, J. Assoc. Comput. Mach., 8, 212-219, (1961) · Zbl 0111.12501
[65]	Huyer, W.; Neumaier, A., Global optimization by multilevel coordinate search, J. Glob. Optim., 14, 331-355, (1999) · Zbl 0956.90045
[66]	Huyer, W.; Neumaier, A., SNOBFIT—stable noisy optimization by branch and fit, ACM Trans. Math. Softw., 35, 1-25, (2008)
[67]	Hvattum, L.M.; Glover, F., Finding local optima of high-dimensional functions using direct search methods, Eur. J. Oper. Res., 195, 31-45, (2009) · Zbl 1156.90440
[68]	Ingber, L.: Adaptive Simulated Annealing (ASA). http://www.ingber.com/#ASA · Zbl 0860.93035
[69]	Järvi, T.: A Random Search Optimizer with an Application to a Max-Min Problem. Technical report, Pulications of the Institute for Applied Mathematics, University of Turku (1973)
[70]	Jones, D.R., A taxonomy of global optimization methods based on response surfaces, J. Glob. Optim., 21, 345-383, (2001) · Zbl 1172.90492
[71]	Jones, D.R.: The DIRECT global optimization algorithm. In: Floudas, C.A., Pardalos, P.M. (eds.) Encyclopedia of Optimization, vol. 1, pp. 431-440. Kluwer, Boston (2001) · Zbl 1225.90162
[72]	Jones, D.R.; Perttunen, C.D.; Stuckman, B.E., Lipschitzian optimization without the Lipschitz constant, J. Optim. Theory Appl., 79, 157-181, (1993) · Zbl 0796.49032
[73]	Jones, D.R.; Schonlau, M.; Welch, W.J., Efficient global optimization of expensive black-box functions, J. Glob. Optim., 13, 455-492, (1998) · Zbl 0917.90270
[74]	Kelley, C.T.: Users Guide for IMFIL version 1.0. http://www4.ncsu.edu/ctk/imfil.html
[75]	Kelley, C.T., Detection and remediation of stagnation in the Nelder-Mead algorithm using a sufficient decrease condition, SIAM J. Optim., 10, 43-55, (1999) · Zbl 0962.65048
[76]	Kelley C.T.: Iterative Methods for Optimization. SIAM, Philadelphia (1999) · Zbl 0934.90082
[77]	Kennedy, J., Eberhart, R.: Particle swarm optimization. In: Proceedings of the IEEE International Conference on Neural Networks, Piscataway, pp. 1942-1948 (1995) · Zbl 0972.90055
[78]	Kirkpatrick, S.; Gelatt, C.D.; Vecchi, M.P., Optimization by simulated annealing, Science, 220, 671-680, (1983) · Zbl 1225.90162
[79]	Kolda, T.G.; Lewis, R.M.; Torczon, V.J., Optimization by direct search: new perspectives on some classical and modern methods, SIAM Rev., 45, 385-482, (2003) · Zbl 1059.90146
[80]	Kolda, T.G.; Torczon, V.J., On the convergence of asynchronous parallel pattern search, SIAM J. Optim., 14, 939-964, (2004) · Zbl 1073.90046
[81]	Lagarias, J.C.; Reeds, J.A.; Wright, M.H.; Wright, P.E., Convergence properties of the Nelder-Mead simplex method in low dimensions, SIAM J. Optim., 9, 112-147, (1998) · Zbl 1005.90056
[82]	LeDigabel, S.: NOMAD User Guide Version 3.3. Technical report, Les Cahiers du GERAD (2009) · Zbl 1031.90048
[83]	Lewis, R.M.; Torczon, V.J., Pattern search algorithms for bound constrained minimization, SIAM J. Optim., 9, 1082-1099, (1999) · Zbl 1031.90047
[84]	Lewis, R.M.; Torczon, V.J., Pattern search algorithms for linearly constrained minimization, SIAM J. Optim., 10, 917-941, (2000) · Zbl 1031.90048
[85]	Lewis, R.M.; Torczon, V.J., A globally convergent augmented Lagrangian pattern search algorithm for optimization with general constraints and simple bounds, SIAM J. Optim., 12, 1075-1089, (2002) · Zbl 1011.65030
[86]	Liepins, G.E.; Hilliard, M.R., Genetic algorithms: foundations and applications, Ann. Oper. Res., 21, 31-58, (1989) · Zbl 0796.68167
[87]	Lin, Y.; Schrage, L., The global solver in the LINDO API, Optim. Methods Softw., 24, 657-668, (2009) · Zbl 1177.90325
[88]	Lucidi, S.; Sciandrone, M., On the global convergence of derivative-free methods for unconstrained minimization, SIAM J. Optim., 13, 97-116, (2002) · Zbl 1027.90112
[89]	Lukšan, L., Vlček, J.: Test Problems for Nonsmooth Unconstrained and Linearly Constrained Optimization. Technical report, Institute of Computer Science, Academy of Sciences of the Czech Republic (2000). http://www3.cs.cas.cz/ics/reports/v798-00.ps
[90]	Marsden, A.L.; Feinstein, J.A.; Taylor, C.A., A computational framework for derivative-free optimization of cardiovascular geometries, Comput. Methods Appl. Mech. Eng., 197, 1890-1905, (2008) · Zbl 1194.76296
[91]	Marsden, A.L.; Wang, M.; Dennis, J.E.; Moin, P., Optimal aeroacustic shape design using the surrogate management framework, Optim. Eng., 5, 235-262, (2004) · Zbl 1116.90417
[92]	Marsden, A.L.; Wang, M.; Dennis, J.E.; Moin, P., Trailing-edge noise reduction using derivative-free optimization and large-eddy simulation, J. Fluid Mech., 5, 235-262, (2007) · Zbl 1145.76044
[93]	Matheron, G., Principles of geostatistics, Econ. Geol., 58, 1246-1266, (1967)
[94]	McKinnon, K.I.M., Convergence of the Nelder-Mead simplex method to a nonstationary point, SIAM J. Optim., 9, 148-158, (1998) · Zbl 0962.65049
[95]	Metropolis, N.; Rosenbluth, A.W.; Rosenbluth, M.N.; Teller, A.H.; Teller, E., Equation of state calculations by fast computing machines, J. Chem. Phys., 21, 1087-1092, (1953)
[96]	Mongeau, M.; Karsenty, H.; Rouzé, V.; Hiriart-Urruty, J.B., Comparison of public-domain software for black box global optimization, Optim. Methods Softw., 13, 203-226, (2000) · Zbl 0963.90062
[97]	Moré, J., Wild, S.: Benchmarking derivative-free optimization algorithms. SIAM J. Optim. 20, 172-191 (2009) · Zbl 1187.90319
[98]	Mugunthan, P.; Shoemaker, C.A.; Regis, R.G., Comparison of function approximation, heuristic, and derivative-based methods for automatic calibration of computationally expensive groundwater bioremediation models, Water Resour. Res., 41, w11427, (2005)
[99]	Nelder, J.A.; Mead, R., A simplex method for function minimization, Comput. J., 7, 308-313, (1965) · Zbl 0229.65053
[100]	Nesterov, Y.: Gradient methods for minimizing composite objective function. CORE Discussion Paper 2007/76 (2007) · Zbl 0687.60070
[101]	Neumaier, A.: MCS: Global Optimization by Multilevel Coordinate Search. http://www.mat.univie.ac.at/neum/software/mcs/ · Zbl 0956.90045
[102]	Neumaier, A.; Shcherbina, O.; Huyer, W.; Vinkó, T., A comparison of complete global optimization solvers, Math. Program., 103, 335-356, (2005) · Zbl 1099.90001
[103]	Oeuvray, R.: Trust-Region Methods Based on Radial Basis Functions with Application to Biomedical Imaging. PhD thesis, Institute of Mathematics, Swiss Federal Institute of Technology, Lausanne (2005, March) · Zbl 1035.90061
[104]	Orosz, J.E.; Jacobson, S.H., Finite-time performance analysis of static simulated annealing algorithms, Comput. Optim. Appl., 21, 21-53, (2002) · Zbl 0988.90053
[105]	Pintér, J.: Homepage of Pintér Consulting Services. http://www.pinterconsulting.com/ · Zbl 1187.90266
[106]	Pintér J.D.: Global Optimization in Action: Continuous and Lipschitz Optimization. Algorithms, Implementations and Applications. Nonconvex Optimization and its Applications. Kluwer, Dordrecht (1995)
[107]	Pintér, J.D., Holmström, K., Göran, A.O., Edvall, M.M.: User’s Guide for TOMLAB/LGO. Tomlab Optimization (2006). http://tomopt.com · Zbl 1138.65043
[108]	Plantenga, T.D.: HOPSPACK 2.0 User Manual. Technical Report SAND2009-6265, Sandia National Laboratories, Albuquerque (2009)
[109]	Powell, M.J.D.; Gomez, S. (ed.); Hennart, J-P. (ed.), A direct search optimization method that models the objective and constraint functions by linear interpolation, 51-67, (1994), Dordrecht · Zbl 0826.90108
[110]	Powell, M.J.D.: A direct search optimization method that models the objective by quadratic interpolation. In: Presentation at the 5th Stockholm Optimization Days (1994) · Zbl 0826.90108
[111]	Powell, M.J.D.: Recent Research at Cambridge on Radial Basis Functions. Technical report, Department of Applied Mathematics and Theoretical Physics, University of Cambridge (1998) · Zbl 0958.41501
[112]	Powell, M.J.D., UOBYQA: unconstrained optimization by quadratic approximation, Math. Program., 92, 555-582, (2002) · Zbl 1014.65050
[113]	Powell, M.J.D.; Di Pillo, G. (ed.); Roma, M. (ed.), The NEWUOA software for unconstrained optimization without derivatives, 255-297, (2006), New York · Zbl 1108.90005
[114]	Powell, M.J.D., Developments of NEWUOA for minimization without derivatives, IMA J. Numer. Anal., 28, 649-664, (2008) · Zbl 1154.65049
[115]	Powell, M.J.D.: The BOBYQA Algorithm for Bound Constrained Optimization Without Derivatives. Technical report, Department of Applied Mathematics and Theoretical Physics, University of Cambridge (2009)
[116]	Princeton Library. http://www.gamsworld.org/performance/princetonlib/princetonlib.htm · Zbl 1180.90252
[117]	Regis, R.G.; Shoemaker, C.A., Constrained global optimization of expensive black box functions using radial basis functions, J. Glob. Optim., 31, 153-171, (2005) · Zbl 1274.90511
[118]	Regis, R.G.; Shoemaker, C.A., Improved strategies for radial basis function methods for global optimization, J. Glob. Optim., 37, 113-135, (2007) · Zbl 1149.90120
[119]	Richtarik, P.: Improved algorithms for convex minimization in relative scale. SIAM J. Optim. (2010, to appear). http://www.optimization-online.org/DB_FILE/2009/02/2226.pdf · Zbl 1231.90313
[120]	Rios, L.M.: Algorithms for Derivative-Free Optimization. PhD thesis, Department of Industrial and Enterprise Systems Engineering, University of Illinois, Urbana (2009, May)
[121]	Romeo, F.; Sangiovanni-Vincentelli, A., A theoretical framework for simulated annealing, Algorithmica, 6, 302-345, (1991) · Zbl 0717.90061
[122]	Sacks, J.; Welch, W.J.; Mitchell, T.J.; Wynn, H.P., Design and analysis of computer experiments, Stat. Sci., 4, 409-423, (1989) · Zbl 0955.62619
[123]	Sahinidis, N.V., Tawarmalani, M.: BARON 7.5: Global Optimization of Mixed-Integer Nonlinear Programs, User’s Manual (2005)
[124]	Sandia National Laboratories: The Coliny Project. https://software.sandia.gov/trac/acro/wiki/Overview/Projects
[125]	Scheinberg, K.: Manual for Fortran Software Package DFO v2.0 (2003)
[126]	Schonlau, M.: Computer Experiments and Global Optimization. PhD thesis, Department of Statistics, University of Waterloo, Waterloo (1997)
[127]	Serafini, D.B.: A Framework for Managing Models in Nonlinear Optimization of Computationally Expensive Functions. PhD thesis, Department of Computational and Applied Mathematics, Rice University, Houston (1998, Nov) · Zbl 0717.90061
[128]	Shah, S.B., Sahinidis, N.V.: SAS-Pro: Simultaneous residue assignment and structure superposition for protein structure alignment. PLoS ONE 7(5), e37493 (2012)
[129]	Shubert, B.O., A sequential method seeking the global maximum of a function, SIAM J. Numer. Anal., 9, 379-388, (1972) · Zbl 0251.65052
[130]	Smith, R.L., Efficient Monte Carlo procedures for generating points uniformly distributed over bounded regions, Oper. Res., 32, 1296-1308, (1984) · Zbl 0552.65004
[131]	Søndergaard, J.: Optimization Using Surrogate Models—by the Space Mapping Technique. PhD thesis, Technical University of Denmark, Department of Mathematical Modelling, Lingby (2003)
[132]	Spendley, W.; Hext, G.R.; Himsworth, F.R., Sequential application for simplex designs in optimisation and evolutionary operation, Technometrics, 4, 441-461, (1962) · Zbl 0121.35603
[133]	Tawarmalani, M.; Sahinidis, N.V., A polyhedral branch-and-cut approach to global optimization, Math. Program., 103, 225-249, (2005) · Zbl 1099.90047
[134]	Torczon, V.J., On the convergence of multidirectional search algorithms, SIAM J. Optim., 1, 123-145, (1991) · Zbl 0752.90076
[135]	Torczon, V.J., On the convergence of pattern search algorithms, SIAM J. Optim., 7, 1-25, (1997) · Zbl 0884.65053
[136]	Tseng, P., Fortified-descent simplicial search method: a general approach, SIAM J. Optim., 10, 269-288, (1999) · Zbl 1030.90122
[137]	Vaz, A.I.F.: PSwarm Home Page. http://www.norg.uminho.pt/aivaz/pswarm/
[138]	Vaz, A.I.F.; Vicente, L.N., A particle swarm pattern search method for bound constrained global optimization, J. Glob. Optim., 39, 197-219, (2007) · Zbl 1180.90252
[139]	Wang, H.: Application of Derivative-Free Algorithms in Powder Diffraction. Master’s thesis, Department of Chemical Engineering, Carnegie Mellon University, Pittsburgh (2011)
[140]	Wild, S.M.; Regis, R.G.; Shoemaker, C.A., ORBIT: optimization by radial basis function interpolation in trust-regions, SIAM J. Sci. Comput., 30, 3197-3219, (2008) · Zbl 1178.65065
[141]	Winfield, D.: Function and Functional Optimization by Interpolation in Data Tables. PhD thesis, Harvard University, Cambridge (1969)
[142]	Winslow, T.A., Trew, R.J., Gilmore, P., Kelley, C.T.: Simulated performance optimization of gaas mesfet amplifiers. In: IEEE/Cornell Conference on Advanced Concepts in High Speed Semiconductor Devices and Circuits, Piscataway, pp. 393-402 (1991) · Zbl 1017.90133
[143]	Zhao, Z.; Meza, J.C.; Van Hove, M., Using pattern search methods for surface structure determination of nanomaterials, J. Phys. Condens. Matter, 18, 8693-8706, (2006)
[144]	Zheng, Y.: Pairs Trading and Portfolio Optimization. Master’s thesis, Department of Chemical Engineering, Carnegie Mellon University, Pittsburgh (2011)