A comparison of general-purpose optimization algorithms for finding optimal approximate experimental designs. (English) Zbl 07160675

Summary: Several common general purpose optimization algorithms are compared for finding \(A\)- and \(D\)-optimal designs for different types of statistical models of varying complexity, including high dimensional models with five and more factors. The algorithms of interest include exact methods, such as the interior point method, the Nelder-Mead method, the active set method, the sequential quadratic programming, and metaheuristic algorithms, such as particle swarm optimization, simulated annealing and genetic algorithms. Several simulations are performed, which provide general recommendations on the utility and performance of each method, including hybridized versions of metaheuristic algorithms for finding optimal experimental designs. A key result is that general-purpose optimization algorithms, both exact methods and metaheuristic algorithms, perform well for finding optimal approximate experimental designs.


62-XX Statistics


Full Text: DOI


[1] Amo-Salas, M.; Delgado-Marquez, E.; Lopez-Fidalgo, J., Optimal experimental designs in the flow rate of particles, Technometrics, 58, 2, 269-276 (2016)
[2] Atwood, C. L., Convergent design sequences, for sufficiently regular optimality criteria, Ann. Statist., 4, 6, 1124-1138 (1976) · Zbl 0344.62064
[3] Berger, M.; Wong, W., An Introduction to Optimal Designs for Social and Biomedical Research (2009), John Wiley and Sons · Zbl 1182.62154
[4] Bogacka, B.; Patan, M.; Johnson, P. J.; Youdim, K.; Atkinson, A. C., Optimum design of experiments for enzyme inhibition kinetic models, J. Biopharm. Statist., 21, 3, 555-572 (2011)
[5] Box, G. E.P.; Hunter, W. G., The experimental study of physical mechanisms, Technometrics, 7, 1, 23-42 (1965)
[6] Broudiscou, A.; Leardi, R.; Phan-Tan-Luu, R., Genetic algorithm as a tool for selection of D-optimal design, Chemometr. Intell. Lab. Syst., 35, 105-116 (1996)
[7] Černý, V., Thermodynamical approach to the traveling salesman problem: An efficient simulation algorithm, J. Optim. Theory Appl., 45, 1, 41-51 (1985) · Zbl 0534.90091
[8] Chaloner, K.; Larntz, K., Optimal Bayesian design applied to logistic regression experiments, J. Statist. Plan. Inf., 21, 191-208 (1989) · Zbl 0666.62073
[9] Chen, R. B.; Hsieh, D. N.; Hung, Y.; Wang, W., Optimizing latin hypercube designs by particle swarm, Stat. Comput., 23, 5, 663-676 (2013) · Zbl 1322.90005
[10] Chen, R. B.; Yang, P.; Wong, W. K., Standardized maximin D-optimal designs for pharmacological models via particle swarm optimization techniques, Chemometr. Intell. Lab. Syst., 169, 79-86 (2017)
[11] Chernoff, H., Locally optimal designs for estimating parameters, Ann. Math. Stat., 24, 4, 586-602 (1953) · Zbl 0053.10504
[12] Cook, R. D.; Nachtsheim, C. J., A comparison of algorithms for constructing exact D-optimal experimental designs, Technometrics, 22, 315-324 (1980) · Zbl 0459.62061
[13] Cuervo, D. P.; Goos, P.; Sorensen, K.; Arraiz, E., An iterated local search algorithm for the vehicle routing problemwith backhauls, European J. Oper. Res., 237, 2, 454-464 (2014) · Zbl 1304.90041
[14] Dette, H.; Melas, V. B., A note on the De la Garza phenomenon for locally optimal designs, Ann. Statist., 39, 2, 1266-1281 (2011) · Zbl 1216.62113
[15] Dette, H.; Melas, V. B.; Wong, W. K., Locally D-optimal designs for exponential regression models, Statist. Sinica, 16, 789-803 (2006) · Zbl 1107.62060
[16] Dorigo, M., Optimization, Learning and Natural Algorithms (1992), Politecnico di Milano, (Ph.D. thesis)
[17] Espinosa-Aranda, J. L.; García-Ródenas, R.; Angulo, E., A framework for derivative free algorithm hybridization, (Adaptive and Natural Computing Algorithms, Vol. 7824 (2013), Springer), 80-89
[18] Espinosa-Aranda, J. L.; García-Ródenas, R.; Ramírez-Flores, M.; López-García, M.; Angulo, E., High-speed railway scheduling based on user preferences, European J. Oper. Res., 246, 3, 772-786 (2015) · Zbl 1346.90341
[19] Esteban-Bravo, M.; Leszkiewicz, A.; Vidal-Sanz, J. M., Exact optimal experimental designs with constraints, Stat. Comput., 21, 4, 1-19 (2016)
[20] Fedorov, V., Theory of Optimal Experiments (1972), Academic Press
[21] García-Ródenas, R.; García-García, J. C.; López-Fidalgo, J.; Martín-Baos, J.Á.; Wong, W. K., ODEm: A Matlab tool for finding optimal approximate experimental designs, MethodsX (2019), (submitted for publication)
[22] Goldberg, D. E., Genetic Algorithms in Search, Optimization and Machine Learning (1989), Addison-Wesley Longman Publishing Co., Inc.: Addison-Wesley Longman Publishing Co., Inc. Boston, MA, USA · Zbl 0721.68056
[23] Haines, L. M., The application of the annealing algorithm to the construction of exact optimal designs for linear-regression models, Technometrics, 29, 439-447 (1987) · Zbl 0632.62071
[24] Hamada, M.; Martz, H. F.; Reese, C. S.; Wilson, A. G., Finding near-optimal Bayesian experimental designs via genetic algorithms, Amer. Statist., 55, 3, 175-181 (2001) · Zbl 1182.62156
[25] Hassan, R.; Cohanim, B.; de Weck, O.; Venter, G., A comparison of particle swarm optimization and the genetic algorithm, (46th AIAA/ASME/ASCE/AHS/ASC Structures, Structural Dynamics and Materials Conference (2005), American Institute of Aeronautics and Astronautics)
[26] Holland, J. H., Adaptation in Natural and Artificial Systems (1975), MIT Press: MIT Press Cambridge, MA, USA, 1992
[27] Huang, Y.; Gilmour, S. G.; Mylona, K.; Goos, P., Optimal design of experiments for non-linear response surface models, J. R. Stat. Soc. Ser. C. Appl. Stat., 68, 3, 623-640 (2019)
[28] Huang, C. L.; Huang, W. C.; Chang, H. Y.; Yeh, Y. C.; Tsai, C. Y., Hybridization strategies for continuous ant colony otimization and particle swarm optimization applied to data clustering, Appl. Soft Comput., 13, 3864-3872 (2013)
[29] Ingber, L., Simulated annealing: Practice versus theory, Math. Comput. Modelling, 18, 11, 20-57 (1993) · Zbl 0819.90080
[30] Johnson, M. E.; Nachtsheim, C. J., Some guidelines for constructing exact D-optimal designs on convex design spaces, Technometrics, 25, 271-277 (1983) · Zbl 0526.62070
[31] Karaboga, D., An Idea Based on Honey Bee Swarm for Numerical OptimizationTechnical Report TR06 (2005), Erciyes University, Engineering Faculty, Computer Engineering Department
[32] Kennedy, J., Eberhart, R., 1995. Particle swarm optimization. In: IEEE International Conference on Neural Networks - Conference Proceedings, Vol. 4, pp. 1942-1948.
[33] Kiefer, J., General equivalence theory for optimum designs (Aproximate theory), Ann. Statist., 2, 5, 848-879 (1974) · Zbl 0291.62093
[34] Kim, S.; Erwin, D.; Wu, D., Efficacy of dual lung cancer screening by chest x-ray and sputum cytology using Johns Hopkins Lung Project Data, Biometrics Biostat., 3, Article 1000139 pp. (2012)
[35] Kim, S.; Li, L., A novel global search algorithm for nonlinear mixed-effects models using particle swarm optimization, J Pharmacokinet. Pharmacodyn., 38, 471-495 (2011)
[36] King, J.; Wong, W. K., Minimax D-optimal designs for the logistic model, Biometrics, 56, 1263-1267 (2000) · Zbl 1060.62545
[37] Kirian, M. S.; Gunduz, M., A recombination-based hybridization of particle swarm optimization and artificial bee colony algorithm for continuous optimization problems, Appl. Soft Comput., 13, 2188-2203 (2013)
[38] Kirpatrick, S.; Gelatt, C. D.; Vechi, M. P., Optimization by simulated annealing, Science, 220, 4598, 671-680 (1983) · Zbl 1225.90162
[39] Latif, A. H.M. M.; Brunner, E., A genetic algorithm for designing microarray experiments, Comput. Statist., 2, 409-424 (2016) · Zbl 1342.65047
[40] Leatherman, E.; Dean, A.; Santner, T., Computer experiment designs via particle swarm optimization, Springer Proc. Math. Stat., 114, 309-317 (2014)
[41] Lin, C. D.; Anderson-Cook, C. M.; Hamada, M. S.; Moore, L. M.; Sitter, R. R., Using genetic algorithms to design experiments: A review, Qual. Reliab. Eng. Int., 31, 2, 155-167 (2015)
[42] López-Fidalgo, J.; Wong, W. K., Design issues for the Michaelis-Menten model, J. Theoret. Biol., 215, 1, 1-11 (2002)
[43] López-García, M. L.; García-Ródenas, R.; Gómez, A. G., Hybrid meta-heuristic optimization algorithms for time-domain-constrained data clustering, Appl. Soft Comput., 23, 319-332 (2014)
[44] Lu, Z.; Pong, T. K., Computing optimal experimental designs via interior point method, SIAM J. Matrix Anal. Appl., 34, 1556-1580 (2013) · Zbl 1425.62096
[45] Lukemire, J.; Mandal, A.; Wong, W. K., d-QPSO: A quantum-behaved particle swarm technique for finding D-optimal designs with discrete and continuous factors and a binary response, Technometrics, 61, 1, 77-87 (2019)
[46] Mak, S.; Joseph, V., Minimax and minimax projection designs using clustering, J. Comput. Graph. Statist., 27, 1, 166-178 (2018)
[47] Mak, S.; Wu, C. F.J., Analysis-of-marginal-tail-means (ATM): A robust method for discrete black-box optimization, Technometrics, 0, 1-15 (2019)
[48] Mandal, A.; Wong, W. K.; Yu, Y., Algorithmic searches for optimal designs, (Handbook of Design and Analysis of Experiments (2015), Chapman and Hall), 89-98
[49] Marini, F.; Walczak, B., Particle swarm optimization (PSO): A tutorial, Chemometr. Intell. Lab. Syst., 315-323 (2015)
[50] Martín-Martín, R.; Torsney, B.; López-Fidalgo, J., Construction of marginally and conditionally restricted designs using multiplicative algorithms, Comput. Statist. Data Anal., 51, 5547-5561 (2007) · Zbl 1445.62201
[51] Meng, X. L., Response: Did Newton-Raphson really fail?, Stat. Methods Med. Res., 3, 23, 312-314 (2014)
[52] Meyer, R.; Nachtsheim, C. J., The coordinate-exchange algorithmm for constructing exact optimal experimental designs, Technometrics, 37, 60-69 (1995) · Zbl 0825.62652
[53] Morales, J. L.; Nocedal, J.; Wu, Y., A sequential quadratic programming algorithm with an additional equality constrained phase, IMA J. Numer. Anal., 32, 2, 553-579 (2011) · Zbl 1246.65093
[54] Nelder, J. A.; Mead, R., A simplex method for function minimization, Comput. J., 7, 308-313 (1965) · Zbl 0229.65053
[55] Nguyen, K.; Miller, A. J., A review of some exchange algorithms for constructing discrete D-optimal designs, Comput. Statist. Data Anal., 14, 489-498 (1992) · Zbl 0937.62628
[56] Pazman, A., Foundations of Optimum Experimental Design, Vol. 14 (1986), Springer
[57] Qin, A. K.; Huang, V. L.; Suganthan, P. N., Differential evolution algorithm with strategy adaptation for global numerical optimization, IEEE Trans. Evol. Comput., 13, 2, 398-417 (2009)
[58] Qiu, J.; Chen, R. B.; Wang, W.; Wong, W. K., Using animal instincts to design efficient biomedical studies via particle swarm optimization, Swarm Evol. Comput., 18, 1-10 (2014)
[59] Royle, J. A., Exchange algorithms for constructing large spatial designs, J. Statist. Plann. Inference, 100, 121-134 (2002) · Zbl 0985.62059
[60] Sengupta, S.; Basak, S.; Peters, R. A., Particle swarm optimization: A survey of historical and recent developments with hybridization perspectives, Mach. Learn. Knowl. Extr., 1, 1, 157-191 (2018)
[61] Shahzad, A.; Kerrigan, E. C.; Constantinides, G. A., A stable and efficient method for solving a convex quadratic program with application to optimal control, SIAM J. Optim., 22, 4, 1369-1393 (2012) · Zbl 1263.90127
[62] Silvey, S. D., Optimal Design (1980), Chapman & Hall · Zbl 0468.62070
[63] Smith, K., On the standard deviations of adjusted and interpolates values of an observed polynomial functions and its constants and the guidance they give towards a proper choice of the distribution of observations, Biometrika, 12, 1-85 (1918)
[64] Socha, K.; Dorigo, M., Ant colony optimization for continuous domains, European J. Oper. Res., 185, 3, 1155-1173 (2008) · Zbl 1146.90537
[65] Stallings, J. W.; Morgan, J. P., General weighted optimality of designed experiments, Biometrika, 102, 4, 925-935 (2015) · Zbl 1372.62032
[66] Su, P. L.; Chen, Y. S., Implementation of a genetic algorithm on MD-optimal designs for multivariate response surface models, Expert Syst. Appl., 39, 3207-3212 (2012)
[67] Sun, W.; Yuan, Y. X., Optimization Theory and Methods: Nonlinear Programming (2006), Springer
[68] Wang, L., Si, G., 2010. Optimal location management in mobile computing with hybrid genetic algorithm and particle swarm optimization (GA-PSO). In: 2010 17th IEEE International Conference on Electronics, Circuits and Systems, pp. 1160-1163.
[69] Wei, W.; Tian, Z., An improved multi-objective optimization method based on adaptive mutation particle swarm optimization and fuzzy statistics algorithm, J. Stat. Comput. Simul., 87, 2480-2493 (2017)
[70] Wolpert, D. H.; Macready, W. G., No free lunch theorems for optimization, IEEE Trans. Evol. Comput., 1, 67-82 (1997)
[71] Wong, W. K.; Chen, R. B.; Huang, C. C.; Wang, W., A modified particle swarm optimization technique for finding optimal designs for mixture models, PLoS One, 10, 6, Article e0124720 pp. (2015)
[72] Wynn, H. P., Results in the theory and construction of D-optimum experimental designs, J. R. Stat. Soc. Ser. B Stat. Methodol., 73, 133-147 (1972) · Zbl 0248.62033
[73] Yang, X. S., Nature-Inspired Metaheuristic Algorithms (2010), Luniver Press
[74] Yang, M.; Biedermann, S.; Tang, E., On optimal designs for nonlinear models: A general and efficient algorithm, J. Amer. Statist. Assoc., 108, 504, 1411-1420 (2013) · Zbl 1283.62161
[75] Yang, J.; Tong, L.; Mandal, A., D-optimal designs with ordered categorical data, Statist. Sinica, 27, 4, 1879-1902 (2017) · Zbl 1392.62233
[76] Yu, Y., D-optimal designs via a cocktail algorithm, Stat. Comput., 21, 4, 475-481 (2011) · Zbl 1270.62107
[77] Zambrano-Bigiarini, M., Clerc, M., Rojas, R., 2013. Standard particle swarm optimisation 2011 at CEC-2013: A baseline for future PSO improvements. In: Evolutionary Computation (CEC), 2013 IEEE Congress on, pp. 2337-2344.
This reference list is based on information provided by the publisher or from digital mathematics libraries. Its items are heuristically matched to zbMATH identifiers and may contain data conversion errors. It attempts to reflect the references listed in the original paper as accurately as possible without claiming the completeness or perfect precision of the matching.