zbMATH — the first resource for mathematics

Examples
Geometry Search for the term Geometry in any field. Queries are case-independent.
Funct* Wildcard queries are specified by * (e.g. functions, functorial, etc.). Otherwise the search is exact.
"Topological group" Phrases (multi-words) should be set in "straight quotation marks".
au: Bourbaki & ti: Algebra Search for author and title. The and-operator & is default and can be omitted.
Chebyshev | Tschebyscheff The or-operator | allows to search for Chebyshev or Tschebyscheff.
"Quasi* map*" py: 1989 The resulting documents have publication year 1989.
so: Eur* J* Mat* Soc* cc: 14 Search for publications in a particular source with a Mathematics Subject Classification code (cc) in 14.
"Partial diff* eq*" ! elliptic The not-operator ! eliminates all results containing the word elliptic.
dt: b & au: Hilbert The document type is set to books; alternatively: j for journal articles, a for book articles.
py: 2000-2015 cc: (94A | 11T) Number ranges are accepted. Terms can be grouped within (parentheses).
la: chinese Find documents in a given language. ISO 639-1 language codes can also be used.

Operators
a & b logic and
a | b logic or
!ab logic not
abc* right wildcard
"ab c" phrase
(ab c) parentheses
Fields
any anywhere an internal document identifier
au author, editor ai internal author identifier
ti title la language
so source ab review, abstract
py publication year rv reviewer
cc MSC code ut uncontrolled term
dt document type (j: journal article; b: book; a: book article)
A study on the use of non-parametric tests for analyzing the evolutionary algorithms’ behaviour: a case study on the CEC’2005 special session on real parameter optimization. (English) Zbl 1191.68828

Summary: In recent years, there has been a growing interest for the experimental analysis in the field of evolutionary algorithms. It is noticeable due to the existence of numerous papers which analyze and propose different types of problems, such as the basis for experimental comparisons of algorithms, proposals of different methodologies in comparison or proposals of use of different statistical techniques in algorithms’ comparison.

In this paper, we focus our study on the use of statistical techniques in the analysis of evolutionary algorithms’ behaviour over optimization problems. A study about the required conditions for statistical analysis of the results is presented by using some models of evolutionary algorithms for real-coding optimization. This study is conducted in two ways: single-problem analysis and multiple-problem analysis. The results obtained state that a parametric statistical analysis could not be appropriate specially when we deal with multiple-problem results. In multiple-problem analysis, we propose the use of non-parametric statistical tests given that they are less restrictive than parametric ones and they can be used over small size samples of results. As a case study, we analyze the published results for the algorithms presented in the CEC’2005 Special Session on Real Parameter Optimization by using non-parametric test procedures.

MSC:
68W05Nonnumerical algorithms
68T05Learning and adaptive systems
Software:
SPOT_
References:
[1]Abramowitz, M.: Handbook of Mathematical Functions, With Formulas, Graphs, and Mathematical Tables. Dover, New York (1974)
[2]Auger, A., Hansen, N.: A restart CMA evolution strategy with increasing population size. In: Proceedings of the 2005 IEEE Congress on Evolutionary Computation (CEC’2005), pp. 1769–1776 (2005a)
[3]Auger, A., Hansen, N.: Performance evaluation of an advanced local search evolutionary algorithm. In: Proceedings of the 2005 IEEE Congress on Evolutionary Computation (CEC’2005), pp. 1777–1784 (2005b)
[4]Ballester, P.J., Stephenson, J., Carter, J.N., Gallagher, K.: Real-parameter optimization performance study on the CEC-2005 benchmark with SPC-PNX. In: Proceedings of the 2005 IEEE Congress on Evolutionary Computation (CEC’2005), pp. 498–505 (2005)
[5]Bartz-Beielstein, T.: Experimental Research in Evolutionary Computation: The New Experimentalism. Springer, New York (2006)
[6]Czarn, A., MacNish, C., Vijayan, K., Turlach, R., Gupta, R.: Statistical exploratory analysis of genetic algorithms. IEEE Trans. Evol. Comput. 8(4), 405–421 (2004) · Zbl 05451909 · doi:10.1109/TEVC.2004.831262
[7]Demšar, J.: Statistical comparisons of classifiers over multiple data sets. J. Mach. Learn. Res. 7, 1–30 (2006)
[8]Gallagher, M., Yuan, B.: A general-purpose tunable landscape generator. IEEE Trans. Evol. Comput. 10(5), 590–603 (2006) · Zbl 05452181 · doi:10.1109/TEVC.2005.863628
[9]García, S., Molina, D., Lozano, M., Herrera, F.: An experimental study on the use of non-parametric tests for analyzing the behaviour of evolutionary algorithms in optimization problems. In: Proceedings of the Spanish Congress on Metaheuristics, Evolutionary and Bioinspired Algorithms (MAEB’2007), pp. 275–285 (2007) (in Spanish)
[10]García-Martínez, C., Lozano, M.: Hybrid real-coded genetic algorithms with female and male differentiation. In: Proceedings of the 2005 IEEE Congress on Evolutionary Computation (CEC’2005), pp. 896–903 (2005)
[11]Hansen, N.: (2005). Compilation of results on the CEC benchmark function set. Tech. Report, Institute of Computational Science, ETH Zurich, Switzerland. Available as http://www.ntu.edu.sg/home/epnsugan/index_files/CEC-05/compareresults.pdf
[12]Hochberg, Y.: A sharper Bonferroni procedure for multiple tests of significance. Biometrika 75, 800–803 (1988) · Zbl 0661.62067 · doi:10.1093/biomet/75.4.800
[13]Holm, S.: A simple sequentially rejective multiple test procedure. Scandinavian J. Statist. 6, 65–70 (1979)
[14]Hooker, J.: Testing heuristics: we have it all wrong. J. Heuristics 1(1), 33–42 (1997) · Zbl 0853.68155 · doi:10.1007/BF02430364
[15]Iman, R.L., Davenport, J.M.: Approximations of the critical region of the Friedman statistic. Commun. Stat. 18, 571–595 (1980) · Zbl 0451.62061 · doi:10.1080/03610928008827904
[16]Kramer, O.: An experimental analysis of evolution strategies and particle swarm optimisers using design of experiments. In: Proceedings of the Genetic and Evolutionary Computation Conference 2007 (GECCO’2007), pp. 674–681 (2007)
[17]Liang, J.J., Suganthan, P.N.: Dynamic multi-swarm particle swarm optimizer with local search. In: Proceedings of the 2005 IEEE Congress on Evolutionary Computation (CEC’2005), pp. 522–528 (2005)
[18]Molina, D., Herrera, F., Lozano, M.: Adaptive local search parameters for real-coded memetic algorithms. In: Proceedings of the 2005 IEEE Congress on Evolutionary Computation (CEC’2005), pp. 888–895 (2005)
[19]Moreno-Pérez, J.A., Campos-Rodríguez, C., Laguna, M.: On the comparison of metaheuristics through non-parametric statistical techniques. In: Proceedings of the Spanish Congress on Metaheuristics, Evolutionary and Bioinspired Algorithms (MAEB’2007), pp. 286–293 (2007) (in Spanish)
[20]Morse, D.T.: Minsize2: a computer program for determining effect size and minimum sample size for statistical significance for univariate, multivariate, and nonparametric tests. Educ. Psychol. Meas. 59(3), 518–531 (1999) · doi:10.1177/00131649921969901
[21]Noether, G.E.: Sample size determination for some common nonparametric tests. J. Am. Stat. Assoc. 82(398), 645–647 (1987) · Zbl 0622.62049 · doi:10.2307/2289477
[22]Ortiz-Boyer, D., Hervás-Martínez, C., García-Pedrajas, N.: Improving crossover operators for real-coded genetic algorithms using virtual parents. J. Heuristics 13, 265–314 (2007) · Zbl 05193467 · doi:10.1007/s10732-007-9018-2
[23]Ozcelik, B., Erzurumlu, T.: Comparison of the warpage optimization in the plastic injection molding using ANOVA, neural network model and genetic algorithm. J. Mater. Process. Technol. 171(3), 437–445 (2006) · doi:10.1016/j.jmatprotec.2005.04.120
[24]Patel, J.K., Read, C.B.: Handbook of the Normal Distribution. Dekker, New York (1982)
[25]Pošík, P.: Real-parameter optimization using the mutation step co-evolution. In: Proceedings of the 2005 IEEE Congress on Evolutionary Computation (CEC’2005), pp. 872–879 (2005)
[26]Qin, A.K., Suganthan, P.N.: Self-adaptive differential evolution algorithm for numerical optimization. In: Proceedings of the 2005 IEEE Congress on Evolutionary Computation (CEC’2005), pp. 1785–1791 (2005)
[27]Rojas, I., González, J., Pomares, H., Merelo, J.J., Castillo, P.A., Romero, G.: Statistical analysis of the main parameters involved in the design of a genetic algorithm. IEEE Trans. Syst. Man Cybern. Part C 32(1), 31–37 (2002) · doi:10.1109/TSMCC.2002.1009128
[28]Rônkkônen, J., Kukkonen, S., Price, K.V.: Real-parameter optimization using the mutation step co-evolution. In: Proceedings of the 2005 IEEE Congress on Evolutionary Computation (CEC’2005), pp. 506–513 (2005)
[29]Shaffer, J.P.: Multiple hypothesis testing. Annu. Rev. Psychol. 46, 561–584 (1995) · doi:10.1146/annurev.ps.46.020195.003021
[30]Sinha, A., Tiwari, S., Deb, K.: A population-based, steady-state procedure for real-parameter optimization. In: Proceedings of the 2005 IEEE Congress on Evolutionary Computation (CEC’2005), pp. 514–521 (2005)
[31]Suganthan, P.N., Hansen, N., Liang, J.J., Deb, K., Chen, Y.P., Auger, A., Tiwari, S.: (2005). Problem definitions and evaluation criteria for the CEC 2005 Special Session on Real Parameter Optimization. Tech. Report, Nanyang Technological University. Available as http://www.ntu.edu.sg/home/epnsugan/index_files/CEC-05/Tech-Report-May-30-05.pdf
[32]Sheskin, D.J.: Handbook of Parametric and Nonparametric Statistical Procedures. CRC Press, Boca Raton (2003)
[33]Whitley, D.L., Beveridge, R., Graves, C., Mathias, K.E.: Test driving three 1995 genetic algorithms: new test functions and geometric matching. J. Heuristics 1(1), 77–104 (1995) · Zbl 0853.68106 · doi:10.1007/BF02430367
[34]Whitley, D.L., Rana, S., Dzubera, J., Mathias, K.E.: Evaluating evolutionary algorithms. Artif. Intell. 85(1–2), 245–276 (1996) · doi:10.1016/0004-3702(95)00124-7
[35]Wolpert, D.H., Macready, W.G.: No free lunch theorems for optimization. IEEE Trans. Evol. Comput. 1(1), 67–82 (1997) · Zbl 05451863 · doi:10.1109/4235.585893
[36]Wright, S.P.: Adjusted p-values for simultaneous inference. Biometrics 48, 1005–1013 (1992) · doi:10.2307/2532694
[37]Yuan, B., Gallagher, M.: On building a principled framework for evaluating and testing evolutionary algorithms: a continuous landscape generator. In: Proceedings of the 2003 Congress on Evolutionary Computation (CEC2003), pp. 451–458 (2003)
[38]Yuan, B., Gallagher, M.: Experimental results for the Special Session on Real-Parameter Optimization at CEC 2005: a simple, continuous EDA. In: Proceedings of the 2005 IEEE Congress on Evolutionary Computation (CEC’2005), pp. 1792–1799 (2005)
[39]Zar, J.H.: Biostatistical Analysis. Prentice Hall, Englewood Cliffs (1999)