Exact permutation/randomization tests algorithms. (English) Zbl 1458.62159

Summary: R. A. Fisher [The design of experiments. Edinburgh, London: Oliver & Boyd (1937; JFM 61.0566.03)] described the exact permutation and randomization tests for comparative experiments without assuming normality or any particular probability distribution (see also [R. A. Fisher, Statistical methods, experimental design and scientific inference. A re-issue of ‘Statistical methods for research workers’, ‘The design of experiments’, and ‘Statistical methods and scientific inference’. Ed. by J. H. Bennett. With a foreword by F. Yates. Oxford: Oxford University Press (1990; Zbl 0705.62003)]). While having this as an attractive feature, the computational challenge was a disadvantage at that time but not now with modern computers. This paper introduces a permutation/randomization data algorithm to generate the permutation/randomization distributions under the null hypotheses for calculating the \(P\)-values. The properties of permutation/randomization data matrices developed by algorithms following the proposed mathematical processes are derived. Two illustrative examples demonstrate the usefulness of the proposed computational methods.


62K10 Statistical block designs
62G10 Nonparametric hypothesis testing
62-08 Computational methods for problems pertaining to statistics
Full Text: DOI


[1] Basu, D., Randomization analysis of experimental data: the Fisher randomization test, J Am Statist Assoc, 75, 575-582 (1980) · Zbl 0444.62089
[2] Berry, KJ; Johnson, JE; Mielke, PW, The measurement of association: a permutation statistical approach (2018), Switzerland: Springer, Switzerland
[3] Boik, RJ, The Fisher-Pitman permutation test: a non-robust alternative to the normal theory F Test when variances are Heterogeneous, British J Math Statist Psychol, 40, 26-42 (1987) · Zbl 0616.62056
[4] Box, GEP; Andersen, SL, Permutation theory in the derivation of robust criteria and the study of departures from assumption, J R Stat Society Ser B, 17, 1-34 (1955) · Zbl 0065.12402
[5] Box, GEP; Hunter, WG; Hunter, JS, Statistics for experimenters: an introduction to design, data analysis, and model building (1978), New York: Wiley, New York
[6] Calinski, T.; Kageyama, S., Block designs: a randomization approach. Vol 1: Analysis (2000), New York: Springer, New York · Zbl 0963.62071
[7] Calinski, T.; Kageyama, S., Block designs: a randomization approach. Vol 2: Design (2003), New York: Springer, New York · Zbl 1108.62318
[8] David, HA, The beginnings of randomization tests, Am Stat, 62, 70-72 (2008)
[9] Dugard, P., Randomization tests: a new gold standard?, J Context Behav Sci, 3, 65-68 (2014)
[10] Easterling, RG, Fundamentals of statistical experimental design and analysis (2015), New York: Wiley, New York
[11] Edgington, ES; Onghena, P., Randomization tests (2007), New York: Chapman & Hall, New York
[12] Effron, B.; Hastie, T., Computer age statistical inference : algorithms, evidence, and data Science (2016), New York: Cambridge University Press, New York
[13] Ernst, MD, Permutation methods: a basis for exact inference, Stat Sci, 19, 676-685 (2004) · Zbl 1100.62563
[14] Ferron, JM; Levin, JR; Kratochwill, TR; Levin, JR, Single-case permutation and randomization statistical tests: Present status, promising new developments, Single-case intervention research: statistical and methodological advances, 153-183 (2014), Washington: American Psychological Association, Washington
[15] Fisher, RA, The design of experiments (1935), London: Oliver & Boyd, London
[16] Good, PI, Permutation, parametric and bootstrap tests of hypotheses (2005), New York: Springer, New York
[17] Hoeffding, W., The large-sample power of tests based on permutations of observations, Ann Math Stat, 23, 169-192 (1952) · Zbl 0046.36403
[18] Kempthorne, O., The randomization theory of experimental inference, J Am Stat Assoc, 50, 946-967 (1955)
[19] Kempthorne, O.; Doerfler, TE, The behaviour of some significance tests under experimental randomization, Biometrika, 56, 231-248 (1969) · Zbl 0175.17003
[20] Lehmann, EL, Non-parametrics: statistical methods based on ranks (1975), San Francisco: Holden-Day, San Francisco
[21] Ludbrook, J.; Dudley, H., Why permutation tests are superior to t and F Tests in biomedical research, Am Statist, 52, 127-132 (1998)
[22] Manly, BFJ, Randomization, bootstrap and Monte Carlo methods in biology (2007), Boca Raton: Chapman & Hall/CRC, Boca Raton
[23] Mielke, PW; Berry, KJ, Permutation methods: a distance function approach (2007), New York: Springer, New York
[24] Onghena, P.; Berger, V., Randomization and the randomization test: two sides of the same coin, Randomization, masking, and allocation concealment, 185-207 (2018), Boca Raton: Chapman & Hall/CRC Press, Boca Raton
[25] Pesarin, F.; Salmaso, L., Permutation tests for complex data: theory, applications and software (2010), Chichester: Wiley, Chichester
[26] Pitman, EJG, Significance tests which may be applied to samples from any population, I, J R Stat Soc Ser B, 4, 119-130 (1937) · JFM 63.1095.01
[27] Pitman, EJG, Significance tests which may be applied to samples from any populations, II, the correlation coefficient test, Suppl J R Stat Soc, 4, 225-232 (1937) · Zbl 0019.03503
[28] Pitman, EJG, Significance tests which may be applied to samples from any populations, III, Analf Var Test Biometrika, 29, 322-335 (1938) · Zbl 0018.22601
[29] Rao, CR, Linear statistical inference and its applications (1973), New York: Wiley, New York
[30] Tukey, JW, The future of data analysis, Ann Math Stat, 33, 1-67 (1962) · Zbl 0107.36401
[31] Tukey, JW, Randomization and rerandomization: the wave of the past in the future (1988), Philadelphia: Ciminera Symposium, Philadelphia
[32] Wald, A.; Wolfowitz, J., Statistical tests based on permutations of the observations, Ann Math Stat, 15, 358-372 (1944) · Zbl 0063.08124
[33] Welch, BL, On the z-test in randomized blocks and Latin squares, Biometrika, 29, 21-52 (1937) · Zbl 0017.12602
[34] Yates, F., Sir Ronald Fisher and the design of experiments, Biometrics, 20, 307-321 (1964)
This reference list is based on information provided by the publisher or from digital mathematics libraries. Its items are heuristically matched to zbMATH identifiers and may contain data conversion errors. It attempts to reflect the references listed in the original paper as accurately as possible without claiming the completeness or perfect precision of the matching.