The sequential rejection principle of familywise error control. (English) Zbl 1204.62140

Summary: Closed testing and partitioning are recognized as fundamental principles of familywise error control. We argue that sequential rejection can be considered equally fundamental as a general principle of multiple testing. We present a general sequentially rejective multiple testing procedure and show that many well-known familywise error controlling methods can be constructed as special cases of this procedure, among which are the procedures of S. Holm [Scand. J. Stat., Theory Appl. 6, 65–70 (1979; Zbl 0402.62058)], J. P. Shaffer [J. Am. Stat. Assoc. 81, 826–831 (1986; Zbl 0603.62087)] and Y. Hochberg [Biometrika 75, No. 4, 800–802 (1988; Zbl 0661.62067)], parallel and serial gatekeeping procedures, modern procedures for multiple testing in graphs, resampling-based multiple testing procedures and even the closed testing and partitioning procedures themselves.
We also give a general proof that sequentially rejective multiple testing procedures strongly control the familywise error if they fulfill simple criteria of monotonicity of the critical values and a limited form of weak familywise error control in each single step. The sequential rejection principle gives a novel theoretical perspective on many well-known multiple testing procedures, emphasizing the sequential aspect. Its main practical usefulness is for the development of multiple testing procedures for null hypotheses, possibly logically related, that are structured in a graph. We illustrate this by presenting a uniform improvement of a recently published procedure.


62L10 Sequential statistical analysis
62H15 Hypothesis testing in multivariate analysis
62J15 Paired and multiple comparisons; multiple testing
05C90 Applications of graph theory
Full Text: DOI arXiv


[1] Bretz, F., Maurer, W., Brannath, W. and Posch, M. (2009). A graphical approach to sequentially rejective multiple test procedures. Stat. Med. 28 586-604.
[2] Burman, C.-F., Sonesson, C. and Guilbaud, O. (2009). A recycling framework for the construction of Bonferroni-based multiple tests. Stat. Med. 28 739-761.
[3] Calian, V., Li, D. M. and Hsu, J. C. (2008). Partitioning to uncover conditions for permutation tests to control multiple testing error rates. Biometrical J. 50 756-766.
[4] Dmitrienko, A., Offen, W. W. and Westfall, P. H. (2003). Gatekeeping strategies for clinical trials that do not require all primary effects to be significant. Stat. Med. 22 2387-2400.
[5] Dmitrienko, A. and Tamhane, A. C. (2007). Gatekekeeping procedures with clinical trial applications. Pharmaceutical Statistics 6 171-180.
[6] Dmitrienko, A., Tamhane, A. C., Wang, X. and Chen, X. (2006). Stepwise gatekeeping procedures in clinical trial applications. Biometrical J. 48 984-991.
[7] Dmitrienko, A., Tamhane, A. C. and Wiens, B. L. (2008). General multistage gatekeeping procedures. Biometrical J. 50 667-677.
[8] Dmitrienko, A., Wiens, B. L., Tamhane, A. C. and Wang, X. (2007). Tree-structured gatekeeping tests in clinical trials with hierarchically ordered multiple objectives. Stat. Med. 26 2465-2478.
[9] Dudoit, S. and Van der Laan, M. J. (2008). Multiple Testing Procedures with Applications to Genomics . Springer, New York. · Zbl 1261.62014
[10] Dudoit, S., Van der Laan, M. J. and Pollard, K. S. (2004). Multiple testing part I: Single-step procedures for control of general type I error rates. Stat. Appl. Genet. Mol. Biol. 3 Article 13. · Zbl 1166.62338
[11] Edwards, D. and Madsen, J. (2007). Constructing multiple test procedures for partially ordered hypotheses sets. Stat. Med. 26 5116-5124.
[12] Finner, H. and Strassburger, K. (2002). The partitioning principle: A powerful tool in multiple decision theory. Ann. Statist. 30 1194-1213. · Zbl 1029.62064
[13] Ge, Y., Dudoit, S. and Speed, T. P. (2003). Resampling-based multiple testing for microarray data analysis. Test 12 1-77. · Zbl 1056.62117
[14] Goeman, J. J. and Mansmann, U. (2008). Multiple testing on the directed acyclic graph of gene ontology. Bioinformatics 24 537-544.
[15] Guilbaud, O. (2007). Bonferroni parallel gatekeeping-Transparant generalizations, adjusted p -values, and short direct proofs. Biometrical J. 49 917-927.
[16] Hochberg, Y. (1988). A sharper Bonferroni procedure for multiple tests of significance. Biometrika 75 800-802. JSTOR: · Zbl 0661.62067
[17] Hochberg, Y. and Rom, D. (1995). Extensions of multiple testing procedures based on Simes’ test. J. Statist. Plann. Inference 48 141-152. · Zbl 0851.62054
[18] Holland, B. S. and DiPinzio Copenhaver, M. (1987). An improved sequentially rejective Bonferroni test procedure. Biometrics 43 417-423. JSTOR: · Zbl 0654.62068
[19] Holm, S. (1979). A simple sequentially rejective multiple test procedure. Scand. J. Statist. 6 65-70. · Zbl 0402.62058
[20] Hommel, G. (1988). A stagewise rejective multiple test procedure based on a modified Bonferroni test. Biometrika 75 383-386. · Zbl 0639.62025
[21] Hommel, G. and Bernhard, G. (1999). Bonferroni procedures for logically related hypotheses. J. Statist. Plann. Inference 82 119-128. · Zbl 1079.62526
[22] Hommel, G., Bretz, F. and Maurer, W. (2007). Powerful short-cuts for multiple testing procedures with special reference to gatekeeping strategies. Stat. Med. 26 4063-4073.
[23] Lehmann, E. L. and Romano, J. P. (2005). Testing Statistical Hypotheses . Springer, New York. · Zbl 1076.62018
[24] Marcus, R., Peritz, E. and Gabriel, K. R. (1976). On closed testing procedures with special reference to ordered analysis of variance. Biometrika 63 655-660. JSTOR: · Zbl 0353.62037
[25] Meinshausen, N. (2008). Hierarchical testing of variable importance. Biometrika 95 265-278. · Zbl 1437.62557
[26] Romano, J. and Wolf, M. (2005). Exact and approximate stepdown methods for multiple hypotheses testing. J. Amer. Statist. Assoc. 100 94-108. · Zbl 1117.62416
[27] Romano, J. and Wolf, M. (2010). Balanced control of generalized error rates. Ann. Statist. 38 598-633. · Zbl 1181.62110
[28] Rosenbaum, P. R. (2008). Testing hypotheses in order. Biometrika 25 248-252. · Zbl 1437.62594
[29] Sarkar, S. K. (1998). Some probability inequalities for ordered MTP 2 random variables: A proof of the Simes conjecture. Ann. Statist. 26 494-504. · Zbl 0929.62065
[30] Shaffer, J. P. (1986). Modified sequentially rejective multiple test procedures. J. Amer. Statist. Assoc. 81 826-831. · Zbl 0603.62087
[31] Šidák, Z. (1967). Rectangular confidence regions for the means of multivariate normal distributions. J. Amer. Statist. Assoc. 62 626-633. · Zbl 0158.17705
[32] Simes, R. J. (1986). An improved Bonferroni procedure for multiple tests of significance. Biometrika 73 751-754. JSTOR: · Zbl 0613.62067
[33] Stefansson, G., Kim, W. and Hsu, J. C. (1988). On confidence sets in multiple comparisons. In Statistical Decision Theory and Related Topics IV (S. S. Gupta and J. O. Berger, eds.) 2 89-104. Springer, New York. · Zbl 0685.62034
[34] Westfall, P. H. and Krishen, A. (2001). Optimally weighted, fixed sequence and gatekeeper multiple testing procedures. J. Statist. Plann. Inference 99 25-40. · Zbl 0985.62056
[35] Westfall, P. H. and Troendle, J. F. (2008). Multiple testing with minimal assumptions. Biometrical J. 50 745-755.
[36] Westfall, P. H. and Young, S. S. (1993). Resampling-Based Multiple Testing: Examples and Methods for p-Value Adjustment . Wiley, New York. · Zbl 0850.62368
[37] Wiens, B. L. and Dmitrienko, A. (2005). The fallback procedure for evaluating a single family of hypotheses. J. Biopharm. Statist. 15 929-942.
This reference list is based on information provided by the publisher or from digital mathematics libraries. Its items are heuristically matched to zbMATH identifiers and may contain data conversion errors. It attempts to reflect the references listed in the original paper as accurately as possible without claiming the completeness or perfect precision of the matching.