MMCTest – a safe algorithm for implementing multiple Monte Carlo tests. (English) Zbl 1305.62270

Summary: Consider testing multiple hypotheses using tests that can only be evaluated by simulation, such as permutation tests or bootstrap tests. This article introduces MMCTest, a sequential algorithm that gives, with arbitrarily high probability, the same classification as a specific multiple testing procedure applied to ideal \(p\)-values. The method can be used with a class of multiple testing procedures that include the Benjamini and Hochberg false discovery rate procedure [Y. Benjamini and Y. Hochberg, J. R. Stat. Soc., Ser. B 57, No. 1, 289–300 (1995; Zbl 0809.62014)] and the Bonferroni correction [C. E. Bonferroni [Teoria statistica delle classi e calcolo delle probabilita. Firenze: Libr. Internaz. Seeber (1936; Zbl 0016.41103)] controlling the familywise error rate. One of the key features of the algorithm is that it stops sampling for all the hypotheses that can already be decided as being rejected or non-rejected. MMCTest can be interrupted at any stage and then returns three sets of hypotheses: the rejected, the non-rejected and the undecided hypotheses. A simulation study motivated by actual biological data shows that MMCTest is usable in practice and that, despite the additional guarantee, it can be computationally more efficient than other methods.


62J15 Paired and multiple comparisons; multiple testing
60J22 Computational methods in Markov chains
Full Text: DOI arXiv


[1] Benjamini, Controlling the false discovery rate: a practical and powerful approach to multiple testing, J. R. Stat. Soc. Ser. B Stat. Methodol. 57 (1) pp 289– (1995) · Zbl 0809.62014
[2] Besag, Sequential Monte Carlo p-values, Biometrika 78 (2) pp 301– (1991)
[3] Bonferroni, Teoria statistica delle classi e calcolo delle probabilit√†, Pubbl. d. R. Ist. Super. di Sci. Econom. e Commerciali di Firenze 8 pp 3– (1936) · Zbl 0016.41103
[4] Clopper, The use of confidence or fiducial limits illustrated in the case of the binomial, Biometrika 26 (4) pp 404– (1934) · JFM 60.1175.02
[5] Cohen, Uncovering the co-evolutionary network among prokaryotic genes, Bioinformatics 28 pp i389– (2012)
[6] Farcomeni, Some results on the control of the false discovery rate under dependence, Scand. J. Stat. 34 (2) pp 275– (2007) · Zbl 1142.62048
[7] Farcomeni, Generalized augmentation to control the false discovery exceedance in multiple testing, Scand. J. Stat. 36 (3) pp 501– (2009) · Zbl 1189.62116
[8] Finner, False discovery rate control of step-up-down tests with special emphasis on the asymptotically optimal rejection curve, Scand. J. Stat. 39 (2) pp 382– (2012) · Zbl 1246.62171
[9] Gandy, Sequential implementation of Monte Carlo tests with uniformly bounded resampling risk, J. Amer. Statist. Assoc. 104 (488) pp 1504– (2009) · Zbl 1205.65016
[10] Gandy, An algorithm to compute the power of Monte Carlo tests with guaranteed precision, Ann. Statist. 41 (1) pp 125– (2013) · Zbl 1347.62011
[11] Gleser, Comment on ’Bootstrap Confidence Intervals’ by T. J. DiCiccio and B. Efron, Statist. Sci. 11 pp 219– (1996)
[12] Guo, Adaptive choice of the number of bootstrap samples in large scale multiple testing, Stat. Appl. Genet. Mol. Biol. 7 (1) pp 1– (2008) · Zbl 1276.62072
[13] Gusenleitner, iBBiG: iterative binary bi-clustering of gene sets, Bioinformatics 28 (19) pp 2484– (2012)
[14] Hoeffding, Probability inequalities for sums of bounded random variables, J. Amer. Statist. Assoc. 58 (301) pp 13– (1963) · Zbl 0127.10602
[15] Jiang, Statistical properties of an early stopping rule for resampling-based multiple testing, Biometrika 99 (4) pp 973– (2012) · Zbl 1452.62557
[16] Jiao, A mixture model based approach for estimating the FDR in replicated microarray data, Journal of Biomedical Science and Engineering 20 (11) pp 317– (2010)
[17] Knijnenburg, Combinatorial effects of environmental parameters on transcriptional regulation in Saccharomyces cerevisiae: a quantitative analysis of a compendium of chemostat-based transcriptome data, BMC Genomics 10 (53) (2009)
[18] Lage-Castellanos, False discovery rate and permutation test: an evaluation in ERP data analysis, Stat. Med. 29 pp 63– (2010)
[19] Li, Confidence interval for the bootstrap P-value and sample size calculation of the bootstrap test, J. Nonparametr. Stat. 21 (5) pp 649– (2009) · Zbl 1165.62316
[20] Meinshausen, False discovery control for multiple tests of association under general dependence, Scand. J. Stat. 33 (2) pp 227– (2006) · Zbl 1125.62077
[21] Nusinow, Network-based inference from complex proteomic mixtures using SNIPE, Bioinformatics 28 (23) pp 3115– (2012)
[22] Pekowska, A unique H3K4me2 profile marks tissue-specific gene regulation, Genome Research 20 (11) pp 1493– (2010)
[23] Pounds, Robust estimation of the false discovery rate, Bioinformatics 22 (16) pp 1979– (2006)
[24] Rahmatallah, Gene set analysis for self-contained tests: complex null and specific alternative hypotheses, Bioinformatics 28 (23) pp 3073– (2012)
[25] Sandve, Sequential Monte Carlo multiple testing, Bioinformatics 27 (23) pp 3235– (2011)
[26] Tamhane, On weighted Hochberg procedures, Biometrika 95 (2) pp 279– (2008) · Zbl 1437.62623
[27] Wieringen, A test for partial differential expression, J. Amer. Statist. Assoc. 103 (483) pp 1039– (2008) · Zbl 1205.62189
[28] Westfall, Resampling-based multiple testing: examples and methods for p-value adjustment (1993)
[29] Westfall, Multiple testing with minimal assumptions, Biom. J. 50 (5) pp 745– (2008) · Zbl 05361932
This reference list is based on information provided by the publisher or from digital mathematics libraries. Its items are heuristically matched to zbMATH identifiers and may contain data conversion errors. It attempts to reflect the references listed in the original paper as accurately as possible without claiming the completeness or perfect precision of the matching.