## On the expected runtime of multiple testing algorithms with bounded error.(English)Zbl 1450.62055

Summary: Consider testing multiple hypotheses in the setting where the $$p$$-values of all hypotheses are unknown and thus have to be approximated using Monte Carlo simulations. One class of algorithms published in the literature for this scenario provides guarantees on the correctness of their testing result through the computation of confidence statements on all approximated $$p$$-values. This article focuses on the expected runtime of such algorithms and derives a variety of finite and infinite expected runtime results.

### MSC:

 62H15 Hypothesis testing in multivariate analysis 62J15 Paired and multiple comparisons; multiple testing 65C05 Monte Carlo methods

### Software:

MMCTest; simctest; QuickMMCTest; ORIOGEN; MCFDR
Full Text:

### References:

 [1] Andrews, D.; Buchinsky, M., A three-step method for choosing the number of bootstrap repetitions, Econometrica, 68, 1, 23-51 (2000) · Zbl 1056.62516 [2] Andrews, D.; Buchinsky, M., Evaluation of a three-step method for choosing the number of bootstrap repetitions, J. Econometrics, 103, 1-2, 345-386 (2003) · Zbl 1025.62019 [3] Armitage, P., Numerical studies in the sequential estimation of a binomial parameter, Biometrika, 45, 1-2, 1-15 (1958) · Zbl 0085.13709 [4] Benjamini, Y.; Hochberg, Y., Controlling the false discovery rate: A practical and powerful approach to multiple testing, J. R. Stat. Soc. B Methodol., 57, 1, 289-300 (1995) · Zbl 0809.62014 [5] Benjamini, Y.; Yekutieli, D., The control of the false discovery rate in multiple testing under dependency, Ann. Statist., 29, 4, 1165-1188 (2001) · Zbl 1041.62061 [6] Besag, J.; Clifford, P., Sequential Monte Carlo p-values, Biometrika, 78, 2, 301-304 (1991) [7] Bonferroni, C., Teoria statistica delle classi e calcolo delle probabilità, Publ. R Ist. Super. Sci. Econ. Commer. Firenze, 8, 3-62 (1936) · Zbl 0016.41103 [8] Clopper, C.; Pearson, E., The use of confidence or fiducial limits illustrated in the case of the binomial, Biometrika, 26, 4, 404-413 (1934) · JFM 60.1175.02 [9] Darling, D.; Robbins, H., Confidence sequences for mean, variance, and median, Proc. Natl. Acad. Sci. USA, 58, 1, 66-68 (1967) · Zbl 0173.21002 [10] Darling, D.; Robbins, H., Iterated logarithm inequalities, Proc. Natl. Acad. Sci. USA, 57, 5, 1188-1192 (1967) · Zbl 0167.46902 [11] Davidson, R.; MacKinnon, J., Bootstrap tests: How many bootstraps?, Econom. Rev., 19, 1, 55-68 (2000) · Zbl 0949.62030 [12] Ding, D.; Gandy, A.; Hahn, G., A simple method for implementing Monte Carlo tests, 1-17 (2018), arXiv:1611.01675 [13] Fay, M.; Follmann, D., Designing Monte Carlo implementations of permutation or bootstrap hypothesis tests, Am. Stat., 56, 1, 63-70 (2002) · Zbl 1182.62094 [14] Fay, M.; Kim, H.-J.; Hachey, M., On using truncated sequential probability ratio test boundaries for Monte Carlo implementation of hypothesis tests, J. Comput. Graph. Stat., 16, 4, 946-967 (2007) [15] Gandy, A., Sequential implementation of Monte Carlo tests with uniformly bounded resampling risk, J. Amer. Statist. Assoc., 104, 488, 1504-1511 (2009) · Zbl 1205.65016 [16] Gandy, A.; Hahn, G., Mmctest - A safe algorithm for implementing multiple Monte Carlo tests, Scand. J. Stat., 41, 4, 1083-1101 (2014) · Zbl 1305.62270 [17] Gandy, A.; Hahn, G., A framework for Monte Carlo based multiple testing, Scand. J. Stat., 43, 4, 1046-1063 (2016) · Zbl 1373.62389 [18] Gandy, A.; Hahn, G., Quickmmctest: quick multiple Monte Carlo testing, Stat. Comput., 27, 3, 823-832 (2017) · Zbl 06737699 [19] Guo, W.; Peddada, S., Adaptive choice of the number of bootstrap samples in large scale multiple testing, Stat. Appl. Genet. Mol. Biol., 7, 1, 1-16 (2008) [20] Hahn, G., Optimal allocation of Monte Carlo simulations to multiple hypothesis tests, Stat. Comput., 1-16 (2019) [21] Hochberg, Y., A sharper Bonferroni procedure for multiple tests of significance, Biometrika, 75, 4, 800-802 (1988) · Zbl 0661.62067 [22] Hoeffding, W., Probability inequalities for sums of bounded random variables, J. Amer. Statist. Assoc., 58, 301, 13-30 (1963) · Zbl 0127.10602 [23] Holm, S., A simple sequentially rejective multiple test procedure, Scand. J. Stat., 6, 2, 65-70 (1979) · Zbl 0402.62058 [24] Kim, H.-J., Bounding the resampling risk for sequential Monte Carlo implementation of hypothesis tests, J. Stat. Plan. Inference, 140, 7, 1834-1843 (2010) · Zbl 1190.62142 [25] Lai, T., On confidence sequences, Ann. Statist., 4, 2, 265-280 (1976) · Zbl 0346.62035 [26] Lin, D., An efficient Monte Carlo approach to assessing statistical significance in genomic studies, Bioinformatics, 21, 6, 781-787 (2005) [27] Robbins, H., Statistical methods related to the law of the iterated logarithm, Ann. Math. Stat., 41, 5, 1397-1409 (1970) · Zbl 0239.62025 [28] Rom, D., A sequentially rejective test procedure based on a modified Bonferroni inequality, Biometrika, 77, 3, 663-665 (1990) [29] Sandve, G.; Ferkingstad, E.; Nygård, S., Sequential Monte Carlo multiple testing, Bioinformatics, 27, 23, 3235-3241 (2011) [30] Shaffer, J., Modified sequentially rejective multiple test procedures, J. Amer. Statist. Assoc., 81, 395, 826-831 (1986) · Zbl 0603.62087 [31] Sidak, Z., Rectangular confidence regions for the means of multivariate normal distributions, J. Amer. Statist. Assoc., 62, 318, 626-633 (1967) · Zbl 0158.17705 [32] Silva, I.; Assunção, R., Optimal generalized truncated sequential Monte Carlo test, J. Multivariate Anal., 121, 33-49 (2013) · Zbl 1315.62070 [33] Silva, I.; Assunção, R., Truncated sequential Monte Carlo test with exact power, Braz. J. Probab. Stat., 32, 2, 215-238 (2018) · Zbl 1395.62250 [34] Silva, I.; Assunção, R.; Costa, M., Power of the sequential Monte Carlo test, Sequential Anal., 28, 2, 163-174 (2009) · Zbl 1162.62081 [35] Simes, R., An improved Bonferroni procedure for multiple tests of significance, Biometrika, 73, 3, 751-754 (1986) · Zbl 0613.62067 [36] van Wieringen, W.; van de Wiel, M.; van der Vaart, A., A test for partial differential expression, J. Amer. Statist. Assoc., 103, 483, 1039-1049 (2008) · Zbl 1205.62189 [37] Wald, A., Sequential tests of statistical hypotheses, Ann. Math. Stat., 16, 2, 117-186 (1945) · Zbl 0060.30207
This reference list is based on information provided by the publisher or from digital mathematics libraries. Its items are heuristically matched to zbMATH identifiers and may contain data conversion errors. It attempts to reflect the references listed in the original paper as accurately as possible without claiming the completeness or perfect precision of the matching.