Multiple hypotheses testing and expected number of type I errors. (English) Zbl 1012.62020

Finner, H.; Roters, M.

Ann. Stat. 30, No. 1, 220-238 (2002).

Summary: The performance of multiple test procedures with respect to error control is an old issue. Assuming that all hypotheses are true we investigate the behavior of the expected number of type I errors (ENE) as a characteristic of certain multiple tests controlling the familywise error rate (FWER) or the false discovery rate (FDR) at a prespecified level. We derive explicit formulas for the distribution of the number of false rejections as well as for the ENE for single-step, step-down and step-up procedures based on independent \(p\)-values. Moreover, we determine the corresponding asymptotic distributions of the number of false rejections as well as explicit formulae for the ENE if the number of hypotheses tends to infinity.
In case of FWER-control we mostly obtain Poisson distributions and in one case a geometric distribution as limiting distributions; in case of FDR control we obtain limiting distributions which are apparently not named in the literature. Surprisingly, the ENE is bounded by a small number regardless of the number of hypotheses under consideration. Finally, it turns out that in case of dependent test statistics the ENE behaves completely differently compared to the case of independent test statistics.

MSC:

62F03	Parametric hypothesis testing
62J15	Paired and multiple comparisons; multiple testing
62F05	Asymptotic properties of parametric tests

References:

[1]	BENJAMINI, Y. and HOCHBERG, Y. (1995). Controlling the false discovery rate: a practical and powerful approach to multiple testing. J. Roy. Statist. Soc. Ser. B 57 289-300. JSTOR: · Zbl 0809.62014
[2]	BENJAMINI, Y. and YEKUTIELI, D. (2001). The control of the false discovery rate in multiple testing under dependency. Ann. Statist. 29 1165-1188. · Zbl 1041.62061 · doi:10.1214/aos/1013699998
[3]	CSÖRG O, M. and HORVÁTH, L. (1993). Weighted Approximations in Probability and Statistics. Wiley, Chichester. · Zbl 0770.60038
[4]	DALAL, S. R. and MALLOWS, C. L. (1992). Buying with exact confidence. Ann. Appl. Probab. 2 752-765. · Zbl 0753.62051 · doi:10.1214/aoap/1177005658
[5]	DEMPSTER, A. P. (1959). Generalized D+ n statistics. Ann. Math. Statist. 30 593-597. · Zbl 0122.37003 · doi:10.1214/aoms/1177706275
[6]	EKLUND G. and SEEGER, P. (1965). Massignifikansanalys. Statistisk Tidskrift Stockholm, 3rd Ser. 4 355-365.
[7]	FELLER, W. (1971). An Introduction to Probability Theory and Its Applications 2, 2nd ed. Wiley, New York. · Zbl 0219.60003
[8]	FINNER, H., HAYTER, A. J. and ROTERS, M. (1993). On the joint distribution function of order statistics with reference to step-up multiple test procedures. Technical report 93-19, 1-11, Univ. Trier.
[9]	FINNER, H. and ROTERS, M. (1994). On the limit behaviour of the joint distribution function of order statistics. Ann. Inst. Statist. Math. 46 343-349. · Zbl 0802.62018 · doi:10.1007/BF01720590
[10]	FINNER, H. and ROTERS, M. (1998). Asymptotic comparison of step-down and step-up multiple test procedures based on exchangeable test statistics. Ann. Statist. 26 505-524. · Zbl 0934.62073 · doi:10.1214/aos/1028144847
[11]	FINNER, H. and ROTERS, M. (1999). Asymptotic comparison of the critical values of step-down and step-up multiple comparison procedures. J. Statist. Plann. Inference 79 11-30. · Zbl 0951.62060 · doi:10.1016/S0378-3758(98)00233-X
[12]	FINNER, H. and ROTERS, M. (2000). On the critical value behaviour of multiple decision procedures. Scand. J. Statist. 27 563-573. · Zbl 0976.62006 · doi:10.1111/1467-9469.00207
[13]	FINNER, H. and ROTERS, M. (2001a). Asymptotic sharpness of product-type inequalities for maxima of random variables with applications in multiple comparisons. J. Statist. Plann. Inference 98 127-144. · Zbl 0981.60009 · doi:10.1016/S0378-3758(00)00304-9
[14]	FINNER, H. and ROTERS, M. (2001b). On the false discovery rate and expected type I errors. Biom. J. 43 985-1005. · Zbl 0989.62061 · doi:10.1002/1521-4036(200112)43:8<985::AID-BIMJ985>3.0.CO;2-4
[15]	HENRICI, P. (1974). Applied and Computational Complex Analysis 1. Wiley, New York. · Zbl 0313.30001
[16]	HOCHBERG, Y. (1988). A sharper Bonferroni procedure for multiple tests of significance. Biometrika 75 800-802. JSTOR: · Zbl 0661.62067 · doi:10.1093/biomet/75.4.800
[17]	HOCHBERG, Y. and TAMHANE, A. C. (1987). Multiple Comparisons Procedures. Wiley, New York.
[18]	HOMMEL, G. (1988). A stagewise rejective multiple test procedure based on a modified Bonferroni test. Biometrika 75 383-386. · Zbl 0639.62025 · doi:10.1093/biomet/75.2.383
[19]	KARLIN, S. and TAYLOR, H. M. (1981). A Second Course in Stochastic Processes. Academic Press, Boston. · Zbl 0469.60001
[20]	MILLER, R. G., JR. (1981). Simultaneous Statistical Inference, 2nd ed. Springer, New York.
[21]	RÉNYI, A. (1973). On a group of problems in the theory of ordered samples. Selected Translations in Math. Statist. Probab. 13 289-298. · Zbl 0269.62011
[22]	ROM, D. M. (1990). A sequentially rejective test procedure based on a modified Bonferroni inequality. Biometrika 77 663-665. JSTOR: · doi:10.1093/biomet/77.3.663
[23]	SARKAR, S. K. (2002). Some results on false discovery rate in stepwise multiple testing procedures. Ann. Statist. 30 240-258. · Zbl 1101.62349 · doi:10.1214/aos/1015362192
[24]	SEEGER, P. (1966). Variance Analysis of Complete Designs. Some Practical Aspects. Almqvist and Wiksell, Uppsala.
[25]	SHORACK, G. R. and WELLNER, J. A. (1986). Empirical Processes with Applications to Statistics. Wiley, New York. · Zbl 1170.62365
[26]	SIMES, R. J. (1986). An improved Bonferroni procedure for multiple tests of significance. Biometrika 73 751-754. JSTOR: · Zbl 0613.62067 · doi:10.1093/biomet/73.3.751
[27]	SPJØTVOLL, E. (1972). On the optimality of some multiple comparison procedures. Ann. Math. Statist. 43 398-411. · Zbl 0241.62045 · doi:10.1214/aoms/1177692621