Control of generalized error rates in multiple testing. (English) Zbl 1127.62063

Romano, Joseph P.; Wolf, Michael

Ann. Stat. 35, No. 4, 1378-1408 (2007).

Summary: Consider the problem of testing \(s\) hypotheses simultaneously. The usual approach restricts attention to procedures that control the probability of even one false rejection, the familywise error rate (FWER). If \(s\) is large, one might be willing to tolerate more than one false rejection, thereby increasing the ability of the procedure to correctly reject false null hypotheses. One possibility is to replace control of the FWER by control of the probability of \(k\) or more false rejections, which is called the \(k\)-FWER. We derive both single-step and step-down procedures that control the \(k\)-FWER in finite samples or asymptotically, depending on the situation. We also consider the false discovery proportion (FDP) defined as the number of false rejections divided by the total number of rejections (and defined to be 0 if there are no rejections).
The false discovery rate proposed by Y. Benjamini and Y. Hochberg [J. R. Stat. Soc., Ser. B 57, No. 1, 289–300 (1995; Zbl 0809.62014)] controls \(E\)(FDP). Here, the goal is to construct methods which satisfy, for a given \(\gamma\) and \(\alpha\), \(P\{\text{FDP}>\gamma\}\leq\alpha\), at least asymptotically. In contrast to the proposals of E. L. Lehmann and J. P. Romano [Ann. Stat. 33, No. 3, 1138–1154 (2005; Zbl 1072.62060)], we construct methods that implicitly take into account the dependence structure of the individual test statistics in order to further increase the ability to detect false null hypotheses. This feature is also shared by related work of M. J. van der Laan, S. Dudoit and K. S. Pollard [Stat. Appl. Genet. Mol. Biol. 3, No. 1, Article 15, electronic only (2004; Zbl 1166.62379)], but our methodology is quite different. Like the work of K. S. Pollard and M. J. van der Laan [Int. Multi-Conf. Computer Sci. Eng., METMBS’03 Conference, 3–9 (2003)] and S. Dudoit, M. J. van der Laan and K. S. Pollard [Stat. Appl. Genet. Mol. Biol. 3, No. 1, Article 13, electronic only (2004; Zbl 1166.62338)], we employ resampling methods to achieve our goals. Some simulations compare finite sample performance to currently available methods.

MSC:

62J15	Paired and multiple comparisons; multiple testing
62G09	Nonparametric statistical resampling methods
62G10	Nonparametric hypothesis testing
62G20	Asymptotic properties of nonparametric inference

Keywords:

bootstrap; false discovery proportion; false discovery rate; generalized familywise error rate; step-down procedure

Citations:

Zbl 0809.62014; Zbl 1072.62060; Zbl 1166.62379; Zbl 1166.62338

Software:

bootlib

References:

[1]	Benjamini, Y. and Hochberg, Y. (1995). Controlling the false discovery rate: A practical and powerful approach to multiple testing. J. Roy. Statist. Soc. Ser. B 57 289–300. JSTOR: · Zbl 0809.62014
[2]	Benjamini, Y. and Yekutieli, D. (2001). The control of the false discovery rate in multiple testing under dependency. Ann. Statist. 29 1165–1188. · Zbl 1041.62061 · doi:10.1214/aos/1013699998
[3]	Beran, R. (1984). Bootstrap methods in statistics. Jahresber. Deutsch. Math.-Verein. 86 14–30. · Zbl 0547.62023
[4]	Beran, R. (1986). Simulated power functions. Ann. Statist. 14 151–173. · Zbl 0622.62051 · doi:10.1214/aos/1176349847
[5]	Beran, R. (1988). Balanced simultaneous confidence sets. J. Amer. Statist. Assoc. 83 679–686. JSTOR: · Zbl 0662.62031 · doi:10.2307/2289291
[6]	Beran, R. (1988). Prepivoting test statistics: A bootstrap view of asymptotic refinements. J. Amer. Statist. Assoc. 83 687–697. JSTOR: · Zbl 0662.62024 · doi:10.2307/2289292
[7]	Davison, A. C. and Hinkley, D. V. (1997). Bootstrap Methods and Their Application . Cambridge Univ. Press. · Zbl 0886.62001
[8]	Dudoit, S., van der Laan, M. J. and Birkner, M. D. (2004). Multiple testing procedures for controlling tail probability error rates. Working Paper 166, Div. Biostatics, Univ. California, Berkeley. Available at www.bepress.com/ucbbiostat/paper166.
[9]	Dudoit, S., van der Laan, M. J. and Pollard, K. S. (2004). Multiple testing. I. Single-step procedures for the control of general type I error rates. Stat. Appl. Genet. Mol. Biol. 3 article 13. Available at www.bepress.com/sagmb/vol3/iss1/art13. · Zbl 1166.62338 · doi:10.2202/1544-6115.1040
[10]	Genovese, C. R. and Wasserman, L. (2004). A stochastic process approach to false discovery control. Ann. Statist. 32 1035–1061. · Zbl 1092.62065 · doi:10.1214/009053604000000283
[11]	Hall, P. (1992). The Bootstrap and Edgeworth Expansion . Springer, New York. · Zbl 0744.62026
[12]	Hall, P. and Wilson, S. (1991). Two guidelines for bootstrap hypothesis testing. Biometrics 47 757–762. JSTOR: · doi:10.2307/2532163
[13]	Holm, S. (1979). A simple sequentially rejective multiple test procedure. Scand. J. Statist. 6 65–70. · Zbl 0402.62058
[14]	Hommel, G. and Hoffman, T. (1988). Controlled uncertainty. In Multiple Hypothesis Testing (P. Bauer, G. Hommel and E. Sonnemann, eds.) 154–161. Springer, Heidelberg.
[15]	Korn, E. L., Troendle, J. F., McShane, L. M. and Simon, R. (2004). Controlling the number of false discoveries: Application to high-dimensional genomic data. J. Statist. Plann. Inference 124 379–398. · Zbl 1074.62070 · doi:10.1016/S0378-3758(03)00211-8
[16]	Lahiri, S. N. (2003). Resampling Methods for Dependent Data . Springer, New York. · Zbl 1028.62002
[17]	Lehmann, E. L. and Romano, J. P. (2005). Generalizations of the familywise error rate. Ann. Statist. 33 1138–1154. · Zbl 1072.62060 · doi:10.1214/009053605000000084
[18]	Lehmann, E. L. and Romano, J. P. (2005). Testing Statistical Hypotheses , 3rd ed. Springer, New York. · Zbl 1076.62018 · doi:10.1007/0-387-27605-X
[19]	Lehmann, E. L., Romano, J. P. and Shaffer, J. P. (2005). On optimality of stepdown and stepup multiple test procedures. Ann. Statist. 33 1084–1108. · Zbl 1073.62063 · doi:10.1214/009053605000000066
[20]	Perone Pacifico, M., Genovese, C. R., Verdinelli, I. and Wasserman, L. (2004). False discovery control for random fields. J. Amer. Statist. Assoc. 99 1002–1014. · Zbl 1055.62105 · doi:10.1198/0162145000001655
[21]	Politis, D. N., Romano, J. P. and Wolf, M. (1999). Subsampling . Springer, New York. · Zbl 0931.62035
[22]	Pollard, K. S. and van der Laan, M. J. (2003). Multiple testing for gene expression data: An investigation of null distributions with consequences for the permutation test. In Proc. 2003 International Multi-Conference in Computer Science and Engineering , METMBS ’ 03 Conference 3–9.
[23]	Rogers, J. and Hsu, J. (2001). Multiple comparisons of biodiversity. Biom. J. 43 617–625. · Zbl 0999.62091 · doi:10.1002/1521-4036(200109)43:5<617::AID-BIMJ617>3.0.CO;2-J
[24]	Romano, J. P. (1988). A bootstrap revival of some nonparametric distance tests. J. Amer. Statist. Assoc. 83 698–708. JSTOR: · Zbl 0658.62059 · doi:10.2307/2289293
[25]	Romano, J. P. and Shaikh, A. M. (2006). On stepdown control of the false discovery proportion. In Optimality : The Second Erich L. Lehmann Symposium (J. Rojo, ed.) 33–50. IMS, Beachwood, OH. · Zbl 1268.62078 · doi:10.1214/074921706000000383
[26]	Romano, J. P. and Shaikh, A. M. (2006). Stepup procedures for control of generalizations of the familywise error rate. Ann. Statist. 34 1850–1873. · Zbl 1246.62172 · doi:10.1214/009053606000000461
[27]	Romano, J. P. and Wolf, M. (2005). Control of generalized error rates in multiple testing. Working Paper 245, IEW, Univ. Zurich. Available at www.iew.unizh.ch/wp/index.php.
[28]	Romano, J. P. and Wolf, M. (2005). Exact and approximate step-down methods for multiple hypothesis testing. J. Amer. Statist. Assoc. 100 94–108. · Zbl 1117.62416 · doi:10.1198/016214504000000539
[29]	Romano, J. P. and Wolf, M. (2005). Stepwise multiple testing as formalized data snooping. Econometrica 73 1237–1282. JSTOR: · Zbl 1153.62310 · doi:10.1111/j.1468-0262.2005.00615.x
[30]	Sarkar, S. K. (2002). Some results on false discovery rate in stepwise multiple testing procedures. Ann. Statist. 30 239–257. · Zbl 1101.62349 · doi:10.1214/aos/1015362192
[31]	Shao, J. and Tu, D. (1995). The Jackknife and Bootstrap . Springer, New York. · Zbl 0947.62501
[32]	Tu, W. and Zhou, X. (2000). Pairwise comparisons of the means of skewed data. J. Statist. Plann. Inference 88 59–74. · Zbl 0964.62062 · doi:10.1016/S0378-3758(99)00206-2
[33]	van der Laan, M. J., Birkner, M. D. and Hubbard, A. E. (2005). Empirical Bayes and resampling based multiple testing procedure controlling tail probability of the proportion of false positives. Stat. Appl. Genet. Mol. Biol. 4 article 29. Available at www.bepress.com/sagmb/vol4/iss1/art29. · Zbl 1108.62303 · doi:10.2202/1544-6115.1143
[34]	van der Laan, M. J., Dudoit, S. and Pollard, K. S. (2004). Augmentation procedures for control of the generalized family-wise error rate and tail probabilities for the proportion of false positives. Stat. Appl. Genet. Mol. Biol. 3 article 15. Available at www.bepress.com/sagmb/vol3/iss1/art15. · Zbl 1166.62379
[35]	Westfall, P. H. and Young, S. S. (1993). Resampling-Based Multiple Testing : Examples and Methods for P-Value Adjustment . Wiley, New York. · Zbl 0850.62368