The control of the false discovery rate in multiple testing under dependency. (English) Zbl 1041.62061

Summary: Y. Benjamini and Y. Hochberg [J. R. Stat. Soc., Ser. B 57, 289–300 (1995; Zbl 0809.62014)] suggest that the false discovery rate may be the appropriate error rate to control in many applied multiple testing problems. A simple procedure was given there as an FDR controlling procedure for independent test statistics and was shown to be much more powerful than comparable procedures which control the traditional familywise error rate. We prove that this same procedure also controls the false discovery rate when the test statistics have positive regression dependency on each of the test statistics corresponding to the true null hypotheses.
This condition for positive dependency is general enough to cover many problems of practical interest, including the comparisons of many treatments with a single control, multivariate normal test statistics with positive correlation matrix and multivariate \(t\). Furthermore, the test statistics may be discrete, and the tested hypotheses composite without posing special difficulties. For all other forms of dependency, a simple conservative modification of the procedure controls the false discovery rate. Thus the range of problems for which a procedure with proven FDR control can be offered is greatly increased.


62J15 Paired and multiple comparisons; multiple testing
62H20 Measures of association (correlation, canonical correlation, etc.)
47N30 Applications of operator theory in probability theory and statistics


Zbl 0809.62014
Full Text: DOI


[1] Abramovich, F. and Benjamini, Y. (1996). Adaptive thresholding of wavelet coefficients. Comput. Statist. Data Anal. 22 351-361.
[2] Barinaga, M. (1994). From fruit flies, rats, mice: evidence of genetic influence. Science 264 1690-1693.
[3] Benjamini, Y. and Hochberg, Y. (1995). Controlling the false discovery rate: a practical and powerful approach to multiple testing. J. Roy. Statist. Soc. Ser. B 57 289-300. JSTOR: · Zbl 0809.62014
[4] Benjamini, Y. and Hochberg, Y. (1997). Multiple hypotheses testing with weights. Scand. J. Statist. 24 407-418. · Zbl 1090.62548 · doi:10.1111/1467-9469.00072
[5] Benjamini, Y. and Hochberg, Y. (2000). The adaptive control of the false discovery rate in multiple hypotheses testing. J. Behav. Educ. Statist. 25 60-83.
[6] Benjamini, Y., Hochberg, Y. and Kling, Y. (1993). False discovery rate control in pairwise comparisons. Working Paper 93-2, Dept. Statistics and O.R., Tel Aviv Univ.
[7] Benjamini, Y., Hochberg, Y. and Kling, Y. (1997). False discovery rate control in multiple hypotheses testing using dependent test statistics. Research Paper 97-1, Dept. Statistics and O.R., Tel Aviv Univ.
[8] Benjamini, Y. and Wei, L. (1999). A step-down multiple hypotheses testing procedure that controls the false discovery rate under independence. J. Statist. Plann. Inference 82 163-170. · Zbl 1063.62558 · doi:10.1016/S0378-3758(99)00040-3
[9] Chang, C. K., Rom, D. M. and Sarkar, S. K. (1996). A modified Bonferroni procedure for repeated significance testing. Technical Report 96-01, Temple Univ.
[10] Eaton, M. L. (1986). Lectures on topics in probability inequalities. CWI Tract 35. · Zbl 0622.60024
[11] Hochberg, Y. (1988). A sharper Bonferroni procedure for multiple tests of significance. Biometrika 75 800-803. JSTOR: · Zbl 0661.62067 · doi:10.1093/biomet/75.4.800
[12] Hochberg, Y. and Hommel, G. (1998). Step-up multiple testing procedures. Encyclopedia Statist. Sci. (Supp.) 2.
[13] Hochberg, Y. and Rom, D. (1995). Extensions of multiple testing procedures based on Simes’ test. J. Statist. Plann. Inference 48 141-152. · Zbl 0851.62054 · doi:10.1016/0378-3758(95)00005-T
[14] Hochberg, Y. and Tamhane, A. (1987). Multiple Comparison Procedures. Wiley, New York. · Zbl 0731.62125
[15] Holland, P. W. and Rosenbaum, P. R. (1986). Conditional association and unidimensionality in monotone latent variable models. Ann. Statist. 14 1523-1543. · Zbl 0625.62102 · doi:10.1214/aos/1176350174
[16] Holm, S. (1979). A simple sequentially rejective multiple test procedure. Scand. J. Statist 6 65-70. · Zbl 0402.62058
[17] Hommel, G. (1988). A stage-wise rejective multiple test procedure based on a modified Bonferroni test. Biometrika 75 383-386. · Zbl 0639.62025 · doi:10.1093/biomet/75.2.383
[18] Hsu, J. (1996). Multiple Comparisons Procedures. Chapman and Hall, London. · Zbl 0898.62090
[19] Karlin, S. and Rinott, Y. (1980). Classes of orderings of measures and related correlation inequalities I. Multivariate totally positive distributions. J. Multivariate Statist. 10 467-498. · Zbl 0469.60006 · doi:10.1016/0047-259X(80)90065-2
[20] Karlin, S. and Rinott, Y. (1981). Total positivity properties of absolute value multinormal variable with applications to confidence interval estimates and related probabilistic inequalities. Ann. Statist. 9 1035-1049. · Zbl 0477.62035 · doi:10.1214/aos/1176345583
[21] Lander E. S. and Botstein D. (1989). Mapping Mendelian factors underlying quantitative traits using RFLP linkage maps. Genetics 121 185-190.
[22] Lander, E. S. and Kruglyak L. (1995). Genetic dissection of complex traits: guidelines for interpreting and reporting linkage results. Nature Genetics 11 241-247.
[23] Lehmann, E. L. (1966). Some concepts of dependence. Ann. Math. Statist. 37 1137-1153. · Zbl 0146.40601 · doi:10.1214/aoms/1177699260
[24] Needleman, H., Gunnoe, C., Leviton, A., Reed, R., Presie, H., Maher, C. and Barret, P. (1979). Deficits in psychologic and classroom performance of children with elevated dentine lead levels. New England J. Medicine 300 689-695.
[25] Paterson, A. H. G., Powles, T. J., Kanis, J. A., McCloskey, E., Hanson, J. and Ashley, S. (1993). Double-blind controlled trial of oral clodronate in patients with bone metastases from breastcancer. J. Clinical Oncology 1 59-65.
[26] Rosenbaum, P. R. (1984). Testing the conditional independence and monotonicity assumptions of item response theory. Psychometrika 49 425-436. · Zbl 0569.62097 · doi:10.1007/BF02306030
[27] Sarkar, T. K. (1969). Some lower bounds of reliability. Technical Report, 124, Dept. Operation Research and Statistics, Stanford Univ.
[28] Sarkar, S. K. (1998). Some probability inequalities for ordered MTP2 random variables: a proof of Simes’ conjecture. Ann. Statist. 26 494-504. · Zbl 0929.62065 · doi:10.1214/aos/1028144846
[29] Sarkar, S. K. and Chang, C. K. (1997). The Simes method for multiple hypotheses testing with positively dependent test statistics. J. Amer. Statist. Assoc. 92 1601-1608. JSTOR: · Zbl 0912.62079 · doi:10.2307/2965431
[30] Seeger, (1968). A note on a method for the analysis of significances en mass. Technometrics 10 586-593. Sen, P. K. (1999a). Some remarks on Simes-type multiple tests of significance. J. Statist. Plann. Inference, 82 139-145. Sen, P. K. (1999b). Multiple comparisons in interim analysis. J. Statist. Plann. Inference 82 5-23.
[31] Shaffer, J. P. (1995). Multiple hypotheses-testing. Ann. Rev. Psychol. 46 561-584.
[32] Simes, R. J. (1986). An improved Bonferroni procedure for multiple tests of significance. Biometrika 73 751-754. JSTOR: · Zbl 0613.62067 · doi:10.1093/biomet/73.3.751
[33] Steel, R. G. D. and Torrie, J. H. (1980). Principles and Procedures of Statistics: A Biometrical Approach, 2nd ed. McGraw-Hill, New York. · Zbl 0544.62001
[34] Tamhane, A. C. (1996). Multiple comparisons. In Handbook of Statistics (S. Ghosh and C. R. Rao, eds.) 13 587-629. North-Holland, Amsterdam. · Zbl 0911.62060
[35] Tamhane, A. C. and Dunnett, C. W. (1999). Stepwise multiple test procedures with biometric applications. J. Statist. Plann. Inference 82 55-68. · Zbl 0979.62049 · doi:10.1016/S0378-3758(99)00031-2
[36] Troendle, J. (2000). Stepwise normal theory tests procedures controlling the false discovery rate. J. Statist. Plann. Inference 84 139-158. · Zbl 1131.62310 · doi:10.1016/S0378-3758(99)00145-7
[37] Wassmer, G., Reitmer, P., Kieser, M. and Lehmacher, W. (1999). Procedures for testing multiple endpoints in clinical trials: an overview. J. Statist. Plann. Inference 82 69-81. · Zbl 0979.62090 · doi:10.1016/S0378-3758(99)00032-4
[38] Weller, J. I., Song, J. Z., Heyen, D. W., Lewin, H. A. and Ron, M. (1998). A new approach to the problem of multiple comparison in the genetic dissection of complex traits. Genetics 150 1699-1706.
[39] Westfall, P. H. and Young, S. S. (1993). Resampling Based Multiple Testing, Wiley, New York. · Zbl 0850.62368
[40] Williams, V. S. L., Jones, L. V. and Tukey, J. W. (1999). Controlling error in multiple comparisons, with special attention to the National Assessment of Educational Progress. J. Behav. Educ. Statist. 24 42-69.
[41] Yekutieli, D. and Benjamini, Y. (1999). A resampling based false discovery rate controlling multiple test procedure. J. Statist. Plann. Inference 82 171-196. · Zbl 1063.62563 · doi:10.1016/S0378-3758(99)00041-5
This reference list is based on information provided by the publisher or from digital mathematics libraries. Its items are heuristically matched to zbMATH identifiers and may contain data conversion errors. In some cases that data have been complemented/enhanced by data from zbMATH Open. This attempts to reflect the references listed in the original paper as accurately as possible without claiming completeness or a perfect matching.