×

zbMATH — the first resource for mathematics

Detecting and adjusting for small-study effects in meta-analysis. (English) Zbl 1209.62325
Summary: Publication bias and related types of small-study effects threaten the validity of systematic reviews. The existence of small-study effects has been demonstrated in empirical studies. Small-study effects are graphically diagnosed by inspection of the funnel plot. Though observed funnel plot asymmetry cannot be easily linked to a specific reason, tests based on funnel plot asymmetry have been proposed. Beyond a vast range of funnel plot tests, there exist several methods for adjusting treatment effect estimates for these biases. We consider the trim-and-fill method, the Copas selection model, and more recent regression-based approaches. The methods are exemplified using a meta-analysis from the literature and compared in a simulation study, based on binary response data. They are also applied to a large set of meta-analyses. Some fundamental differences between the approaches are discussed. An assumption common to the trim-and-fill method and the Copas selection model is that the small-study effect is caused by selection. The trim-and-fill method corresponds to an unknown implicit model generated by the symmetry assumption, whereas the Copas selection model is a parametric statistical model. However, it requires a sensitivity analysis. Regression-based approaches are easier to implement and not based on a specific selection model. Both simulations and applications suggest that in the presence of strong selection both the trim-and-fill method and the Copas selection model may not fully eliminate bias, while regression-based approaches seem to be a promising alternative.
MSC:
62P10 Applications of statistics to biology and medical sciences; meta analysis
65C60 Computational problems in statistics (MSC2010)
Software:
copas; netmeta; snowfall
PDF BibTeX XML Cite
Full Text: DOI
References:
[1] Altman, Measurement in medicine: the analysis of method comparison studies, The Statistician 32 pp 307– (1983) · doi:10.2307/2987937
[2] Begg, Publication bias: a problem in interpreting medical data, Journal of the Royal Statistical Society, Series A 151 pp 419– (1988) · Zbl 04542256 · doi:10.2307/2982993
[3] Begg, Operating characteristics of a rank correlation test for publication bias, Biometrics 50 pp 1088– (1994) · Zbl 0825.62457 · doi:10.2307/2533446
[4] Berlin, An assessment of publication bias using a sample of published clinical trials, Journal of the American Statistical Association 84 pp 381– (1989) · doi:10.2307/2289921
[5] Borm, The evidence provided by a single trial is less reliable than its statistical analysis suggests, Journal of Clinical Epidemiology 62 pp 711– (2009) · doi:10.1016/j.jclinepi.2008.09.013
[6] Carpenter, opas: an R package for fitting the Copas selection model, The R Journal 1 pp 31– (2009a)
[7] Carpenter, Empirical evaluation showed that the Copas selection model provided a useful summary in 80% of meta-analyses, Journal of Clinical Epidemiology 62 pp 624– (2009b) · doi:10.1016/j.jclinepi.2008.12.002
[8] Carpenter, Assessing the sensitivity of meta-analysis to selection bias: a multiple imputation approach, Biometrics (2010) · Zbl 1226.62106 · doi:10.1111/j.1541-0420.2010.01498.x
[9] Chan, Empirical evidence for selective reporting of outcomes in randomized trials. Comparison of protocols to published articles, Journal of the American Medical Association 291 pp 2457– (2004a) · doi:10.1001/jama.291.20.2457
[10] Chan, Outcome reporting bias in randomized trials funded by the Canadian Institutes of Health Research, Canadian Medical Association Journal 171 pp 735– (2004b) · doi:10.1503/cmaj.1041086
[11] Copas, What works?: selectivity models and meta-analysis, Journal of the Royal Statistical Society, Series A 162 pp 95– (1999) · Zbl 04549849 · doi:10.1111/1467-985X.00123
[12] Copas, Meta-analysis, funnel plots and sensitivity analysis, Biostatistics 1 pp 247– (2000) · Zbl 0958.62102 · doi:10.1093/biostatistics/1.3.247
[13] Copas, A robust p-value for treatment effect in meta-analysis with publication bias, Statistics in Medicine 27 pp 4267– (2008) · doi:10.1002/sim.3284
[14] Copas, A sensitivity analysis for publication bias in systematic reviews, Statistical Methods in Medical Research 10 pp 251– (2001) · Zbl 1121.62593 · doi:10.1191/096228001678227776
[15] DerSimonian, Meta-analysis in clinical trials, Controlled Clinical Trials 7 pp 177– (1986) · doi:10.1016/0197-2456(86)90046-2
[16] Dickersin, The existence of publication bias and risk factors for its occurrence, Journal of the American Medical Association 263 pp 1385– (1990) · doi:10.1001/jama.263.10.1385
[17] Duval, A nonparametric ”trim and fill” method of accounting for publication bias in meta-analysis, Journal of the American Statistical Association 95 pp 89– (2000a) · Zbl 1011.62112 · doi:10.2307/2669529
[18] Duval, Trim and fill: a simple funnel-plot-based method of testing and adjusting for publication bias in meta-analysis, Biometrics 56 pp 455– (2000b) · Zbl 1060.62600 · doi:10.1111/j.0006-341X.2000.00455.x
[19] Easterbrook, Publication bias in clinical research, The Lancet 337 pp 867– (1991) · doi:10.1016/0140-6736(91)90201-Y
[20] Egger, Bias in meta-analysis detected by a simple, graphical test, British Medical Journal 315 pp 629– (1997a) · doi:10.1136/bmj.315.7109.629
[21] Egger, Language bias in randomised controlled trials published in English and German, The Lancet 350 pp 326– (1997b) · doi:10.1016/S0140-6736(97)02419-7
[22] Galandi, The demise of the randomised controlled trial: bibliometric study of the German-language health care literature, 1948 to 2004, BMC Medical Research Methodology 6 pp 30– (2006) · doi:10.1186/1471-2288-6-30
[23] Grégoire, Selecting the language of the publications included in a meta-analysis: is there a tower of babel bias?, Journal of Clinical Epidemiology 48 pp 159– (1995) · doi:10.1016/0895-4356(94)00098-B
[24] Harbord, A modified test for small-study effects in meta-analyses of controlled trials with binary endpoints, Statistics in Medicine 25 pp 3443– (2006) · doi:10.1002/sim.2380
[25] Higgins , J. P. Green , S. 2009 http://www.cochrane-handlook.org
[26] Higgins, A re-evaluation of random-effects meta-analysis, Journal of the Royal Statistical Society 172 pp 137– (2009) · Zbl 05622745 · doi:10.1111/j.1467-985X.2008.00552.x
[27] Ioannidis, Interpretation of tests of heterogeneity and bias in meta-analysis, Journal of Evaluation in Clinical Practice 14 pp 951– (2008) · doi:10.1111/j.1365-2753.2008.00986.x
[28] Ioannidis, The appropriateness of asymmetry tests for publication bias in meta-analyses: a large survey, Canadian Medical Association Journal 176 pp 1091– (2007) · doi:10.1503/cmaj.060410
[29] Jüni , P. 2006
[30] Kjaergard, Citation bias of hepato-biliary randomized clinical trials, Journal of Clinical Epidemiology 55 pp 407– (2002) · doi:10.1016/S0895-4356(01)00513-3
[31] Knapp, Assessing the amount of heterogeneity in random-effects meta-analysis, Biometrical Journal 48 pp 271– (2006) · doi:10.1002/bimj.200510175
[32] Knaus, Easier parallel computing in R with snowfall and sfCluster, The R Journal 1 pp 54– (2009)
[33] Lau, The case of the misleading funnel plot, British Medical Journal 333 pp 597– (2006) · doi:10.1136/bmj.333.7568.597
[34] Light, Summing Up. The Science of Reviewing Research (1984)
[35] Macaskill, A comparison of methods to detect publication bias in meta-analysis, Statistics in Medicine 20 pp 641– (2001) · doi:10.1002/sim.698
[36] McCullagh, Generalized Linear Models (1989) · Zbl 0588.62104 · doi:10.1007/978-1-4899-3242-6
[37] Moher, Completeness of reporting of trials published in languages other than English: implications for conduct and reporting of systematic reviews, The Lancet 347 pp 363– (1996) · doi:10.1016/S0140-6736(96)90538-3
[38] Moher, What contributions do languages other than English make on the results of meta-analyses?, Journal of Clinical Epidemiology 53 pp 964– (2000) · doi:10.1016/S0895-4356(00)00188-8
[39] Moore, Quantitative systematic review of topically applied non-steroidal anti-inflammatory drugs, British Medical Journal 316 pp 333– (1998) · doi:10.1136/bmj.316.7128.333
[40] Moreno, Assessment of regression-based methods to adjust for publication bias through a comprehensive simulation study, BMC Medical Research Methodology 9 pp 2– (2009a) · doi:10.1186/1471-2288-9-2
[41] Moreno, Novel methods to deal with publication biases: secondary analysis of antidepressant trials in the FDA trial registry database and related journal publications, British Medical Journal 339 (2009b) · doi:10.1136/bmj.b2981
[42] Nieminen, The relationship between quality of research and citation frequency, BMC Medical Research Methodology 6 pp 42– (2006) · doi:10.1186/1471-2288-6-42
[43] Nieminen, Statistically significant papers in psychiatry were cited more often than others, Journal of Clinical Epidemiology 60 pp 939– (2007) · doi:10.1016/j.jclinepi.2006.11.014
[44] Peters, Comparison of two methods to detect publication bias in meta-analysis, Journal of the American Medical Association 295 pp 676– (2006) · doi:10.1001/jama.295.6.676
[45] Peters, Assessing publication bias in meta-analyses in the presence of between-study heterogeneity, Journal of the Royal Statistical Society Series A 173 pp 575– (2010) · Zbl 05729458 · doi:10.1111/j.1467-985X.2009.00629.x
[46] Rothstein, Publication Bias in Meta Analysis: Prevention, Assessment and Adjustments (2005) · Zbl 1231.62209 · doi:10.1002/0470870168
[47] Rücker , G. 2010 http://www.freidok.uni-freiburg.de/volltexte/7486/
[48] Rücker, Arcsine test for publication bias in meta-analyses with binary outcomes, Statistics in Medicine 27 pp 746– (2008a) · doi:10.1002/sim.2971
[49] Rücker, Undue reliance on I2 in assessing heterogeneity may mislead, BMC Medical Research Methodology 8 pp 79– (2008b) · doi:10.1186/1471-2288-8-79
[50] Rücker, Treatment effect estimates adjusted for small-study effects via a limit meta-analysis, Biostatistics (2010)
[51] Rücker, Are large trials less reliable than small trials? Letter to the editor, Journal of Clinical Epidemiology 62 pp 886– (2009) · doi:10.1016/j.jclinepi.2009.03.007
[52] Rust, Estimating publication bias in meta-analysis, Journal of Marketing Research 27 pp 220– (1990) · doi:10.2307/3172848
[53] Schwarzer, Inflation of type I error rate in two statistical tests for the detection of publication bias in meta-analyses with binary outcomes, Statistics in Medicine 21 pp 2465– (2002) · doi:10.1002/sim.1224
[54] Schwarzer, A test for publication bias in meta-analysis with sparse binary data, Statistics in Medicine 26 pp 721– (2007) · doi:10.1002/sim.2588
[55] Schwarzer, Empirical evaluation suggests Copas selection model preferable to Trim-and-Fill for selection bias in meta-analysis, Journal of Clinical Epidemiology 63 pp 282– (2010) · doi:10.1016/j.jclinepi.2009.05.008
[56] Sharp, Analysing the relationship between treatment effect and underlying risk in meta-analysis: comparison and development of approaches, Statistics in Medicine 19 pp 3251– (2000) · doi:10.1002/1097-0258(20001215)19:23<3251::AID-SIM625>3.0.CO;2-2
[57] Sidik, Simple heterogeneity variance estimation for meta-analysis, Journal of the Royal Statistical Society Series C 54 pp 367– (2005) · Zbl 05188691 · doi:10.1111/j.1467-9876.2005.00489.x
[58] Simes, Publication bias: the case for an international registry of clinical trials, Journal of Clinical Oncology 4 pp 1529– (1986) · Zbl 0613.62067
[59] Stanley, Could it be better to discard 90% of the data? A statistical paradox, The American Statistician 64 pp 70– (2010) · Zbl 05798281 · doi:10.1198/tast.2009.08205
[60] Stanley, Meta-regression methods for detecting and estimating empirical effects in the presence of publication selection, Oxford Bulletin of Economics and Statistics 70 (2008)
[61] Sterling, Publication decisions and their possible effects on inferences drawn from tests of significance - or vice versa, Journal of the American Statistical Association 54 pp 30– (1959) · doi:10.2307/2282137
[62] Sterling, Publication decisions revisited: the effect of the outcome of statistical tests on the decision to publish and vice versa, The American Statistician 49 pp 108– (1995) · doi:10.2307/2684823
[63] Stern, Publication bias: evidence of delayed publication in a cohort study of clinical research projects, British Medical Journal 315 pp 640– (1997) · doi:10.1136/bmj.315.7109.640
[64] Sterne, Funnel plots for detecting bias in meta-analysis: Guideline on choice of axis, Journal of Clinical Epidemiology 54 pp 1046– (2001) · doi:10.1016/S0895-4356(01)00377-8
[65] Sterne, Publication and related bias in meta-analysis: power of statistical tests and prevalence in the literature, Journal of Clinical Epidemiology 53 pp 1119– (2000) · doi:10.1016/S0895-4356(00)00242-0
[66] Sutton, Empirical assessment of effect of publication bias on meta-analyses, British Medical Journal 320 pp 1574– (2000a) · doi:10.1136/bmj.320.7249.1574
[67] Sutton, Modelling publication bias in meta-analysis: a review, Statistical Methods in Medical Research 9 pp 421– (2000b) · Zbl 1121.62668 · doi:10.1191/096228000701555244
[68] Tang, Misleading funnel plot for detection of bias in meta-analysis, Journal of Clinical Epidemiology 53 pp 477– (2000) · doi:10.1016/S0895-4356(99)00204-8
[69] Taylor, Practical estimates of the effect of publication bias in meta-analysis, Australian Epidemiologist 5 pp 14– (1998)
[70] Terrin, In an empirical evaluation of the funnel plot, researchers could not visually identify publication bias, Journal of Clinical Epidemiology 58 pp 894– (2005) · doi:10.1016/j.jclinepi.2005.01.006
[71] Terrin, Adjusting for publication bias in the presence of heterogeneity, Statistics in Medicine 22 pp 2113– (2003) · doi:10.1002/sim.1461
[72] Thompson, How should meta-regression analyses be undertaken and interpreted?, Statistics in Medicine 21 pp 1559– (2002) · doi:10.1002/sim.1187
[73] Tramèr, Impact of covert duplicate publication on meta-analysis: a case study, British Medical Journal 315 pp 635– (1997) · doi:10.1136/bmj.315.7109.635
[74] Viechtbauer, Confidence intervals for the amount of heterogeneity in meta-analysis, Statistics in Medicine 26 pp 37– (2007) · doi:10.1002/sim.2514
[75] Williamson, Identification and impact of outcome selection bias in meta-analysis, Statistics in Medicine 24 pp 1547– (2005) · doi:10.1002/sim.2025
This reference list is based on information provided by the publisher or from digital mathematics libraries. Its items are heuristically matched to zbMATH identifiers and may contain data conversion errors. It attempts to reflect the references listed in the original paper as accurately as possible without claiming the completeness or perfect precision of the matching.