Publication Bias in Meta-Analysis: Confidence Intervals for Rosenthal’s Fail-Safe Number

Fragkos, Konstantinos C. and Tsagris, Michail and Frangos, Christos C. (2014): Publication Bias in Meta-Analysis: Confidence Intervals for Rosenthal’s Fail-Safe Number. Published in: International Scholarly Research Notices , Vol. 2014, (December 2014): pp. 1-17.

Abstract

The purpose of the present paper is to assess the efficacy of confidence intervals for Rosenthal’s fail-safe number. Although Rosenthal’s estimator is highly used by researchers, its statistical properties are largely unexplored. First of all, we developed statistical theory which allowed us to produce confidence intervals for Rosenthal’s fail-safe number.This was produced by discerning whether the number of studies analysed in a meta-analysis is fixed or random. Each case produces different variance estimators. For a given number of studies and a given distribution, we provided five variance estimators. Confidence intervals are examined with a normal approximation and a nonparametric bootstrap. The accuracy of the different confidence interval estimates was then tested by methods of simulation under different distributional assumptions. The half normal distribution variance estimator has the best probability coverage. Finally, we provide a table of lower confidence intervals for Rosenthal’s estimator.

• Item Type: MPRA Paper
• Language: English
• Keywords: Meta-analysis, Rosenthal's fail safe number, file-drawer problem, bootstrap
• Subjects: C - Mathematical and Quantitative Methods > C1 - Econometric and Statistical Methods and Methodology: General > C18 - Methodological Issues: General; C - Mathematical and Quantitative Methods > C1 - Econometric and Statistical Methods and Methodology: General > C19 - Other

References

[1] M. Borenstein, L. V. Hedges, J. P. T. Higgins, and H. R. Rothstein, Introduction to Meta-Analysis, John Wiley & Sons, Chichester, UK, 2nd edition, 2011.

[2] L. V. Hedges and J. L. Vevea, “Fixed- and random-effects models in meta-analysis,” Psychological Methods, vol. 3, no. 4, pp. 486–504, 1998.

[3] A. Whitehead and J. Whitehead, “A general parametric approach to the meta-analysis of randomized clinical trials,” Statistics in Medicine, vol. 10, no. 11, pp. 1665–1677, 1991.

[4] K. J. Rothman, S. Greenland, and T. L. Lash, Modern Epidemiology, Lippincott Williams & Wilkins, Philadelphia, Pa, USA, 2008.

[5] S. L. T. Normand, “Meta-analysis: formulating, evaluating, combining, and reporting,” Statistics in Medicine, vol. 18, no. 3, pp. 321–359, 1999.

[6] S. E. Brockwell and I. R. Gordon, “A comparison of statistical methods for meta-analysis,” Statistics in Medicine, vol. 20, no. 6, pp. 825–840, 2001.

[7] R. Rosenthal, “Combining results of independent studies,” Psychological Bulletin, vol. 85, no. 1, pp. 185–193, 1978.

[8] L. V. Hedges and I. Olkin, Statistical Methods for Meta-Analysis, Academic Press, New York, NY, USA, 1985.

[9] H. Cooper, L. V. Hedges, and J. C. Valentine, Handbook of Research Synthesis and Meta-Analysis, Russell Sage Foundation, New York, NY, USA, 2nd edition, 2009.

[10] A. J. Sutton, F. Song, S. M. Gilbody, and K. R. Abrams, “Modelling publication bias in meta-analysis: a review,” Statistical Methods in Medical Research, vol. 9, no. 5, pp. 421–445, 2000.

[11] R. Rosenthal, “The file drawer problem and tolerance for null results,” Psychological Bulletin, vol. 86, no. 3, pp. 638–641, 1979.

[12] S. Iyengar and J. B. Greenhouse, “Selection models and the file drawer problem,” Statistical Science, vol. 3, no. 1, pp. 109–117, 1988.

[13] C. B. Begg and J. A. Berlin, “Publication bias: a problem in interpreting medical data (C/R: pp. 445-463),” Journal of the Royal Statistical Society Series A: Statistics in Society, vol. 151, no. 3, pp. 445–463, 1988.

[14] A. Thornton and P. Lee, “Publication bias in meta-analysis: its causes and consequences,” Journal of Clinical Epidemiology, vol. 53, no. 2, pp. 207–216, 2000.

[15] J. D. Kromrey and G. Rendina-Gobioff, “On knowing what we do not know: an empirical comparison of methods to detect publication bias in meta-analysis,” Educational and Psychological Measurement, vol. 66, no. 3, pp. 357–373, 2006.

[16] H. R. Rothstein, A. J. Sutton, and M. Borenstein, Publication Bias in Meta-Analysis: Prevention, Assessment and Adjustments, John Wiley & Sons, Chichester, UK, 2006.

[17] S. Nakagawa and E. S. A. Santos, “Methodological issues and advances in biological meta-analysis,” Evolutionary Ecology, vol. 26, no. 5, pp. 1253–1274, 2012.

[18] S. Kepes, G. Banks, and I.-S. Oh, “Avoiding bias in publication bias research: the value of “null” findings,” Journal of Business and Psychology, vol. 29, no. 2, pp. 183–203, 2014.

[19] J. P. A. Ioannidis and T. A. Trikalinos, “An exploratory test for an excess of significant findings,” Clinical Trials, vol. 4, no. 3, pp. 245–253, 2007.

[20] U. Simonsohn, L. D. Nelson, and J. P. Simmons, “P-curve: a key to the file-drawer,” Journal of Experimental Psychology: General, vol. 143, no. 2, pp. 534–547, 2014.

[21] C. J. Ferguson and M. T. Brannick, “Publication bias in psychological science: prevalence, methods for identifying and controlling, and implications for the use of meta-analyses,” Psychological Methods, vol. 17, no. 1, pp. 120–128, 2012.

[22] J. Cohen, Statistical Power Analysis for the Behavioral Sciences, Lawrence Erlbaum, Hillsdale , NJ, USA, 1988.

[23] B. J. Becker, “The failsafe N or file-drawer number,” in Publication Bias in Meta-Analysis: Prevention, Assessment and Adjustments, H. R. Rothstein, A. J. Sutton, and M. Borenstein, Eds., pp. 111–126, John Wiley & Sons, Chichester, UK, 2005.

[24] S. Hempel, S. J. Newberry, A. R. Maher et al., “Probiotics for the prevention and treatment of antibiotic-associated diarrhea: a systematic review and meta-analysis,” The Journal of the American Medical Association, vol. 307, no. 18, pp. 1959–1969, 2012.

[25] D. Balliet, L. B. Mulder, and P. A. M. van Lange, “Reward, punishment, and cooperation: a meta-analysis,” Psychological Bulletin, vol. 137, no. 4, pp. 594–615, 2011.

[26] M. A. McDaniel, H. R. Rothotein, and D. L. Whetz, “Publication bias: a case study of four test vendors,” Personnel Psychology, vol. 59, no. 4, pp. 927–953, 2006.

[27] S. A. Stouffer, E. A. Suchman, L. C. De Vinney et al., The American Soldier: Adjustment During Army Life, vol. 1 of Studies in social psychology in World War II, Princeton University Press, Oxford, UK, 1949.

[28] H. M. Cooper, “Statistically combining independent studies: a meta-analysis of sex differences in conformity research,” Journal of Personality and Social Psychology, vol. 37, no. 1, pp. 131–146, 1979.

[29] H. M. Cooper and R. Rosenthal, “Statistical versus traditional procedures for summarizing research findings,” Psychological Bulletin, vol. 87, no. 3, pp. 442–449, 1980.

[30] K. K. Zakzanis, “Statistics to tell the truth, the whole truth, and nothing but the truth: formulae, illustrative numerical examples, and heuristic interpretation of effect size analyses for neuropsychological researchers,” Archives of Clinical Neuropsychology, vol. 16, no. 7, pp. 653–667, 2001.

[31] L. M. Hsu, “Fail-safe Ns for one- versus two-tailed tests lead to different conclusions about publication bias,” Understanding Statistics, vol. 1, no. 2, pp. 85–100, 2002.

[32] B. J. Becker, “Combining significance levels,” in The Handbook of Research Synthesis, H. Cooper and L. V. Hedges, Eds., pp. 215–230, Russell Sage, New York, NY, USA, 1994.

[33] H. Aguinis, C. A. Pierce, F. A. Bosco, D. R. Dalton, and C. M. Dalton, “Debunking myths and urban legends about metaanalysis,” Organizational Research Methods, vol. 14, no. 2, pp. 306–331, 2011.

[34] B. Mullen, P. Muellerleile, and B. Bryant, “Cumulative metaanalysis: a consideration of indicators of sufficiency and stability,” Personality and Social Psychology Bulletin, vol. 27, no. 11, pp. 1450–1462, 2001.

[35] K. P. Carson, C. A. Schriesheim, and A. J. Kinicki, “The usefulness of the “fail-safe” statistic in meta-analysis,” Educational and Psychological Measurement, vol. 50, no. 2, pp. 233–243, 1990.

[36] S. Duval and R. Tweedie, “A nonparametric “trim and fill” method of accounting for publication bias in meta-analysis,” Journal of the American Statistical Association, vol. 95, no. 449, pp. 89–98, 2000.

[37] S. Duval and R. Tweedie, “Trim and fill: a simple funnel-plot-based method of testing and adjusting for publication bias in meta-analysis,” Biometrics, vol. 56, no. 2, pp. 455–463, 2000.

[38] R. G. Orwin, “A fail-safe N for effect size in meta-analysis,” Journal of Educational Statistics, vol. 8, no. 2, pp. 157–159, 1983.

[39] M. S. Rosenberg, “The file-drawer problem revisited: a general weighted method for calculating fail-safe numbers in metaanalysis,” Evolution, vol. 59, no. 2, pp. 464–468, 2005.

[40] L. J. Gleser and I. Olkin, “Models for estimating the number of unpublished studies,” Statistics in Medicine, vol. 15, no. 23, pp. 2493–2507, 1996.

[41] L. E. Eberly and G. Casella, “Bayesian estimation of the number of unseen studies in a meta-analysis,” Journal of Official Statistics, vol. 15, no. 4, pp. 477–494, 1999.

[42] J. W. Lindeberg, “Eine neue herleitung des exponentialgesetzes in der Wahrscheinlichkeitsrechnung,” Mathematische Zeitschrift, vol. 15, no. 1, pp. 211–225, 1922.

[43] N. L. Johnson, S. Kotz, and N. Balakrishnan, Continuous Univariate Distributions, vol. 2, John Wiley & Sons, New York, NY, USA, 2nd edition, 1994.

[44] G. Casella and R. L. Berger, Statistical Inference, Duxbury/Thomson Learning, Pacific Grove, Calif, USA, 2nd edition, 2002.

[45] R. C. Elandt, “The folded normal distribution: two methods of estimating parameters from moments,” Technometrics, vol. 3, no. 4, pp. 551–562, 1961.

[46] F. C. Leone, L. S. Nelson, and R. B. Nottingham, “The folded normal distribution,” Technometrics, vol. 3, no. 4, pp. 543–550, 1961.

[47] D. Mavridis, A. Sutton, A. Cipriani, and G. Salanti, “A fully Bayesian application of the Copas selection model for publication bias extended to network meta-analysis,” Statistics in Medicine, vol. 32, no. 1, pp. 51–66, 2013.

[48] P. H. Schonemann and J. D. Scargle, “A generalized publication bias model,” Chinese Journal of Psychology, vol. 50, no. 1, pp. 21–29, 2008.

[49] A. Azzalini, “A class of distributions which includes the normal ones,” Scandinavian Journal of Statistics, vol. 12, no. 2, pp. 171–178, 1985.

[50] T. I. Lin, J. C. Lee, and S. Y. Yen, “Finite mixture modelling using the skew normal distribution,” Statistica Sinica, vol. 17, no. 3, pp. 909–927, 2007.

[51] B. C. Arnold, R. J. Beaver, R. A. Groeneveld, and W. Q. Meeker, “The nontruncated marginal of a truncated bivariate normal distribution,” Psychometrika, vol. 58, no. 3, pp. 471–488, 1993.

[52] C. C. Frangos, “An updated bibliography on the jackknife method,” Communications in Statistics—Theory and Methods, vol. 16, no. 6, pp. 1543–1584, 1987.

[53] M. Tsagris, I. Elmatzoglou, and C. C. Frangos, “The assessment of performance of correlation estimates in discrete bivariate distributions using bootstrap methodology,” Communications in Statistics—Theory and Methods, vol. 41, no. 1, pp. 138–152, 2012.

[54] B. Efron, “Bootstrap methods: another look at the jackknife,” The Annals of Statistics, vol. 7, no. 1, pp. 1–26, 1979.

[55] B. Efron and R. J. Tibshirani, An Introduction to the Bootstrap, Chapman & Hall, Boca Raton, Fla, USA, 1993.

[56] A. C. Davison and D. V. Hinkley, Bootstrap Methods and Their Application, Cambridge University Press, New York, NY, USA, 1997.

[57] J. Carpenter and J. Bithell, “Bootstrap confidence intervals: when, which, what? A practical guide for medical statisticians,” Statistics in Medicine, vol. 19, no. 9, pp. 1141–1164, 2000.

[58] J. D. Scargle, “Publication bias: the “File-Drawer” problem in scientific inference,” Journal of Scientific Exploration, vol. 14, no. 1, pp. 91–106, 2000.

[59] J. K. Patel and C. B. Read, Handbook of the Normal Distribution, Statistics: Textbooks and Monographs, Marcel Dekker, New York, NY, USA, 2nd edition, 1982.

[60] F. A. Haight, Handbook of the Poisson Distribution, Wiley, New York, NY, USA, 1967.