Do better journals publish better estimates?

Slichter, David and Tran, Nhan (2023): Do better journals publish better estimates?

Abstract

Are estimates typically closer to the true parameter value when those estimates are published in highly-ranked economics journals? Using 14,387 published estimates from 24 large literatures, we find that, within literatures, the mean and variance of parameter estimates have little or no correlation with journal rank. Therefore, regardless of what the true parameter value is that a literature is attempting to estimate, it cannot be that estimates in higher-ranked journals are on average noticeably closer to it. We discuss possible explanations and implications.

• Item Type: MPRA Paper
• Language: English
• Keywords: Meta-analysis, scientific methods, publication, science of science
• Subjects: A - General Economics and Teaching > A1 - General Economics > A11 - Role of Economics ; Role of Economists ; Market for Economists; C - Mathematical and Quantitative Methods > C1 - Econometric and Statistical Methods and Methodology: General > C13 - Estimation: General; C - Mathematical and Quantitative Methods > C1 - Econometric and Statistical Methods and Methodology: General > C18 - Methodological Issues: General

References

Askarov, Z., & Doucouliagos, H. (2020). A meta-analysis of the effects of remittances on household education expenditure. World Development, 129, 104860.

Bajzik, J., Havranek, T., Irsova, Z., & Schwarz, J. (2020). Estimating the Armington elasticity: The importance of study design and publication bias. Journal of International Economics, 127, 103383.

Brembs, B., Button, K., & Munafo, M. (2013). Deep impact: Unintended consequences of journal rank. Frontiers in Human Neuroscience, 7: 291.

Brodeur, A., Cook, N., & Heyes, A. (2020). Methods matter: P-hacking and publication bias in causal analysis in economics. American Economic Review, 110(11), 3634-3660.

Chang, A. C., & Li, P. (2015). Is economics research replicable? Sixty published papers from thirteen journals say ’usually not’. Working paper.

Dahl, C. A. (2012). Measuring global gasoline and diesel price and income elasticities. Energy Policy, 41, 2-13.

Demena, B. A. (2015). Publication bias in FDI spillovers in developing countries: a meta-regression analysis. Applied Economics Letters, 22(14), 1170-1174.

Dewald, W. G., Thursby, J. G., & Anderson, R. G. (1986). Replication in empirical economics: The journal of money, credit and banking project. American Economic Review, 587-603.

Doucouliagos, C. (2005). Publication bias in the economic freedom and economic growth literature. Journal of Economic Surveys, 19(3), 367-387.

Doucouliagos, H., & Stanley, T. D. (2009). Publication selection bias in minimum-wage research? A meta-regression analysis. British Journal of Industrial Relations, 47(2), 406-428.

Fleury, N., & Gilles, F. (2018). The intergenerational transmission of education. A meta-regression analysis. Education Economics, 26(6), 557-573.

Gallet, C. A., & Doucouliagos, H. (2017). The impact of healthcare spending on health outcomes: A meta-regression analysis. Social Science & Medicine, 179, 9-17.

Gechert, S., Havranek, T., Irsova, Z., & Kolcunova, D. (2022). Measuring capital-labor substitution: The importance of method choices and publication bias. Review of Economic Dynamics, 45, 55-82.

Gelman, A., & Loken, E. (2013). The garden of forking paths: Why multiple comparisons can be a problem, even when there is no “fishing expedition” or “p-hacking” and the research hypothesis was posited ahead of time. Department of Statistics, Columbia University, 348, 1-17.

Havranek, T. (2013). Publication bias in measuring intertemporal substitution (No. 15/2013). IES Working Paper.

Havranek, T. (2015). Measuring intertemporal substitution: The importance of method choices and selective reporting. Journal of the European Economic Association, 13(6), 1180-1204.

Havranek, T., Irsova, Z., & Janda, K. (2012). Demand for gasoline is more price-inelastic than commonly thought. Energy Economics, 34(1), 201-207.

Havranek, T., Irsova, Z., Janda, K., & Zilberman, D. (2015). Selective reporting and the social cost of carbon. Energy Economics, 51, 394-406.

Havranek, T., Irsova, Z., Laslopova, L., & Zeynalova, O. (2022). Publication and Attenuation Biases in Measuring Skill Substitution. The Review of Economics and Statistics, forthcoming.

Havranek, T., Irsova, Z., Vlach, T. (2018). Measuring the income elasticity of water demand: the importance of publication and endogeneity biases. Land Economics, 94(2), 259-283.

Havranek, T., Irsova, Z., & Zeynalova, O. (2018). Tuition fees and university enrolment: a meta-regression analysis. Oxford Bulletin of Economics and Statistics, 80(6), 1145-1184.

Havranek, T., & Sokolova, A. (2020). Do consumers really follow a rule of thumb? Three thousand estimates from 144 studies say “probably not”. Review of Economic Dynamics, 35, 97-122.

Huber, J., Inoua, S., Kerschbamer, R., & Smith, V. L. (2022). Nobel and novice: Author prominence affects peer review. Proceedings of the National Academy of Sciences, 119(41), e2205779119.

Huntington-Klein, N., Arenas, A., Beam, E., Bertoni, M., Bloem, J. R., Burli, P., Chen, N., Grieco, P., Ekpe, G., Pugatch, T., Saavedra, M., & Stopnitzky, Y. (2021). The influence of hidden researcher decisions in applied microeconomics. Economic Inquiry, 59(3), 944-960.

Ioannidis, J. P. (2005). Why most published research findings are false. PLoS medicine, 2(8), e124.

Ioannidis, J., & Doucouliagos, C. (2013). What’s to know about the credibility of empirical economics?. Journal of Economic Surveys, 27(5), 997-1004.

Ioannidis, J. P., Stanley, T. D., & Doucouliagos, H. (2017). The power of bias in economics research. Economic Journal, 127(605), F236-F265.

Kroupova, K., Havranek, T., & Irsova, Z. (2021). Student Employment and Education: A Meta-Analysis (No. 28/2021). IES Working Paper.

Lichter, A., Peichl, A., & Siegloch, S. (2015). The own-wage elasticity of labor demand: A meta-regression analysis. European Economic Review, 80, 94-119.

Longhi, S., Nijkamp, P., Poot, J. (2008). Meta-analysis of empirical evidence on the labour market impacts of immigration. Working paper.

Ma, X., Iwasaki, I. (2021). Return to schooling in China: A large meta-analysis. Education Economics, 29(4), 379-410.

Matousek, J., Havranek, T., & Irsova, Z. (2022). Individual discount rates: a meta-analysis of experimental evidence. Experimental Economics, 25(1), 318-358.

Neisser, C. (2021). The elasticity of taxable income: A meta-regression analysis. The Economic Journal, 131(640), 3365-3391.

Nelson, J. P. (2013). Meta-analysis of alcohol price and income elasticities—with corrections for publication bias. Health Economics Review, 3(1), 1-10.

Nijkamp, P., & Poot, J. (2005). The last word on the wage curve?. Journal of Economic Surveys, 19(3), 421-450.

Simonsohn, U., Simmons, J. P., & Nelson, L. D. (2020). Specification curve analysis. Nature Human Behaviour, 4(11), 1208-1214.

Siontis, K. C. M., Evangelou, E., & Ioannidis, J. P. A. (2011). Magnitude of effects in clinical trials published in high-impact general medical journals. International Journal of Epidemiology, 40(5): 1280-1291.

Xue, X., Cheng, M., & Zhang, W. (2021). Does education really improve health? A meta-analysis. Journal of Economic Surveys, 35(1), 71-105.

Young, A. (2022). Consistency without inference: Instrumental variables in practical application. European Economic Review, 147, 104112.