Evaluating replicability of laboratory experiments in Economics

Camerer, Colin and Dreber, Anna and Forsell, Eskil and Ho, Teck-Hua and Huber, Jurgen and Johannesson, Magnus and Kirchler, Michael and Almenberg, Johan and Altmejd, Adam and Chan, Taizan and Heikensten, Emma and Holzmeister, Felix and Imai, Taisuke and Isaksson, Siri and Nave, Gideon and Pfeiffer, Thomas and Razen, Michael and Wu, Hang (2016): Evaluating replicability of laboratory experiments in Economics. Published in: Science , Vol. 351, No. 6277 (3 March 2016)

Abstract

The reproducibility of scientific findings has been called into question. To contribute data about reproducibility in economics, we replicate 18 studies published in the American Economic Review and the Quarterly Journal of Economics in 2011-2014. All replications follow predefined analysis plans publicly posted prior to the replications, and have a statistical power of at least 90% to detect the original effect size at the 5% significance level. We find a significant effect in the same direction as the original study for 11 replications (61%); on average the replicated effect size is 66% of the original. The reproducibility rate varies between 67% and 78% for four additional reproducibility indicators, including a prediction market measure of peer beliefs.

• Item Type: MPRA Paper
• Language: English
• Keywords: replication, economics, experiment
• Subjects: A - General Economics and Teaching > A1 - General Economics > A12 - Relation of Economics to Other Disciplines; B - History of Economic Thought, Methodology, and Heterodox Approaches > B4 - Economic Methodology > B41 - Economic Methodology; C - Mathematical and Quantitative Methods > C9 - Design of Experiments

References

1. M. McNutt, Reproducibility. Science 343, 229 (2014). Medline doi:10.1126/science.1250475

2. J. P. A. Ioannidis, Why most published research findings are false. PLOS Med. 2, e124 (2005). Medline doi:10.1371/journal.pmed.0020124

3. F. Prinz, T. Schlange, K. Asadullah, Believe it or not: How much can we rely on published data on potential drug targets? Nat. Rev. Drug Discov. 10, 712 (2011). Medline doi:10.1038/nrd3439-c1

4. C. G. Begley, L. M. Ellis, Drug development: Raise standards for preclinical cancer research. Nature 483, 531–533 (2012). Medline doi:10.1038/483531a

5. L. P. Freedman, I. M. Cockburn, T. S. Simcoe, The economics of reproducibility in preclinical research. PLOS Biol. 13, e1002165 (2015). Medline doi:10.1371/journal.pbio.1002165

6. K. S. Button, J. P. Ioannidis, C. Mokrysz, B. A. Nosek, J. Flint, E. S. Robinson, M. R. Munafò, Power failure: Why small sample size undermines the reliability of neuroscience. Nat. Rev. Neurosci. 14, 365–376 (2013). Medline doi:10.1038/nrn3475

7. J. K. Hewitt, Editorial policy on candidate gene association and candidate gene-by-environment interaction studies of complex traits. Behav. Genet. 42, 1–2 (2012). Medline doi:10.1007/s10519-011-9504-z

8. M. S. Lawrence, P. Stojanov, P. Polak, G. V. Kryukov, K. Cibulskis, A. Sivachenko, S. L. Carter, C. Stewart, C. H. Mermel, S. A. Roberts, A. Kiezun, P. S. Hammerman, A. McKenna, Y. Drier, L. Zou, A. H. Ramos, T. J. Pugh, N. Stransky, E. Helman, J. Kim, C. Sougnez, L. Ambrogio, E. Nickerson, E. Shefler, M. L. Cortés, D. Auclair, G. Saksena, D. Voet, M. Noble, D. DiCara, P. Lin, L. Lichtenstein, D. I. Heiman, T. Fennell, M. Imielinski, B. Hernandez, E. Hodis, S. Baca, A. M. Dulak, J. Lohr, D. A. Landau, C. J. Wu, J. Melendez-Zajgla, A. Hidalgo-Miranda, A. Koren, S. A. McCarroll, J. Mora, R. S. Lee, B. Crompton, R. Onofrio, M. Parkin, W. Winckler, K. Ardlie, S. B. Gabriel, C. W. Roberts, J. A. Biegel, K. Stegmaier, A. J. Bass, L. A. Garraway, M. Meyerson, T. R. Golub, D. A. Gordenin, S. Sunyaev, E. S. Lander, G. Getz, Mutational heterogeneity in cancer and the search for new cancer-associated genes. Nature 499, 214–218 (2013). Medline doi:10.1038/nature12213

9. E. E. Leamer, Let’s take the con out of econometrics. Am. Econ. Rev. 73, 31 (1983).

10. A. E. Roth, Let’s keep the con out of experimental econ.: A methodological note. Empir. Econ. 19, 279–289 (1994). doi:10.1007/BF01175875

11. Z. Maniadis, F. Tufano, J. A. List, One swallow doesn’t make a summer: New evidence on anchoring effects. Am. Econ. Rev. 104, 277–290 (2014). doi:10.1257/aer.104.1.277

12. J. P. Simmons, L. D. Nelson, U. Simonsohn, False-positive psychology: Undisclosed flexibility in data collection and analysis allows presenting anything as significant. Psychol. Sci. 22, 1359–1366 (2011). Medline doi:10.1177/0956797611417632

13. S. Carpenter, Psychology’s bold initiative. Science 335, 1558–1561 (2012). Medline doi:10.1126/science.335.6076.1558

14. Open Science Collaboration, An open, large-scale, collaborative effort to estimate the reproducibility of psychological science. Perspect. Psychol. Sci. 7, 657–660 (2012). Medline doi:10.1177/1745691612462588

15. J. Bohannon, Replication effort provokes praise—and ‘bullying’ charges. Science 344, 788–789 (2014). Medline doi:10.1126/science.344.6186.788

16. S. Doyen, O. Klein, C.-L. Pichon, A. Cleeremans, Behavioral priming: It’s all in the mind, but whose mind? PLOS ONE 7, e29081 (2012). Medline doi:10.1371/journal.pone.0029081

17. S. J. Ritchie, R. Wiseman, C. C. French, Failing the future: Three unsuccessful attempts to replicate Bem’s ‘retroactive facilitation of recall’ effect. PLOS ONE 7, e33423 (2012). Medline doi:10.1371/journal.pone.0033423

18. B. A. Nosek, G. Alter, G. C. Banks, D. Borsboom, S. D. Bowman, S. J. Breckler, S. Buck, C. D. Chambers, G. Chin, G. Christensen, M. Contestabile, A. Dafoe, E. Eich, J. Freese, R. Glennerster, D. Goroff, D. P. Green, B. Hesse, M. Humphreys, J. Ishiyama, D. Karlan, A. Kraut, A. Lupia, P. Mabry, T. Madon, N. Malhotra, E. Mayo-Wilson, M. McNutt, E. Miguel, E. L. Paluck, U. Simonsohn, C. Soderberg, B. A. Spellman, J. Turitto, G. VandenBos, S. Vazire, E. J. Wagenmakers, R. Wilson, T. Yarkoni, Promoting an open research culture. Science 348, 1422–1425 (2015). Medline doi:10.1126/science.aab2374

19. Open Science Collaboration, Estimating the reproducibility of psychological science. Science 349, aac4716 (2015). Medline doi:10.1126/science.aac4716

20. A. Gelman, H. Stern, The difference between “significant” and “not significant” is not itself statistically significant. Am. Stat. 60, 328–331 (2006). doi:10.1198/000313006X152649

21. G. Cumming, Replication and p intervals: P values predict the future only vaguely, but confidence intervals do much better. Perspect. Psychol. Sci. 3, 286–300 (2008). Medline doi:10.1111/j.1745-6924.2008.00079.x

22. J. Verhagen, E.-J. Wagenmakers, Bayesian tests to quantify the result of a replication attempt. J. Exp. Psychol. Gen. 143, 1457–1475 (2014). Medline doi:10.1037/a0036731

23. U. Simonsohn, Small telescopes: Detectability and the evaluation of replication results. Psychol. Sci. 26, 559–569 (2015). Medline doi:10.1177/0956797614567341

24. J. T. Leek, P. Patil, R. D. Peng, http://arxiv.org/abs/1509.08968 (2015).

25. K. J. Arrow, R. Forsythe, M. Gorham, R. Hahn, R. Hanson, J. O. Ledyard, S. Levmore, R. Litan, P. Milgrom, F. D. Nelson, G. R. Neumann, M. Ottaviani, T. C. Schelling, R. J. Shiller, V. L. Smith, E. Snowberg, C. R. Sunstein, P. C. Tetlock, P. E. Tetlock, H. R. Varian, J. Wolfers, E. Zitzewitz, The promise of prediction markets. Science 320, 877–878 (2008). Medline doi:10.1126/science.1157679

26. A. Dreber, T. Pfeiffer, J. Almenberg, S. Isaksson, B. Wilson, Y. Chen, B. A. Nosek, M. Johannesson, Using prediction markets to estimate the reproducibility of scientific research. Proc. Natl. Acad. Sci. U.S.A. 112, 15343–15347 (2015). Medline doi:10.1073/pnas.1516179112

27. J. Wolfers, E. Zitzewitz, Interpreting Prediction Market Prices as Probabilities (Working Paper No. 12200, National Bureau of Economic Research, 2006).

28. J. P. A. Ioannidis, Why most discovered true associations are inflated. Epidemiology 19, 640–648 (2008). Medline doi:10.1097/EDE.0b013e31818131e7

29. B. D. McCullough, H. D. Vinod, Verifying the solution from a nonlinear solver: A case study. Am. Econ. Rev. 93, 873–892 (2003). doi:10.1257/000282803322157133

30. W. G. Dewald, J. G. Thursby, R. G. Anderson, Replication in empirical economics: The journal of money, credit and banking project. Am. Econ. Rev. 76, 587 (1986).

31. B. D. McCullough, K. A. McGeary, T. D. Harrison, Lessons from the JMCB Archive. J. Money Credit Bank. 38, 1093–1107 (2006). doi:10.1353/mcb.2006.0061

32. A. Brodeur, M. Lé, M. Sangnier, Y. Zylberberg, Star Wars: The empirics strike back. AEJ Applied 8, 1–32 (2016).

33. J. Abeler, A. Falk, L. Goette, D. Huffman, Reference points and effort provision. Am. Econ. Rev. 101, 470–492 (2011). doi:10.1257/aer.101.2.470

34. A. Ambrus, B. Greiner, Imperfect public monitoring with costly punishment: An experimental study. Am. Econ. Rev. 102, 3317–3332 (2012). doi:10.1257/aer.102.7.3317

35. B. Bartling, E. Fehr, K. M. Schmidt, Screening, competition, and job design: Economic origins of good jobs. Am. Econ. Rev. 102, 834–864 (2012). doi:10.1257/aer.102.2.834

36. G. Charness, M. Dufwenberg, Participation. Am. Econ. Rev. 101, 1211–1237 (2011). doi:10.1257/aer.101.4.1211

37. R. Chen, Y. Chen, The potential of social identity for equilibrium selection. Am. Econ. Rev. 101, 2562–2589 (2011). doi:10.1257/aer.101.6.2562

38. G. de Clippel, K. Eliaz, B. G. Knight, On the selection of arbitrators. Am. Econ. Rev. 104, 3434–3458 (2014). doi:10.1257/aer.104.11.3434

39. J. Duffy, D. Puzzello, Gift exchange versus monetary exchange: Theory and evidence. Am. Econ. Rev. 104, 1735–1776 (2014). doi:10.1257/aer.104.6.1735

40. U. Dulleck, R. Kerschbamer, M. Sutter, The economics of credence goods: An experiment on the role of liability, verifiability, reputation, and competition. Am. Econ. Rev. 101, 526–555 (2011). doi:10.1257/aer.101.2.526

41. K. M. Marzilli Ericson, A. Fuster, Expectations as endowments: Evidence on reference-dependent preferences from exchange and valuation experiments. Q. J. Econ. 126, 1879–1907 (2011). doi:10.1093/qje/qjr034

42. E. Fehr, H. Herz, T. Wilkening, The lure of authority: Motivation and incentive effects of power. Am. Econ. Rev. 103, 1325–1359 (2013). doi:10.1257/aer.103.4.1325

43. D. Friedman, R. Oprea, A continuous dilemma. Am. Econ. Rev. 102, 337–363 (2012). doi:10.1257/aer.102.1.337

44. D. Fudenberg, D. G. Rand, A. Dreber, Slow to anger and fast to forgive: Cooperation in an uncertain world. Am. Econ. Rev. 102, 720–749 (2012). doi:10.1257/aer.102.2.720

45. S. Huck, A. J. Seltzer, B. Wallace, Deferred compensation in multiperiod labor contracts: An experimental test of Lazear’s model. Am. Econ. Rev. 101, 819–843 (2011). doi:10.1257/aer.101.2.819

46. J. Ifcher, H. Zarghamee, Happiness and time preference: The effect of positive affect in a random-assignment experiment. Am. Econ. Rev. 101, 3109–3129 (2011). doi:10.1257/aer.101.7.3109

47. J. B. Kessler, A. E. Roth, Organ allocation policy and the decision to donate. Am. Econ. Rev. 102, 2018–2047 (2012). doi:10.1257/aer.102.5.2018

48. M. Kirchler, J. Huber, T. Stöckl, Thar she bursts: Reducing confusion reduces bubbles. Am. Econ. Rev. 102, 865–883 (2012). doi:10.1257/aer.102.2.865

49. S. Kogan, A. M. Kwasnica, R. A. Weber, Coordination in the presence of asset markets. Am. Econ. Rev. 101, 927–947 (2011). doi:10.1257/aer.101.2.927

50. I. Kuziemko, R. W. Buell, T. Reich, M. I. Norton, “Last-place aversion”: Evidence and redistributive implications. Q. J. Econ. 129, 105–149 (2014). doi:10.1093/qje/qjt035

51. B. Merlob, C. R. Plott, Y. Zhang, The CMS auction: Experimental studies of a median-bid procurement auction with nonbinding bids. Q. J. Econ. 127, 793–827 (2012). doi:10.1093/qje/qjs013

52. C. C. Eckel, R. Petrie, Face value. Am. Econ. Rev. 101, 1497–1513 (2011). doi:10.1257/aer.101.4.1497

53. D. Gill, V. Prowse, A structural analysis of disappointment aversion in a real effort competition. Am. Econ. Rev. 102, 469–503 (2012). doi:10.1257/aer.102.1.469

54. N. Erkal, L. Gangadharan, N. Nikiforakis, Relative earnings and giving in a real-effort experiment. Am. Econ. Rev. 101, 3330–3348 (2011). doi:10.1257/aer.101.7.3330

55. U. Fischbacher, z-Tree: Zurich toolbox for ready-made economic experiments. Exp. Econ. 10, 171–178 (2007). doi:10.1007/s10683-006-9159-4

56. S. Palan, GIMS-Software for asset market experiments. J. Behav. Exp. Finance 5, 1–14 (2015). Medline doi:10.1016/j.jbef.2015.02.001

57. R. Hanson, Could gambling save science? Encouraging an honest consensus. Soc. Epistemology 9, 3–33 (1995). doi:10.1080/02691729508578768

58. J. Almenberg, K. Kittlitz, T. Pfeiffer, An experiment on prediction markets in science. PLOS ONE 4, e8500 (2009). Medline doi:10.1371/journal.pone.0008500

59. J. Wolfers, E. Zitzewitz, Prediction markets. J. Econ. Perspect. 18, 107–126 (2004). doi:10.1257/0895330041371321

60. G. Tziralis, I. Tatsiopoulos, Prediction markets: An extended literature review. J. Pred. Mark. 1, 75 (2007).

61. J. Berg, R. Forsythe, F. Nelson, T. Rietz, Results from a dozen years of election futures markets research, Handbook of Experimental Economics Results 1, 742 (2008).

62. C. F. Horn, B. S. Ivens, M. Ohneberg, A. Brem, Prediction markets – a literature review 2014. J. Pred. Mark. 8, 89 (2014).

63. C. F. Manski, Interpreting the predictions of prediction markets. Econ. Lett. 91, 425–429 (2006). doi:10.1016/j.econlet.2006.01.004

64. U. Sonnemann, C. F. Camerer, C. R. Fox, T. Langer, How psychological framing affects economic market prices in the lab and field. Proc. Natl. Acad. Sci. U.S.A. 110, 11779–11784 (2013). Medline doi:10.1073/pnas.1206326110

65. R. Hanson, Logarithmic market scoring rules for modular combinatorial information aggregation. J. Pred. Mark. 1, 3 (2007).

66. Y. Chen, “Markets as an information aggregation mechanism for decision support,” thesis, The Pennsylvania State University, State College, PA (2005).