Cherry Picking with Synthetic Controls

Ferman, Bruno and Pinto, Cristine and Possebom, Vitor (2018): Cherry Picking with Synthetic Controls.

Abstract

We evaluate whether a lack of guidance on how to choose the matching variables used in the Synthetic Control (SC) estimator creates specification-searching opportunities. We first provide theoretical results showing that specification-searching opportunities would be asymptotically irrelevant when the number of pre-treatment periods goes to infinity when we restrict to a subset of SC specifications. However, based on Monte Carlo simulations and simulations with real datasets, we show significant room for specification searching when the number of pre-treatment periods is finite and when alternative specifications commonly used in SC applications are also considered. This undermines one of the potential advantages of the method, which is providing a transparent way of choosing comparison units and, therefore, being less susceptible to specification searching than alternative methods. To address this problem, we provide recommendations to limit the possibilities for specification searching in the SC method. Finally, we analyze the possibilities for specification searching and our recommendations in two empirical applications.

• Item Type: MPRA Paper
• Language: English
• Keywords: inference; synthetic control; p-hacking; specification searching
• Subjects: C - Mathematical and Quantitative Methods > C1 - Econometric and Statistical Methods and Methodology: General > C12 - Hypothesis Testing: General; C - Mathematical and Quantitative Methods > C2 - Single Equation Models ; Single Variables > C21 - Cross-Sectional Models ; Spatial Models ; Treatment Effect Models ; Quantile Regressions; C - Mathematical and Quantitative Methods > C3 - Multiple or Simultaneous Equation Models ; Multiple Variables > C33 - Panel Data Models ; Spatio-temporal Models

References

Abadie, A., Diamond, A. & Hainmueller, J. (2010), ‘Synthetic Control Methods for Compara- tive Case Studies: Estimating the Effect of California’s Tobacco Control Program’, Journal of the American Statiscal Association 105(490), 493–505.

Abadie, A., Diamond, A. & Hainmueller, J. (2011), ‘Synth: An R Package for Synthetic Control Methods in Comparative Case Studies’, Journal of Statistical Software 42(13), 1– 17.

Abadie, A., Diamond, A. & Hainmueller, J. (2015), ‘Comparative Politics and the Synthetic Control Method’, American Journal of Political Science 59(2), 495–510.

Abadie, A. & Gardeazabal, J. (2003), ‘The Economic Costs of Conflict: A Case Study of the Basque Country’, American Economic Review 93(1), 113–132.

Acemoglu, D., Johnson, S., Kermani, A., Kwak, J. & Mitton, T. (2013), The Value of Connec- tions in Turbulent Times: Evidence from the United States. NBER Working Paper 19701. Available at: http://www.nber.org/papers/w19701.pdf.

Adhikari, B. & Alm, J. (2016), ‘Evaluating the economic effects of flat tax reforms using synthetic control methods’, Southern Economic Journal 83(2), 437–463.URL: http://dx.doi.org/10.1002/soej.12152

Ando, M. (2015), ‘Dreams of Urbanization: Quantitative Case Studies on the Local Impacts of Nuclear Power Facilities using the Synthetic Control Method’, Journal of Urban Economics 85, 68–85.

Athey, S. & Imbens, G. W. (2016), ‘The state of applied econometrics - causality and policy evaluation’, mimeo .

Barone, G. & Mocetti, S. (2014), ‘Natural Disasters, Growth and Institutions: a Tale of Two Earthquakes’, Journal of Urban Economics pp. 52–66.

Bauhoff, S. (2014), ‘The Effect of School Nutrition Policies on Dietary Intake and Overweight: a Synthetic Control Approach’, Economics and Human Biology pp. 45–55.

Belot, M. & Vandenberghe, V. (2014), ‘Evaluating the Threat Effects of Grade Repetition: Exploiting the 2001 Reform by the French-Speaking Community of Belgium’, Education Economics 22(1), 73–89.

Bertrand, M., Duflo, E. & Mullainathan, S. (2004), ‘How much should we trust differences- in-differences estimates?’, Quarterly Journal of Economics p. 24975.

Billmeier, A. & Nannicini, T. (2009), ‘Trade Openness and Growth: Pursuing Empirical Glasnost’, IMF Staff Papers 56(3), 447–475.

Billmeier, A. & Nannicini, T. (2013), ‘Assessing Economic Liberalization Episodes: A Syn- thetic Control Approach’, The Review of Economics and Statistics 95(3), 983–1001.

Bohn, S., Lofstrom, M. & Raphael, S. (2014), ‘Did the 2007 Legal Arizona Workers Act Reduce the State’s Unauthorized Immigrant Population?’, The Review of Economics and Statistics 96(2), 258–269.

Botosaru, I. & Ferman, B. (2017), On the Role of Covariates in the Synthetic Control Method, MPRA Paper 80796, University Library of Munich, Germany.

Bove, V., Elia, L. & Smith, R. P. (2014), The Relationship between Panel and Synthetic Control Estimators on the Effect of Civil War. Working Paper, http://www.bbk.ac.uk/ ems/research/BirkCAM/working-papers/BCAM1406.pdf.

Brodeur, A., L´e, M., Sangnier, M. & Zylberberg, Y. (2016), ‘Star Wars: The Empirics Strike Back’, American Economic Journal: Applied Economics 8(1), 1–32.

Calderon, G. (2014), The Effects of Child Care Provision in Mexico. Working paper, http://goo.gl/YSEs9B.

Carrasco, V., de Mello, J. M. P. & Duarte, I. (2014), A D´ecada Perdida: 2003 – 2012. Texto para Discussa˜o, http://www.econ.puc-rio.br/uploads/adm/trabalhos/files/td626. pdf.

Cavallo, E., Galiani, S., Noy, I. & Pantano, J. (2013), ‘Catastrophic Natural Disasters and Economic Growth’, The Review of Economics and Statistics 95(5), 1549–1561.

Chan, H. F., Frey, B. S., Gallus, J. & Torgler, B. (2014), ‘Academic Honors and Performance’, Labour Economics 31, 188–204.

Christensen, G. & Miguel, E. (2016), Transparency, reproducibility, and the credibility of economics research, Technical report.

Coffman, L. C. & Niederle, M. (2015), ‘Pre-analysis plans have limited upside, especially where replications are feasible’, Journal of Economic Perspectives 29(3), 81–98. URL: http://www.aeaweb.org/articles.php?doi=10.1257/jep.29.3.81

Coffman, M. & Noy, I. (2011), ‘Hurricane Iniki: Measuring the Long-Term Economic Impact of Natural Disaster Using Synthetic Control’, Environment and Development Economics 17, 187–205.

Cohen-Cole, E., Durlauf, S., Fagan, J. & Nagin, D. (2009), ‘Model Uncertainty and the Deterrent Effect of Capital Punishment’, American Law and Economics Review 11(2), 335– 369.

De Long, J. B. & Lang, K. (1992), ‘Are all economic hypotheses false?’, Journal of Political Economy pp. 1257–1272.

de Souza, F. F. A. (2014), Tax Evasion and Inflation: Evidence from the Nota Fiscal Paulista Program, Master’s thesis, Pontif´ıcia Universidade Cat´olica. Available at http://www.dbd. puc-rio.br/pergamum/tesesabertas/1212327_2014_completo.pdf.

Dhungana, S. (2011), Identifying and Evaluating Large Scale Policy Interventions: What Questions Can We Answer? Available at: https://openknowledge.worldbank.org/ bitstream/handle/10986/3688/WPS5918.pdf?sequence=1.

Dube, A. & Zipperer, B. (2015), Pooling Multiple Case Studies Using Synthetic Controls: An Application to Minimum Wage Policies, IZA Discussion Papers 8944, Institute for the Study of Labor (IZA). URL: https://ideas.repec.org/p/iza/izadps/dp8944.html

DuPont, W. & Noy, I. (2012), What Happened to Kobe? A Reassessment of the Impact of the 1995 Earthquake in Japan. Available at: http://www.economics.hawaii.edu/research/ workingpapers/WP_12-4.pdf.

Ferman, B. & Pinto, C. (2016), Revisiting the synthetic control estimator. Ferman, B. & Pinto, C. (2017), Placebo Tests for Synthetic Controls.

Ferman, B. & Pinto, C. (2017), Placebo Tests for Synthetic Controls, MPRA Paper 78079, University Library of Munich, Germany.

Firpo, S. & Possebom, V. (2016), Synthetic Control Estimator: A Generalized Inference Procedure and Confidence Sets. Working Paper, https://goo.gl/oQTX9c.

Gardeazabal, J. & Vega-Bayo, A. (2016), ‘An empirical comparison between the synthetic control method and hsiao et al.’s panel data approach to program evaluation’, Journal of Applied Econometrics pp. n/a–n/a. jae.2557. URL: http://dx.doi.org/10.1002/jae.2557

Gathani, S., Santini, M. & Stoelinga, D. (2013), Innovative Techniques to Evaluate the Im- pacts of Private Sector Developments Reforms: An Application to Rwanda and 11 other Countries. Working Paper, https://blogs.worldbank.org/impactevaluations/files/ impactevaluations/methods_for_impact_evaluations_feb06-final.pdf.

Gobillon, L. & Magnac, T. (2016), ‘Regional Policy Evaluation: Interative Fixed Effects and Synthetic Controls’, Review of Economics and Statistics . Forthcoming.

Hahn, J. & Shi, R. (2016), `Synthetic Control and Inference'.

Hinrichs, P. (2012), ‘The Effects of Affirmative Action Bans on College Enrollment, Educa- tional Attainment, and the Demographic Composition of Universities’, Review of Economics and Statistics 94(3), 712–722.

Hosny, A. S. (2012), ‘Algeria’s Trade with GAFTA Countries: A Synthetic Control Approach’, Transition Studies Review 19, 35–42.

Hsiao, C., Steve Ching, H. & Ki Wan, S. (2012), ‘A panel data approach for program eval- uation: Measuring the benefits of political and economic integration of hong kong with mainland china’, Journal of Applied Econometrics 27(5), 705–740. URL: http://dx.doi.org/10.1002/jae.1230

Imbens, G. W. & Rubin, D. B. (2015), Causal Inference for Statistics, Social and Biomedical Sciences: An Introduction, 1st edn, Cambridge University Press, United Kingdom.

Jinjarak, Y., Noy, I. & Zheng, H. (2013), ‘Capital Controls in Brazil — Stemming a Tide with a Signal?’, Journal of Banking & Finance 37, 2938–2952.

Kaul, A., Klo¨bner, S., Pfeifer, G. & Schieler, M. (2015), Synthetic Control Methods: Never Use All Pre-Intervention Outcomes as Economic Predictors. Working Paper. Available at: http://www.oekonometrie.uni-saarland.de/papers/SCM_Predictors.pdf.

Kirkpatrick, A. J. & Bennear, L. S. (2014), ‘Promoting Clean Enery Investment: an Empirical Analysis of Property Assessed Clean Energy’, Journal of Environmental Economics and Management 68, 357–375.

Kleven, H. J., Landais, C. & Saez, E. (2013), ‘Taxation and International Migration of Superstars: Evidence from European Football Market’, American Economic Review 103(5), 1892–1924.

Klo¨bner, S., Kaul, A., Pfeifer, G. & Schieler, M. (2016), Comparative Politics and the Syn- thetic Control Method Reviseted: A Note on Abadie et al. (2015).

Kreif, N., Grieve, R., Hangartner, D., Turner, A. J., Nikolova, S. & Sutton, M. (2015), ‘Examination of the Synthetic Control Method for Evaluating Health Policies with Multiple Treated Units’, Health Economics .

Li, Q. (2012), ‘Economics Consequences of Civil Wars in the Post-World War II Period’, The Macrotheme Review 1(1), 50–60.

Liu, S. (2015), ‘Spillovers from Universities: Evidence from the Land-Grant Program’, Journal of Urban Economics 87, 25–41.

Lovell, M. (1983), ‘Data Mining’, The Review of Economics and Statistics 65(1), 1–12. Mideksa, T. K. (2013), ‘The Economic Impact of Natural Resources’, Journal of Environmental Economics and Management 65, 277–289.

Miguel, E., Camerer, C., Casey, K., Cohen, J., Esterling, K. M., Gerber, A., Glennerster, R., Green, D. P., Humphreys, M., Imbens, G., Laitin, D., Madon, T., Nelson, L., Nosek, B. A., Petersen, M., Sedlmayr, R., Simmons, J. P., Simonsohn, U. & Van der Laan, M. (2014), ‘Promoting transparency in social science research’, Science 343(6166), 30–31. URL: http://science.sciencemag.org/content/343/6166/30

Montalvo, J. G. (2011), ‘Voting after the Bombings: A Natural Experiment on the Ef- fect of Terrorist Attacks on Democratic Elections’, Review of Economics and Statistics 93(4), 1146–1154.

Olken, B. A. (2015), ‘Promises and perils of pre-analysis plans’, Journal of Economic Per- spectives 29(3), 61–80. URL: http://www.aeaweb.org/articles.php?doi=10.1257/jep.29.3.61

Pinotti, P. (2012a), Organized Crime, Violence and the Quality of Politicians: Evidence from Southern Italy. Available at: http://dx.doi.org/10.2139/ssrn.2144121.

Pinotti, P. (2012b), The Economic Costs of Organized Crime: Evidence from Southern Italy. Temi di Discussione (Working Papers), http://www.bancaditalia.it/pubblicazioni/ temi-discussione/2012/2012-0868/en_tema_868.pdf.

Possebom, V. (2017), `Free Trade Zone of Manaus: An Impact Evaluation using the Synthetic Control Method', Revista Brasileira de Economia 71(2), 217-231.

Ribeiro, F., Stein, G. & Kang, T. (2013), The Cuban Experiment: Measuring the Role of the 1959 Revolution on Economic Performance using Synthetic Control. Available at: http://economics.ca/2013/papers/SG0030-1.pdf.

Romano, J. P. & Wolf, M. (2005), ‘Stepwise multiple testing as formalized data snooping’, Econometrica 73(4), 1237–1282.

Rosenthal, R. (1979), ‘The file drawer problem and tolerance for null results.’, Psychological bulletin 86(3), 638.

Sanso-Navarro, M. (2011), ‘The effects on American Foreign Direct Investment in the United Kingdom from Not Adopting the Euro’, Journal of Common Markets Studies 49(2), 463– 483.

Saunders, J., Lundberg, R., Braga, A. A., Ridgeway, G. & Miles, J. (2014), ‘A Synthetic Control Approach to Evaluating Place-Based Crime Interventions’, Journal of Quantitative Criminology .

Severnini, E. R. (2014), The Power of Hydroelectric Dams: Agglomeration Spillovers. IZA Discussion Paper, No. 8082, http://ftp.iza.org/dp8082.pdf.

Sills, E. O., Herrera, D., Kirkpatrick, A. J., Brandao, A., Dickson, R., Hall, S., Pattanayak, S., Shoch, D., Vedoveto, M., Young, L. & Pfaff, A. (2015), ‘Estimating the Impact of a Local Policy Innovation: The Synthetic Control Method Applied to Tropica Desforestation’, PLOS One .

Simmons, J. P., Nelson, L. D. & Simonsohn, U. (2011), ‘False-positive psychology undis- closed flexibility in data collection and analysis allows presenting anything as significant’, Psychological science p. 0956797611417632.

Simonsohn, U., Nelson, L. D. & Simmons, J. P. (2014), ‘P-curve: A key to the file-drawer.’, Journal of Experimental Psychology: General 143(2), 534–547. URL: http://dx.doi.org/10.1037/a0033242

Smith, B. (2015), ‘The Resource Curse Exorcised: Evidence from a Panel of Countries’, Journal of Development Economics 116, 57–73.

White, H. (2000), ‘A reality check for data snooping’, Econometrica 68(5), 1097–1126. URL: http://dx.doi.org/10.1111/1468-0262.00152

Yu, J. & Wang, C. (2013), ‘Political Risk and Economic Development: A Case Study of China’, Eknomska Istrazianja - Economic Research 26(2), 35–50.