Placebo Tests for Synthetic Controls

Ferman, Bruno and Pinto, Cristine (2017): Placebo Tests for Synthetic Controls.

Abstract

The synthetic control (SC) method has been recently proposed as an alternative to estimate treatment effects in comparative case studies. An important feature of the SC method is the inferential procedures based on placebo studies, suggested in Abadie et al. (2010). In this paper, we evaluate the statistical properties of these inferential techniques. We first show that the graphical analysis with placebos can be misleading, as placebo runs with lower expected squared prediction errors would still be considered in the analysis. Then we show that a test based on the the post/pre-intervention mean squared prediction error, as suggested in Abadie et al. (2010), ameliorates this problem. However, we show that such test can still have some size distortions, even if we consider a case in which the test statistic has the same marginal distribution for all placebo runs. Finally, we show that the fact that the SC weights are estimated can lead to important additional size distortions.

• Item Type: MPRA Paper
• Language: English
• Keywords: synthetic control, difference-in-differences; linear factor model, inference, permutation test
• Subjects: C - Mathematical and Quantitative Methods > C1 - Econometric and Statistical Methods and Methodology: General > C12 - Hypothesis Testing: General; C - Mathematical and Quantitative Methods > C1 - Econometric and Statistical Methods and Methodology: General > C13 - Estimation: General; C - Mathematical and Quantitative Methods > C2 - Single Equation Models ; Single Variables > C21 - Cross-Sectional Models ; Spatial Models ; Treatment Effect Models ; Quantile Regressions; C - Mathematical and Quantitative Methods > C2 - Single Equation Models ; Single Variables > C23 - Panel Data Models ; Spatio-temporal Models

References

Abadie, Alberto, Alexis Diamond, and Jens Hainmueller, “Synthetic Control Methods for Com- parative Case Studies: Estimating the Effect of California’s Tobacco Control Program,” Journal of the American Statiscal Association, 2010, 105 (490), 493–505.

Abadie, Alberto, Alexis Diamond, and Jens Hainmueller,, “Comparative Politics and the Synthetic Control Method,” American Journal of Political Science, 2015, 59 (2), 495–510.

Abadie, Alberto, and Javier Gardeazabal, “The Economic Costs of Conflict: A Case Study of the Basque Country,” American Economic Review, 2003, 93 (1), 113–132.

Ando, Michihito and Fredrik S¨avje, “Hypothesis Testing with the Synthetic Control Method,” 2013. Working Paper.

Athey, Susan and Guido Imbens, “The State of Applied Econometrics-Causality and Policy Evaluation,” arXiv preprint arXiv:1607.00699, 2016.

Canay, Ivan A., Joseph P. Romano, and Azeem M. Shaikh, “Randomization Tests under an Ap- proximate Symmetry Assumption?,” 2014.

Carvalho, Carlos V., Ricardo Mansini, and Marcelo C. Medeiros, “ArCo: An Artificial Counter- factual Approach for Aggregate Data,” February 2015. Working Paper.

Carvalho, Carlos V., Ricardo Mansini, and Marcelo C. Medeiros, “The Perils of Counterfactual Analysis with Integrated Processes,” December 2016. Working Paper.

Doudchenko, Nikolay and Guido Imbens, “Balancing, regression, difference-in-differences and synthetic control methods: A synthesis,” 2016.

Ferman, Bruno and Cristine Pinto, “Inference in Differences-in-Differences with Few Treated Groups and Heteroskedasticity,” 2016. Working Paper.

Ferman, Bruno and Cristine Pinto, “Revisiting the Synthetic Control Estimator,” 2016. Working Paper, Available at https://mpra.ub.uni-muenchen.de/77930/1/MPRA_paper_77930.pdf.

Ferman, Bruno, Cristine Pinto, and Vitor Possebom, “Cherry Picking with Synthetic Controls,” 2016. Working Paper.

Firpo, Sergio and Vitor Possebom, “Synthetic Control Estimator: A Generalized Inference Procedure and Confidence Sets,” April 2016.

Fisher, R.A., The design of experiments. 1935, Edinburgh: Oliver and Boyd, 1935.

Hahn, Jinyong and Ruoyao Shi, “Synthetic Control and Inference,” 2016.

Powell, David, “Synthetic Control Estimation Beyond Case Studies: Does the Minimum Wage Reduce Employment?,” 2016. RAND Corporation, WR-1142.