Design Flaw of the Synthetic Control Method

Kuosmanen, Timo and Zhou, Xun and Eskelinen, Juha and Malo, Pekka (2021): Design Flaw of the Synthetic Control Method.

Abstract

Synthetic control method (SCM) identifies causal treatment effects by constructing a counterfactual treatment unit as a convex combination of donors in the control group, such that the weights of donors and predictors are jointly optimized during the pre-treatment period. This paper demonstrates that the true optimal solution to the SCM problem is typically a corner solution where all weight is assigned to a single predictor, contradicting the intended purpose of predictors. To address this inherent design flaw, we propose to determine the predictor weights and donor weights separately. We show how the donor weights can be optimized when the predictor weights are given, and consider alternative data-driven approaches to determine the predictor weights. Re-examination of the two original empirical applications to Basque terrorism and California's tobacco control program demonstrates the complete and utter failure of the existing SCM algorithms and illustrates our proposed remedies.

• Item Type: MPRA Paper
• Language: English
• Keywords: Causal effects; Comparative case studies; Policy impact assessment; Treatment effect models
• Subjects: C - Mathematical and Quantitative Methods > C5 - Econometric Modeling > C54 - Quantitative Policy Modeling; C - Mathematical and Quantitative Methods > C6 - Mathematical Methods ; Programming Models ; Mathematical and Simulation Modeling > C61 - Optimization Techniques ; Programming Models ; Dynamic Analysis; C - Mathematical and Quantitative Methods > C7 - Game Theory and Bargaining Theory > C71 - Cooperative Games

References

Abadie, A. (2019). Using Synthetic Controls: Feasibility, Data Requirements, and Methodological Aspects. Journal of Economic Literature, forthcoming.

Abadie, A., Diamond, A., & Hainmueller, J. (2010). Synthetic Control Methods for Comparative Case Studies: Estimating the Effect of California's Tobacco Control Program. Journal of the American Statistical Association, 105 (490), 493-505.

Abadie, A., Diamond, A., & Hainmueller, J. (2011). Synth: An R package for synthetic control methods in comparative case studies. Journal of Statistical Software, 42(13), 1-17.

Abadie, A., Diamond, A., & Hainmueller, J. (2015). Comparative Politics and the Synthetic Control Method. American Journal of Political Science, 59(2), 495-510.

Abadie, A., & Gardeazabal, J. (2003). The Economic Costs of Conflict: A Case Study of the Basque Country. American Economic Review, 93 (1), 113-132.

Abadie, A., & L'Hour, J. (2020). A Penalized Synthetic Control Estimator for Disaggregated Data. Work. Pap., Mass. Inst. Technol., Cambridge, MA.

Acemoglu, D., Johnson, S., Kermani, A., Kwak, J., & Mitton, T. (2016). The value of connections in turbulent times: Evidence from the United States. Journal of Financial Economics, 121(2), 368-391.

Amjad, M., Shah, D., & Shen, D. (2018). Robust synthetic control. Journal of Machine Learning Research, 19(1), 802-852.

Arkhangelsky, D., Athey, S., Hirshberg, D. A., Imbens, G. W., & Wager, S. (2018). Synthetic Difference in Differences. arXiv preprint arXiv:1812.09970.

Athey, S., & Imbens, G. W. (2017). The state of applied econometrics: Causality and policy evaluation. Journal of Economic Perspectives, 31(2), 3-32.

Becker, M., & Klössner, S. (2017). Estimating the economic costs of organized crime by synthetic control methods. Journal of Applied Econometrics, 32(7), 1367-1369.

Becker, M., & Klössner, S. (2018). Fast and reliable computation of generalized synthetic controls. Econometrics and Statistics, 5, 1-19.

Becker, M., Klössner, S., & Pfeifer, G. (2018). Cross-validating synthetic controls. Economics Bulletin, 38 (1), 603-609.

Ben-Michael, E., Feller, A., & Rothstein, J. (2018). The Augmented Synthetic Control Method. arXiv preprint arXiv:1811.04170.

Bloom, N., & Van Reenen, J. (2007). Measuring and Explaining Management Practices Across Firms and Countries. The Quarterly Journal of Economics, 122 (4), 1351- 1408.

Bohn, S., Lofstrom, M., & Raphael, S. (2014). Did the 2007 legal Arizona workers act reduce the state's unauthorized immigrant population? Review of Economics and Statistics, 96(2), 258-269.

Cavallo, E., Galiani, S., Noy, I., & Pantano, J. (2013). Catastrophic natural disasters and economic growth. Review of Economics and Statistics, 95(5), 1549-1561.

Cherchye, L., Moesen, W., Rogge, N., & Puyenbroeck, T. V. (2007). An introduction to 'benefit of the doubt' composite indicators. Social Indicators Research, 82 (1), 111-145.

Chernozhukov, V., Wuthrich, K, & Zhu, Y. (2020). An Exact and Robust Conformal Inference Method for Counterfactual and Synthetic Controls. arXiv preprint arXiv:1712.09089.

Cole, M. A., Elliott, R. J., & Liu, B. (2020). The Impact of the Wuhan Covid-19 Lockdown on Air Pollution and Health: A Machine Learning and Augmented Synthetic Control Approach. Environmental and Resource Economics, 76(4), 553-580.

Doudchenko, N., & Imbens, G. W. (2017). Balancing, Regression, Difference-In-Differences and Synthetic Control Methods: A Synthesis. arXiv preprint arXiv:1610.07748.

Ferman, B., Pinto, C., & Possebom, V. (2018). Cherry Picking with Synthetic Controls. MPRA Paper 85138.

Gobillon, L., & Magnac, T. (2016). Regional policy evaluation: Interactive fixed effects and synthetic controls. Review of Economics and Statistics, 98(3), 535-551.

Hansen, P., Jaumard, B., & Savard, G. (1992). New Branch-and-Bound Rules for Linear Bilevel Programming. SIAM Journal on Scientific and Statistical Computing, 13 (5), 1194-1217.

Kaul, A., Klössner, S., Pfeifer, G., & Schieler, M. (2015). Synthetic Control Methods: Never Use All Pre-Intervention Outcomes Together With Covariates. MPRA Paper 83790.

Kleven, H. J., Landais, C., & Saez, E. (2013). Taxation and international migration of superstars: Evidence from the European football market. American Economic Review, 103(5), 1892-1924.

Klössner, S., Kaul, A., Pfeifer, G., & Schieler, M. (2018). Comparative politics and the synthetic control method revisited: A note on Abadie et al. (2015). Swiss Journal of Economics and Statistics, 154(1), 11.

Malo, P., Eskelinen, J., Zhou, X., & Kuosmanen, T. (2020). Computing Synthetic Controls Using Bilevel Optimization. MPRA Paper 104085.

Powell, D. (2018). Imperfect Synthetic Controls: Did the Massachusetts Health Care Reform Save Lives? SSRN Electronic Journal.

Sinha, A., Malo, P., & Deb, K. (2013). Efficient Evolutionary Algorithm for Single-Objective Bilevel Optimization. arXiv preprint arXiv:1303.3901.

Tardella, F. (2011). The fundamental theorem of linear programming: Extensions and applicationsy. Optimization, 60 (1-2), 283-301.

Vicente, L., Savard, G., & Judice, J. (1994). Descent approaches for quadratic bilevel programming. Journal of Optimization Theory and Applications, 81(2), 379-399.

Xu, Y. (2017). Generalized synthetic control method: Causal inference with interactive fixed effects models. Political Analysis, 25(1), 57-76.