A note on the impact evaluation of public policies: the counterfactual analysis

Loi, Massimo and Rodrigues, Margarida (2012): A note on the impact evaluation of public policies: the counterfactual analysis. Published in:

Abstract

This report describes concisely, and in an intuitive way, the policy evaluation framework and the different counterfactual analysis evaluation strategies: propensity score matching, regression discontinuity design, differences-in-differences and instrumental variables. For each method we present the main assumptions it relies on and the data requirements. These methodologies apply to any type of policy and, in general, to any type of intervention. A selection of papers applying this approach in the context of labour market interventions is also included.

• Item Type: MPRA Paper
• Language: English
• Keywords: counterfactual analysis, propensity score matching, regression discontinuity, difference in difference, instrumental variables
• Subjects: C - Mathematical and Quantitative Methods > C1 - Econometric and Statistical Methods and Methodology: General

References

Abadie, A., Angrist, J. & Imbens, G.. (2002). Instrumental variables estimates of the effect of subsidized training on the quantiles of trainee earnings. Econometrica, vol. 70 (1), 91-117. Angrist, J., Imbens, G. & Rubin, D., (1996). Identification of Causal Effects Using Instrumental Variables, Journal of the American Statistical Association, vol. 91: 444-472. Angrist, J. & Pischke, J.S., (2009). Mostly harmless econometrics. An empiricist’s companion. Princeton University Press. Barnow, B.S., Cain, G.C. & Goldberger, A.S., (1980). Issues in the analysis of selectivity bias. In Stromsdorfer, E., Farkas (Eds.), Education studies (vol. 5, pp. 42-59). San Francisco: Sage. Battistin, E. & Rettore, E., (2008). Ineligibles and eligible non-participants as a double comparison group in regression discontinuity designs. Journal of Econometrics, 142: 715-730. Bell, B., Blundell, R. & Van Reenen, J., (1999). Getting the unemployed back to work: the role of wage subsidies. International Tax and Public Finance. Vol. 6 (3): 339-360. Bloom, H., (1984). Accounting for no-shows in experimental evaluation designs. Evaluation review, 8: 225-246. Blundell, R. & Costa Dias, M., (2008). Alternative approaches to evaluation in empirical microeconometrics. Bonn, IZA: IZA DP No. 3800. Blundell, R. & Costa Dias, M., (2002). Alternative Approaches to Evaluation in Empirical Microeconomic, Portuguese Economic Journal, vol. 1: 91-115. Caliendo, M., Hujer, R. & Thomsen, S., (2005). The employment effects of job creation schemes in Germany. A microeconomic evaluation. Bonn, IZA: IZA DP No. 1512. Caliendo, M. & Kopeinig, S., (2008). Some practical guidance for the implementation of propensity score matching. Journal of Economic Surveys, 22(1): 31-72. Card, D., Ibarrarán, P. & Villa, J.M., (2011). Building in an evaluation component for active labor market programs: a practitioner’s guide. Bonn, IZA: IZA DP No. 6085. Crépon, B., Gurgand, M., Rathelot, R. & Zamora, P., (2011). Do Labor Market Policies have Displacement Effect? Evidence from a Clustered Randomized Experiment. Guo, S. & Fraser, M.W., (2010). Propensity score analysis. Statistical methods and applications. London: SAGE publications. Heckman, J., Ichimura, H. & Todd, P., (1997). Matching as an econometric evaluation estimator: Evidence from evaluating a job training program. The Review of Economics Studies, vol. 64 (4): 605-654. Hujer R. & Caliendo, M., (2000). Evaluation of active labour market policy: methodological concepts and empirical estimates. Bonn, IZA: IZA DP No. 236. Imbens, G.W., (2004). Nonparametric estimation of average treatment effects under exogeneity: a Review. Review of Economics and Statistics, 86: 4-29. Imbens, G. & Angrist, J., (1994). Identification and estimation of local average treatment effects. Econometrica, vol. 62 (2), 467-475. Khandker, S., Koolwal, G. & Samad, H., (2010). Handbook in Impact Evaluation: Quantitative Methods and Practices. Washington D.C.: The World Bank. Lechner, M., (1999). Earnings and employment effects of continuous off-the-job training in East Germany after unification. Journal of Business and Economic Statistics, 17: 74-90. Lechner, M., (2001). The Estimation of causal effects by differences-in-differences methods. Discussion paper 2010-28, Department of Economics, University of St. Gallen Lechner, M., (2002). Some practical issues in the evaluation of heterogeneous labour market programmes by matching methods. Journal of the Royal Statistical Society, 165: 59-82. Martini, A. & Trivellato, U., (2011). Sono soldi ben spesi? Perché e come valutare l’efficacia delle politiche pubbliche. Venezia: Marsilio Editore Martini, A., (2008). How counterfactuals got lost on the way to Brussels. Prepared for the Symposium “Policy and programme evaluation in Europe: cultures and prospects”, Strasbourg. Rosenbaum P. & Rubin D., (1983). The Central Role of the Propensity Score in Observational Studies for Causal Effects. Biometrika, 70: 41-50. Rosenbaum, P.R. & Rubin, D.B., (1985). Constructing a control group using multivariate matched sampling methods that incorporate the propensity score. The American Statistician, 39: 33–38. Rossi, P.H., Lipsey, M.W. & Freeman, H.E., (2004). Evaluation. A systematic approach (7th edition). Sage Publications, Inc. Serrano-Valerede, N., (2008). The Financing Structure of Corporate R&D - Evidence from Regression Discontinuity Design. Working Paper, European University Institute, Department of Economics, Firenze. Schochet, P., Burghardt, J. & McConnell, S., (2008). Does Job Corps work? Impact findings from the National Job Corps study. American Economic Review, 98(5): 1864-1886. Sianesi, B., (2004). An Evaluation of the Swedish System of Active Labor Market Programs in the 1990s. Review of Economics and Statistics, 86(1): 133-155. Shadish, W.R., Cook, T.D. & Campbell, D.T., (2002). Experimental and quasi-experimental designs for generalized causal inference. Boston: Houghton Mifflin. Smith, J. & Todd, P., (2005). Does matching overcome LaLonde’s critique of nonexperimental estimators? Journal of Econometrics, vol. 125: 305-353. Stock, J. & Yogo, M., (2005). Testing for weak instruments in linear IV regression. In Identification and Inference for Econometrics Models: Essays in Honor of Thomas Rothenberg, ed. D. W. K. Andrews and J. H. Stock, 80-108. Cambridge: Cambridge University Press.