Inference in Differences-in-Differences with Few Treated Groups and Heteroskedasticity

Ferman, Bruno and Pinto, Cristine (2015): Inference in Differences-in-Differences with Few Treated Groups and Heteroskedasticity.

Abstract

Differences-in-Differences (DID) is one of the most widely used identification strategies in applied economics. However, inference in DID models when there are few treated groups is still an open question. We show that usual inference methods used in DID models might not perform well when there are few treated groups and residuals are heteroskedastic. In particular, when there is variation in the number of observations per group, we show that inference methods designed to work when there are few treated groups would tend to (under-) over-reject the null hypothesis when the treated groups are (large) small relative to the control groups. This happens because larger groups would have lower variance, generating heteroskedasticity in the group x time aggregate DID model. We provide evidence from Monte Carlo simulations and from placebo DID regressions with the American Community Survey (ACS) dataset to show that this problem is relevant even in datasets with large number of observations per group. Then we derive alternative inference methods that provide accurate hypothesis testing in situations of few treated groups and many control groups in the presence of heteroskedasticity (including the case of only one treated group). The main assumption is that we know how the heteroskedasticity is generated, which is the case when it is generated by variation in the number of observations per group. Finally, we also show that an inference method for the Synthetic Control Estimator proposed by Abadie et al. (2010) can correct for the heteroskedasticity problem, and derive conditions under which this inference method provides accurate hypothesis testing.

• Item Type: MPRA Paper
• Language: English
• Keywords: differences-in-differences; inference; heteroskedasticity; clustering; few clusters; bootstrap
• Subjects: C - Mathematical and Quantitative Methods > C1 - Econometric and Statistical Methods and Methodology: General > C12 - Hypothesis Testing: General; C - Mathematical and Quantitative Methods > C2 - Single Equation Models ; Single Variables > C21 - Cross-Sectional Models ; Spatial Models ; Treatment Effect Models ; Quantile Regressions; C - Mathematical and Quantitative Methods > C3 - Multiple or Simultaneous Equation Models ; Multiple Variables > C33 - Panel Data Models ; Spatio-temporal Models

References

Abadie, Alberto, Alexis Diamond, and Jens Hainmueller, “Synthetic Control Methods for Com- parative Case Studies: Estimating the Effect of Californias Tobacco Control Program,” Journal of the American Statistical Association, 2010, 105 (490), 493–505.

Abadie, Alberto and Javier Gardeazabal, “The Economic Costs of Conflict: A Case Study of the Basque Country,” American Economic Review, March 2003, 93 (1), 113–132.

Angrist, J.D. and J.S. Pischke, Mostly Harmless Econometrics: An Empiricist’s Companion, Princeton University Press, 2009.

Assuncao, J. and B. Ferman, “Does affirmative action enhance or under- cut investment incentives? Evidence from quotas in Brazilian Public Universi- ties,” Unpublished Manuscript, February 2015, Can be found (as of Feb. 2015), at https://dl.dropboxusercontent.com/u/12654869/Assuncao%20and%20Ferman022015.pdf.

Bedard, Kelly and Chau Do, “Are Middle Schools More Effective?: The Impact of School Structure on Student Outcomes,” Journal of Human Resources, 2005, 40 (3).

Bell, R. M. and D. F. McCaffrey, “Bias Reduction in Standard Errors for Linear Regression with Multi-Stage Samples,” Survey Methodology, 2002, 28 (2), 169–181.

Bertrand, Marianne, Esther Duflo, and Sendhil Mullainathan, “How Much Should We Trust Differences-in-Differences Estimates?,” Quarterly Journal of Economics, 2004, p. 24975.

Brewer, Mike, Thomas F. Crossley, and Robert Joyce, “Inference with Difference-in-Differences Revisited,” IZA Discussion Papers 7742, Institute for the Study of Labor (IZA) November 2013.

Cameron, A Colin and Douglas L Miller, “A practitioners guide to cluster-robust inference,” Journal of Human Resources, 2015, 50 (2), 317–372.

Cameron, A.C., J.B. Gelbach, and D.L. Miller, “Bootstrap-based improvements for inference with clustered errors,” The Review of Economics and Statistics, 2008, 90 (3), 414–427.

Choi, Moonkyung Kate, “The impact of Medicaid insurance coverage on dental service use,” Journal of Health Economics, 2011, 30 (5), 1020–1031.

Conley, Timothy G. and Christopher R. Taber, “Inference with “Difference in Differences with a Small Number of Policy Changes,” The Review of Economics and Statistics, February 2011, 93 (1), 113–125.

Davison, A.C. and D.V. Hinkley, Bootstrap Methods and Their Application Cambridge Series in Statis- tical and Probabilistic Mathematics, Cambridge University Press, 1997.

Donald, Stephen G. and Kevin Lang, “Inference with Difference-in-Differences and Other Panel Data,” The Review of Economics and Statistics, May 2007, 89 (2), 221–233.

Ibragimov, Rustam and Ulrich K. Mller, “Inference with Few Heterogenous Clusters,” 2013.

Imbens, Guido W. and Michal Kolesar, “Robust Standard Errors in Small Samples: Some Practical Advice,” Working Paper 18478, National Bureau of Economic Research October 2012.

Liang, KUNG-YEE and SCOTT L. Zeger, “Longitudinal data analysis using generalized linear mod- els,” Biometrika, 1986, 73 (1), 13–22.

MacKinnon, James G. and Matthew D. Webb, “Differences-in-Differences Inference with Few Treated Clusters,” 2015.

MacKinnon, James G. and Matthew D. Webb, “Wild Bootstrap Inference for Wildly Different Cluster Sizes,” Working Papers 1314, Queen’s University, Department of Economics February 2015.

Pettersson-Lidbom, Per, “Does the size of the legislature affect the size of government? Evidence from two natural experiments,” Journal of Public Economics, 2012, 96 (3), 269–278.

van der Vaart, A. W., Asymptotic statistics Cambridge series in statistical and probabilistic mathematics, Cambridge (UK), New York (N.Y.): Cambridge University Press, 1998. Autre tirage : 2000 (dition broche), 2005, 2006, 2007.

White, Halbert, “A Heteroskedasticity-Consistent Covariance Matrix Estimator and a Direct Test for Heteroskedasticity,” Econometrica, May 1980, 48 (4), 817–838.

Wooldridge, Jeffrey M., “Cluster-Sample Methods in Applied Econometrics,” American Economic Re- view, 2003, 93 (2), 133–138.