Inference in Differences-in-Differences with Few Treated Groups and Heteroskedasticity

Ferman, Bruno and Pinto, Cristine (2015): Inference in Differences-in-Differences with Few Treated Groups and Heteroskedasticity.

Abstract

Differences-in-Differences (DID) is one of the most widely used identification strategies in applied economics. However, inference in DID models when there are few treated groups remains an open question. We show that the usual inference methods used in DID models might not perform well when there are few treated groups and errors are heteroskedastic. In particular, we show that when there is variation in the number of observations per group, inference methods designed to work when there are few treated groups tend to (under-) over-reject the null hypothesis when the treated groups are (large) small relative to the control groups. This happens because larger groups tend to have lower variance, generating heteroskedasticity in the group x time aggregate DID model. We provide evidence from Monte Carlo simulations and from placebo DID regressions with the American Community Survey (ACS) and the Current Population Survey (CPS) datasets to show that this problem is relevant even in datasets with large numbers of observations per group. We then derive an alternative inference method that provides accurate hypothesis testing in situations where there are few treated groups (or even just one) and many control groups in the presence of heteroskedasticity. Our method assumes that we know how the heteroskedasticity is generated, which is the case when it is generated by variation in the number of observations per group. We only need to know the structure of the heteroskedasticity of a linear combination of the errors, which implies that we do not need strong assumptions on the intra-group and serial correlation structure of the errors. Our method provided accurate hypothesis testing with one treated and 24 control groups in simulations with real datasets. Finally, we also show that an inference method for the Synthetic Control Estimator proposed by Abadie et al. (2010) can correct for the heteroskedasticity problem, and derive conditions under which this inference method provides accurate hypothesis testing.

• Item Type: MPRA Paper
• Language: English
• Keywords: differences-in-differences; inference; heteroskedasticity; clustering; few clusters; bootstrap; synthetic control
• Subjects: C - Mathematical and Quantitative Methods > C1 - Econometric and Statistical Methods and Methodology: General > C12 - Hypothesis Testing: General; C - Mathematical and Quantitative Methods > C2 - Single Equation Models ; Single Variables > C21 - Cross-Sectional Models ; Spatial Models ; Treatment Effect Models ; Quantile Regressions; C - Mathematical and Quantitative Methods > C3 - Multiple or Simultaneous Equation Models ; Multiple Variables > C33 - Panel Data Models ; Spatio-temporal Models

References

Abadie, Alberto, Alexis Diamond, and Jens Hainmueller, “Synthetic Control Methods for Comparative Case Studies: Estimating the Effect of Californias Tobacco Control Program,” Journal of the American Statistical Association, 2010, 105 (490), 493–505.

Abadie, Alberto and Javier Gardeazabal, “The Economic Costs of Conflict: A Case Study of the Basque Country,” American Economic Review, March 2003, 93 (1), 113–132.

Angrist, J.D. and J.S. Pischke, Mostly Harmless Econometrics: An Empiricist’s Companion, Princeton University Press, 2009.

Assuncao, J. and B. Ferman, “Does affirmative action enhance or undercut investment incentives? Evidence from quotas in Brazilian Public Universities,” Unpublished Manuscript, February 2015, Can be found (as of Feb. 2015), at https://dl.dropboxusercontent.com/u/12654869/Assuncao%20and%20Ferman022015.pdf.

Bell, R. M. and D. F. McCaffrey, “Bias Reduction in Standard Errors for Linear Regression with Multi-Stage Samples,” Survey Methodology, 2002, 28 (2), 169–181.

Bertrand, Marianne, Esther Duflo, and Sendhil Mullainathan, “How Much Should We Trust Differences-in-Differences Estimates?,” Quarterly Journal of Economics, 2004, p. 24975.

Brewer, Mike, Thomas F. Crossley, and Robert Joyce, “Inference with Difference-in-Differences Revisited,” IZA Discussion Papers 7742, Institute for the Study of Labor (IZA) November 2013.

Cameron, A.C., J.B. Gelbach, and D.L. Miller, “Bootstrap-based improvements for inference with clustered errors,” The Review of Economics and Statistics, 2008, 90 (3), 414–427.

Canay, Ivan A., Joseph P. Romano, and Azeem M. Shaikh, “Randomization Tests under an Ap- proximate Symmetry Assumption?,” 2014.

Conley, Timothy G. and Christopher R. Taber, “Inference with “Difference in Differences with a Small Number of Policy Changes,” The Review of Economics and Statistics, February 2011, 93 (1), 113–125.

Donald, Stephen G. and Kevin Lang, “Inference with Difference-in-Differences and Other Panel Data,” The Review of Economics and Statistics, May 2007, 89 (2), 221–233.

Fisher, R.A., The design of experiments. 1935, Edinburgh: Oliver and Boyd, 1935.

Hansen, Christian B., “Generalized least squares inference in panel and multilevel models with serial correlation and fixed effects,” Journal of Econometrics, October 2007, 140 (2), 670–694.

Hausman, Jerry and Guido Kuersteiner, “Difference in difference meets generalized least squares: Higher order properties of hypotheses tests,” Journal of Econometrics, June 2008, 144 (2), 371–391.

Ibragimov, Rustam and Ulrich K. Mller, “Inference with Few Heterogenous Clusters,” 2013.

Imbens, Guido W. and Michal Kolesar, “Robust Standard Errors in Small Samples: Some Practical Advice,” Working Paper 18478, National Bureau of Economic Research October 2012.

Lehmann, E.L. and J.P. Romano, Testing Statistical Hypotheses Springer Texts in Statistics, Springer New York, 2008.

Liang, Kung-Yee and Scott L. Zeger, “Longitudinal data analysis using generalized linear mod- els,” Biometrika, 1986, 73 (1), 13–22.

MacKinnon, James G. and Matthew D. Webb, “Differences-in-Differences Inference with Few Treated Clusters,” 2015.

MacKinnon, James G. and Matthew D. Webb, “Wild Bootstrap Inference for Wildly Different Cluster Sizes,” Working Papers 1314, Queen’s University, Department of Economics February 2015.

Moulton, Brent R., “Random group effects and the precision of regression estimates,” Journal of Econo- metrics, August 1986, 32 (3), 385–397.

Webb, Matthew D., “Reworking Wild Bootstrap Based Inference for Clustered Errors,” Working Papers 1315, Queen’s University, Department of Economics November 2014.

White, Halbert, “A Heteroskedasticity-Consistent Covariance Matrix Estimator and a Direct Test for Heteroskedasticity,” Econometrica, May 1980, 48 (4), 817–838.

Wooldridge, Jeffrey M., “Cluster-Sample Methods in Applied Econometrics,” American Economic Review, 2003, 93 (2), 133–138.

Yates, F., “Tests of Significance for 2 2 Contingency Tables,” Journal of the Royal Statistical Society. Series A (General), 1984, 147 (3), 426–463.

Young, Alwyn, “Channeling Fisher: Randomization Tests and the Statistical Insignificance of Seemingly Significant Experimental Results,” 2015.