Inference in Differences-in-Differences with Few Treated Groups and Heteroskedasticity

Ferman, Bruno and Pinto, Cristine (2015): Inference in Differences-in-Differences with Few Treated Groups and Heteroskedasticity.

Abstract

Differences-in-Differences (DID) is one of the most widely used identification strategies in applied economics. However, how to draw inferences in DID models when there are few treated groups remains an open question. We show that the usual inference methods used in DID models might not perform well when there are few treated groups and errors are heteroskedastic. In particular, when there is variation in the number of observations per group, inference methods designed to work when there are few treated groups tend to (under-) over-reject the null hypothesis when the treated groups are (large) small relative to the control groups. This happens because larger groups tend to have lower variance, generating heteroskedasticity in the group x time aggregate DID model. We provide evidence from Monte Carlo simulations and from placebo DID regressions with the American Community Survey (ACS) and the Current Population Survey (CPS) datasets to show that this problem is relevant even in datasets with large numbers of observations per group. We then derive an alternative inference method that provides accurate hypothesis testing in situations where there are few treated groups (or even just one) and many control groups in the presence of heteroskedasticity. Our method assumes that we can model the heteroskedasticity of a linear combination of the errors. We show that this assumption can be satisfied without imposing strong assumptions on the errors in common DID applications. Importantly, we do not need to specify the structure of the serial correlation of the errors. Our inference method can also be combined with feasible generalized least square (FGLS) estimation. This way, it is possible to attain an asymptotically uniformly most powerful (UMP) test if the FGLS t-test is asymptotically UMP, while still provide a test with correct size if the serial correlation is misspecified. With many pre-treatment periods, we provide an alternative inference method that relies on strict stationarity and ergodicity of the time series instead of relying on the correct specification of the heteroskedasticity. Finally, we extend our inference methods to linear factor models when there are few treated groups.

• Item Type: MPRA Paper
• Language: English
• Keywords: differences-in-differences; inference; heteroskedasticity; clustering; few clusters; bootstrap; randomization inference; linear factor model
• Subjects: C - Mathematical and Quantitative Methods > C1 - Econometric and Statistical Methods and Methodology: General > C12 - Hypothesis Testing: General; C - Mathematical and Quantitative Methods > C2 - Single Equation Models ; Single Variables > C21 - Cross-Sectional Models ; Spatial Models ; Treatment Effect Models ; Quantile Regressions; C - Mathematical and Quantitative Methods > C3 - Multiple or Simultaneous Equation Models ; Multiple Variables > C33 - Panel Data Models ; Spatio-temporal Models

References

Abadie, Alberto, Alexis Diamond, and Jens Hainmueller, “Synthetic Control Methods for Com- parative Case Studies: Estimating the Effect of Californias Tobacco Control Program,” Journal of the American Statistical Association, 2010, 105 (490), 493–505.

Abadie, Alberto, and Javier Gardeazabal, “The Economic Costs of Conflict: A Case Study of the Basque Country,” American Economic Review, March 2003, 93 (1), 113–132.

Andrews, D. W. K., “End-of-Sample Instability Tests,” Econometrica, 2003, 71 (6), 1661–1694.

Angrist, J.D. and J.S. Pischke, Mostly Harmless Econometrics: An Empiricist’s Companion, Princeton University Press, 2009.

Assuncao, J. and B. Ferman, “Does affirmative action enhance or under- cut investment incentives? Evidence from quotas in Brazilian Public Universi- ties,” Unpublished Manuscript, February 2015, Can be found (as of Feb. 2015), at https://dl.dropboxusercontent.com/u/12654869/Assuncao%20and%20Ferman022015.pdf.

Bai, Jushan, “Panel Data Models With Interactive Fixed Effects,” Econometrica, 2009, 77 (4), 1229–1279.

Barndorff-Nielsen, O. E., “Conditionality Resolutions,” Biometrika, 1980, 67 (2), 293–310.

Barndorff-Nielsen, O. E., “On a formula for the distribution of the maximum likelihood estimator,” Biometrika, 1983, 70, 343–65.

Barndorff-Nielsen, O. E., “On Conditionality Resolution and the Likelihood Ratio for Curved Exponential Models,” Scandinavian Journal of Statistics, 1984, 11 (3), 157–170.

Behrens, W. U., “Ein Beitrag zur Fehlerberechnung bei wenigen Beobachtungen,” Landwirtschaftliche Jahrbucher., 1929, 68, 807–837.

Bell, R. M. and D. F. McCaffrey, “Bias Reduction in Standard Errors for Linear Regression with Multi-Stage Samples,” Survey Methodology, 2002, 28 (2), 169–181.

Bertrand, Marianne, Esther Duflo, and Sendhil Mullainathan, “How Much Should We Trust Differences-in-Differences Estimates?,” Quarterly Journal of Economics, 2004, p. 24975.

Brewer, Mike, Thomas F. Crossley, and Robert Joyce, “Inference with Difference-in-Differences Revisited,” IZA Discussion Papers 7742, Institute for the Study of Labor (IZA) November 2013.

Cameron, A.C., J.B. Gelbach, and D.L. Miller, “Bootstrap-based improvements for inference with clustered errors,” The Review of Economics and Statistics, 2008, 90 (3), 414–427.

Canay, Ivan A., Joseph P. Romano, and Azeem M. Shaikh, “Randomization Tests under an Ap- proximate Symmetry Assumption?,” 2014.

Carvalho, Carlos V., Ricardo Mansini, and Marcelo C. Medeiros, “ArCo: An Artificial Counter- factual Approach for Aggregate Data,” February 2015. Working Paper.

Casella, G. and C. Goutis, “Frequentist Post-Data Inference,” International Statistical Review, 1995, 63, 325–344.

Conley, Timothy G. and Christopher R. Taber, “Inference with “Difference in Differences with a Small Number of Policy Changes,” The Review of Economics and Statistics, February 2011, 93 (1), 113–125.

Cox, D. R., “Some Problems Connected with Statistical Inference,” Ann. Math. Statist., 06 1958, 29 (2), 357–372.

Cox, D. R., “Local Ancillarity,” Biometrika, 1980, 67, 279–86.

Cox, D.R. and D.V. Hinkley, Theoretical Statistics, Taylor & Francis, 1979.

Donald, Stephen G. and Kevin Lang, “Inference with Difference-in-Differences and Other Panel Data,” The Review of Economics and Statistics, May 2007, 89 (2), 221–233.

Efron, David V. Hinkley Bradley, “Assessing the Accuracy of the Maximum Likelihood Estimator: Observed Versus Expected Fisher Information,” Biometrika, 1978, 65 (3), 457–482.

Ferman, Bruno and Cristine Campos de Xavier Pinto, “Revisiting the synthetic control estimator,” Textos para discusso 421, Escola de Economia de So Paulo, Getulio Vargas Foundation (Brazil) June 2016.

Ferman, Bruno and Cristine Pinto, “Inference in Differences-in-Differences with Few Treated Groups and Heteroskedas- ticity,” MPRA Paper 67665, University Library of Munich, Germany November 2015.

Fisher, R. A., “Two New Properties of Mathematical Likelihood,” Proceedings of the Royal Society of London A: Mathematical, Physical and Engineering Sciences, 1934, 144 (852), 285–307.

Fisher, R. A., The design of experiments. 1935, Edinburgh: Oliver and Boyd, 1935.

Fisher, R. A., “The comparison of sample with possibly unequal variances,” Annals of Eugenics, 1939, 9 (2), 380–385.

Fraser, D.A.S., The structure of inference Wiley series in probability and mathematical statistics, Wiley, 1968.

Gobillon, Laurent and Thierry Magnac, “Regional Policy Evaluation: Interactive Fixed Effects and Synthetic Controls,” PSE Working Papers halshs-00849071, HAL July 2013.

Hansen, Christian B., “Generalized least squares inference in panel and multilevel models with serial correlation and fixed effects,” Journal of Econometrics, October 2007, 140 (2), 670–694.

Hausman, Jerry and Guido Kuersteiner, “Difference in difference meets generalized least squares: Higher order properties of hypotheses tests,” Journal of Econometrics, June 2008, 144 (2), 371–391.

Hinkley, D. V., “Likelihood as Approximate Pivotal Distribution,” Biometrika, 1980, 67 (2), 287–292.

Ibragimov, Rustam and Ulrich K. Mu¨ller, “Inference with Few Heterogenous Clusters,” 2013.

Imbens, Guido W. and Michal Kolesar, “Robust Standard Errors in Small Samples: Some Practical Advice,” Working Paper 18478, National Bureau of Economic Research October 2012.

Lehmann, E.L. and J.P. Romano, Testing Statistical Hypotheses Springer Texts in Statistics, Springer New York, 2008.

Liang, Kung-Yee and Scott L. Zeger, “Longitudinal data analysis using generalized linear models,” Biometrika, 1986, 73 (1), 13–22.

MacKinnon, James G. and Matthew D. Webb, “Differences-in-Differences Inference with Few Treated Clusters,” 2015.

MacKinnon, James G. and Matthew D. Webb, “Wild Bootstrap Inference for Wildly Different Cluster Sizes,” Working Papers 1314, Queen’s University, Department of Economics February 2015.

Mcullagh, P., “Local Sufficiency,” Biometrika, 1984, 71, 233–44.

Mcullagh, P., “Conditional Inference and Cauchy Models,” Biometrika, 1992, 79 (2), 247–259.

Moulton, Brent R., “Random group effects and the precision of regression estimates,” Journal of Econo- metrics, August 1986, 32 (3), 385–397.

Pengyuan, Wang, Traskin Mikhail, and Small Dylan S., “Robust Inferences from a Before-and-After Study with Multiple Unaffected Control Groups,” Journal of Causal Inference, 2013, 1 (2), 209–234.

Pitman, E. J. G., “The Estimation of the Location and Scale Parameters of a Continuous Population of any Given Form,” Biometrika, 1938, 30 (3-4), 391–421.

Powell, David, “Synthetic Control Estimation Beyond Case Studies: Does the Minimum Wage Reduce Employment?,” 2016. RAND Corporation, WR-1142.

Rosenbaum, Paul R., “Covariance Adjustment in Randomized Experiments and Observational Studies,” Statist. Sci., 08 2002, 17 (3), 286–327.

Ruggles, Steven, Katie Genadek, Ronald Goeken, Josiah Grover, and Matthew Sobek, “Inte- grated Public Use Microdata Series: Version 6.0 [Machine-readable database].,” 2015.

Scheffe, H., “Practical solutions of the Behrens-Fisher problem,” Journal of the American Statistical As- sociation., 1970, 65, 1501–1508.

Wang, Y. Y., “Probabilities or the Type l errors of the Welch tests for the Behrens-Fisher problem,” Journal of the American Statistical Association., 1971, 66, 605–608.

Webb, Matthew D., “Reworking Wild Bootstrap Based Inference for Clustered Errors,” Working Papers 1315, Queen’s University, Department of Economics November 2014.

White, Halbert, “A Heteroskedasticity-Consistent Covariance Matrix Estimator and a Direct Test for Heteroskedasticity,” Econometrica, May 1980, 48 (4), 817–838.

Wooldridge, Jeffrey M., “Cluster-Sample Methods in Applied Econometrics,” American Economic Re- view, 2003, 93 (2), 133–138.

Yates, F., “Tests of Significance for 2 2 Contingency Tables,” Journal of the Royal Statistical Society. Series A (General), 1984, 147 (3), 426–463.

Young, Alwyn, “Channeling Fisher: Randomization Tests and the Statistical Insignificance of Seemingly Significant Experimental Results,” 2015.