Inference in Differences-in-Differences with Few Treated Groups and Heteroskedasticity

Ferman, Bruno and Pinto, Cristine (2015): Inference in Differences-in-Differences with Few Treated Groups and Heteroskedasticity.

Abstract

We show that existing inference methods used in Differences-in-Differences might not perform well with few treated groups and heteroskedastic errors. This is restrictive because variation in the number of observations per group inherently leads to heteroskedasticity in the group x time aggregate model. We provide theoretical justification and empirical evidence from placebo simulations with real datasets showing that this problem may remain relevant even in datasets with a large number of observations per group. We then derive an alternative inference method that works when there are few treated groups (or even just one) and many control groups in the presence of heteroskedasticity. Combined with feasible generalized least squares estimation, our test is uniformly most powerful under normality and a consistent estimator for the variance-covariance matrix, while it can still provide a test with correct size if the serial correlation is misspecified or errors are not normally distributed.

• Item Type: MPRA Paper
• Language: English
• Keywords: differences-in-differences; inference; heteroskedasticity; clustering; bootstrap; permutation tests; Behrens-Fisher problem
• Subjects: C - Mathematical and Quantitative Methods > C1 - Econometric and Statistical Methods and Methodology: General > C12 - Hypothesis Testing: General; C - Mathematical and Quantitative Methods > C2 - Single Equation Models ; Single Variables > C21 - Cross-Sectional Models ; Spatial Models ; Treatment Effect Models ; Quantile Regressions; C - Mathematical and Quantitative Methods > C3 - Multiple or Simultaneous Equation Models ; Multiple Variables > C33 - Panel Data Models ; Spatio-temporal Models

References

Abadie, Alberto, Alexis Diamond, and Jens Hainmueller, “Synthetic Control Methods for Com- parative Case Studies: Estimating the Effect of Californias Tobacco Control Program,” Journal of the American Statistical Association, 2010, 105 (490), 493–505.

Abadie, Alberto, and Javier Gardeazabal, “The Economic Costs of Conflict: A Case Study of the Basque Country,” American Economic Review, March 2003, 93 (1), 113–132.

Andrews, D. W. K., “End-of-Sample Instability Tests,” Econometrica, 2003, 71 (6), 1661–1694.

Angrist, J.D. and J.S. Pischke, Mostly Harmless Econometrics: An Empiricist’s Companion, Princeton University Press, 2009.

Assuncao, J. and B. Ferman, “Does affirmative action enhance or under- cut investment incentives? Evidence from quotas in Brazilian Public Universi- ties,” Unpublished Manuscript, February 2015, Can be found (as of Feb. 2015), at https://dl.dropboxusercontent.com/u/12654869/Assuncao%20and%20Ferman022015.pdf.

Bai, Jushan, “Panel Data Models With Interactive Fixed Effects,” Econometrica, 2009, 77 (4), 1229–1279.

Barndorff-Nielsen, O. E., “Conditionality Resolutions,” Biometrika, 1980, 67 (2), 293–310.

Barndorff-Nielsen, O. E., “On a formula for the distribution of the maximum likelihood estimator,” Biometrika, 1983, 70, 343–65.

Barndorff-Nielsen, O. E., “On Conditionality Resolution and the Likelihood Ratio for Curved Exponential Models,” Scandinavian Journal of Statistics, 1984, 11 (3), 157–170.

Behrens, W. U., “Ein Beitrag zur Fehlerberechnung bei wenigen Beobachtungen,” Landwirtschaftliche Jahrbucher., 1929, 68, 807–837.

Bell, R. M. and D. F. McCaffrey, “Bias Reduction in Standard Errors for Linear Regression with Multi-Stage Samples,” Survey Methodology, 2002, 28 (2), 169–181.

Bertrand, Marianne, Esther Duflo, and Sendhil Mullainathan, “How Much Should We Trust Differences-in-Differences Estimates?,” Quarterly Journal of Economics, 2004, p. 24975.

Brewer, Mike, Thomas F. Crossley, and Robert Joyce, “Inference with Difference-in-Differences Revisited,” IZA Discussion Papers 7742, Institute for the Study of Labor (IZA) November 2013.

Cameron, A.C., J.B. Gelbach, and D.L. Miller, “Bootstrap-based improvements for inference with clustered errors,” The Review of Economics and Statistics, 2008, 90 (3), 414–427.

Canay, Ivan A., Joseph P. Romano, and Azeem M. Shaikh, “Randomization Tests under an Ap- proximate Symmetry Assumption?,” 2014.

Carvalho, Carlos V., Ricardo Mansini, and Marcelo C. Medeiros, “ArCo: An Artificial Counter- factual Approach for Aggregate Data,” February 2015. Working Paper.

Casella, G. and C. Goutis, “Frequentist Post-Data Inference,” International Statistical Review, 1995, 63, 325–344.

Conley, Timothy G. and Christopher R. Taber, “Inference with “Difference in Differences with a Small Number of Policy Changes,” The Review of Economics and Statistics, February 2011, 93 (1), 113–125.

Cox, D. R., “Some Problems Connected with Statistical Inference,” Ann. Math. Statist., 06 1958, 29 (2), 357–372.

Cox, D. R., “Local Ancillarity,” Biometrika, 1980, 67, 279–86.

Cox, D.R. and D.V. Hinkley, Theoretical Statistics, Taylor & Francis, 1979.

Donald, Stephen G. and Kevin Lang, “Inference with Difference-in-Differences and Other Panel Data,” The Review of Economics and Statistics, May 2007, 89 (2), 221–233.

Efron, David V. Hinkley Bradley, “Assessing the Accuracy of the Maximum Likelihood Estimator: Observed Versus Expected Fisher Information,” Biometrika, 1978, 65 (3), 457–482.

Ferman, Bruno and Cristine Campos de Xavier Pinto, “Revisiting the synthetic control estimator,” Textos para discusso 421, Escola de Economia de So Paulo, Getulio Vargas Foundation (Brazil) June 2016.

Ferman, Bruno and Cristine Pinto, “Inference in Differences-in-Differences with Few Treated Groups and Heteroskedas- ticity,” MPRA Paper 67665, University Library of Munich, Germany November 2015.

Fisher, R. A., “Two New Properties of Mathematical Likelihood,” Proceedings of the Royal Society of London A: Mathematical, Physical and Engineering Sciences, 1934, 144 (852), 285–307.

Fisher, R. A., The design of experiments. 1935, Edinburgh: Oliver and Boyd, 1935.

Fisher, R. A., “The comparison of sample with possibly unequal variances,” Annals of Eugenics, 1939, 9 (2), 380–385.

Fraser, D.A.S., The structure of inference Wiley series in probability and mathematical statistics, Wiley, 1968.

Gobillon, Laurent and Thierry Magnac, “Regional Policy Evaluation: Interactive Fixed Effects and Synthetic Controls,” PSE Working Papers halshs-00849071, HAL July 2013.

Hansen, Christian B., “Generalized least squares inference in panel and multilevel models with serial correlation and fixed effects,” Journal of Econometrics, October 2007, 140 (2), 670–694.

Hausman, Jerry and Guido Kuersteiner, “Difference in difference meets generalized least squares: Higher order properties of hypotheses tests,” Journal of Econometrics, June 2008, 144 (2), 371–391.

Hinkley, D. V., “Likelihood as Approximate Pivotal Distribution,” Biometrika, 1980, 67 (2), 287–292.

Ibragimov, Rustam and Ulrich K. Mu¨ller, “Inference with Few Heterogenous Clusters,” 2013.

Imbens, Guido W. and Michal Kolesar, “Robust Standard Errors in Small Samples: Some Practical Advice,” Working Paper 18478, National Bureau of Economic Research October 2012.

Lehmann, E.L. and J.P. Romano, Testing Statistical Hypotheses Springer Texts in Statistics, Springer New York, 2008.

Liang, Kung-Yee and Scott L. Zeger, “Longitudinal data analysis using generalized linear models,” Biometrika, 1986, 73 (1), 13–22.

MacKinnon, James G. and Matthew D. Webb, “Differences-in-Differences Inference with Few Treated Clusters,” 2015.

MacKinnon, James G. and Matthew D. Webb, “Wild Bootstrap Inference for Wildly Different Cluster Sizes,” Working Papers 1314, Queen’s University, Department of Economics February 2015.

Mcullagh, P., “Local Sufficiency,” Biometrika, 1984, 71, 233–44.

Mcullagh, P., “Conditional Inference and Cauchy Models,” Biometrika, 1992, 79 (2), 247–259.

Moulton, Brent R., “Random group effects and the precision of regression estimates,” Journal of Econo- metrics, August 1986, 32 (3), 385–397.

Pengyuan, Wang, Traskin Mikhail, and Small Dylan S., “Robust Inferences from a Before-and-After Study with Multiple Unaffected Control Groups,” Journal of Causal Inference, 2013, 1 (2), 209–234.

Pitman, E. J. G., “The Estimation of the Location and Scale Parameters of a Continuous Population of any Given Form,” Biometrika, 1938, 30 (3-4), 391–421.

Powell, David, “Synthetic Control Estimation Beyond Case Studies: Does the Minimum Wage Reduce Employment?,” 2016. RAND Corporation, WR-1142.

Rosenbaum, Paul R., “Covariance Adjustment in Randomized Experiments and Observational Studies,” Statist. Sci., 08 2002, 17 (3), 286–327.

Ruggles, Steven, Katie Genadek, Ronald Goeken, Josiah Grover, and Matthew Sobek, “Inte- grated Public Use Microdata Series: Version 6.0 [Machine-readable database].,” 2015.

Scheffe, H., “Practical solutions of the Behrens-Fisher problem,” Journal of the American Statistical As- sociation., 1970, 65, 1501–1508.

Wang, Y. Y., “Probabilities or the Type l errors of the Welch tests for the Behrens-Fisher problem,” Journal of the American Statistical Association., 1971, 66, 605–608.

Webb, Matthew D., “Reworking Wild Bootstrap Based Inference for Clustered Errors,” Working Papers 1315, Queen’s University, Department of Economics November 2014.

White, Halbert, “A Heteroskedasticity-Consistent Covariance Matrix Estimator and a Direct Test for Heteroskedasticity,” Econometrica, May 1980, 48 (4), 817–838.

Wooldridge, Jeffrey M., “Cluster-Sample Methods in Applied Econometrics,” American Economic Re- view, 2003, 93 (2), 133–138.

Yates, F., “Tests of Significance for 2 2 Contingency Tables,” Journal of the Royal Statistical Society. Series A (General), 1984, 147 (3), 426–463.

Young, Alwyn, “Channeling Fisher: Randomization Tests and the Statistical Insignificance of Seemingly Significant Experimental Results,” 2015.