Ferman, Bruno and Pinto, Cristine (2015): Inference in DifferencesinDifferences with Few Treated Groups and Heteroskedasticity.
Preview 
PDF
MPRA_paper_73683.pdf Download (1MB)  Preview 
Abstract
DifferencesinDifferences (DID) is one of the most widely used identification strategies in applied economics. However, how to draw inferences in DID models when there are few treated groups remains an open question. We show that the usual inference methods used in DID models might not perform well when there are few treated groups and errors are heteroskedastic. In particular, when there is variation in the number of observations per group, inference methods designed to work when there are few treated groups tend to (under) overreject the null hypothesis when the treated groups are (large) small relative to the control groups. This happens because larger groups tend to have lower variance, generating heteroskedasticity in the group x time aggregate DID model. We provide evidence from Monte Carlo simulations and from placebo DID regressions with the American Community Survey (ACS) and the Current Population Survey (CPS) datasets to show that this problem is relevant even in datasets with large numbers of observations per group. We then derive an alternative inference method that provides accurate hypothesis testing in situations where there are few treated groups (or even just one) and many control groups in the presence of heteroskedasticity. Our method assumes that we can model the heteroskedasticity of a linear combination of the errors. We show that this assumption can be satisfied without imposing strong assumptions on the errors in common DID applications. Importantly, we do not need to specify the structure of the serial correlation of the errors. Our inference method can also be combined with feasible generalized least square (FGLS) estimation. This way, it is possible to attain an asymptotically uniformly most powerful (UMP) test if the FGLS ttest is asymptotically UMP, while still provide a test with correct size if the serial correlation is misspecified. With many pretreatment periods, we provide an alternative inference method that relies on strict stationarity and ergodicity of the time series instead of relying on the correct specification of the heteroskedasticity. Finally, we extend our inference methods to linear factor models when there are few treated groups.
Item Type:  MPRA Paper 

Original Title:  Inference in DifferencesinDifferences with Few Treated Groups and Heteroskedasticity 
Language:  English 
Keywords:  differencesindifferences; inference; heteroskedasticity; clustering; few clusters; bootstrap; randomization inference; linear factor model 
Subjects:  C  Mathematical and Quantitative Methods > C1  Econometric and Statistical Methods and Methodology: General > C12  Hypothesis Testing: General C  Mathematical and Quantitative Methods > C2  Single Equation Models ; Single Variables > C21  CrossSectional Models ; Spatial Models ; Treatment Effect Models ; Quantile Regressions C  Mathematical and Quantitative Methods > C3  Multiple or Simultaneous Equation Models ; Multiple Variables > C33  Panel Data Models ; Spatiotemporal Models 
Item ID:  73683 
Depositing User:  Bruno Ferman 
Date Deposited:  14 Sep 2016 06:00 
Last Modified:  27 Sep 2019 18:09 
References:  Abadie, Alberto, Alexis Diamond, and Jens Hainmueller, “Synthetic Control Methods for Com parative Case Studies: Estimating the Effect of Californias Tobacco Control Program,” Journal of the American Statistical Association, 2010, 105 (490), 493–505. Abadie, Alberto, and Javier Gardeazabal, “The Economic Costs of Conflict: A Case Study of the Basque Country,” American Economic Review, March 2003, 93 (1), 113–132. Andrews, D. W. K., “EndofSample Instability Tests,” Econometrica, 2003, 71 (6), 1661–1694. Angrist, J.D. and J.S. Pischke, Mostly Harmless Econometrics: An Empiricist’s Companion, Princeton University Press, 2009. Assuncao, J. and B. Ferman, “Does affirmative action enhance or under cut investment incentives? Evidence from quotas in Brazilian Public Universi ties,” Unpublished Manuscript, February 2015, Can be found (as of Feb. 2015), at https://dl.dropboxusercontent.com/u/12654869/Assuncao%20and%20Ferman022015.pdf. Bai, Jushan, “Panel Data Models With Interactive Fixed Effects,” Econometrica, 2009, 77 (4), 1229–1279. BarndorffNielsen, O. E., “Conditionality Resolutions,” Biometrika, 1980, 67 (2), 293–310. BarndorffNielsen, O. E., “On a formula for the distribution of the maximum likelihood estimator,” Biometrika, 1983, 70, 343–65. BarndorffNielsen, O. E., “On Conditionality Resolution and the Likelihood Ratio for Curved Exponential Models,” Scandinavian Journal of Statistics, 1984, 11 (3), 157–170. Behrens, W. U., “Ein Beitrag zur Fehlerberechnung bei wenigen Beobachtungen,” Landwirtschaftliche Jahrbucher., 1929, 68, 807–837. Bell, R. M. and D. F. McCaffrey, “Bias Reduction in Standard Errors for Linear Regression with MultiStage Samples,” Survey Methodology, 2002, 28 (2), 169–181. Bertrand, Marianne, Esther Duflo, and Sendhil Mullainathan, “How Much Should We Trust DifferencesinDifferences Estimates?,” Quarterly Journal of Economics, 2004, p. 24975. Brewer, Mike, Thomas F. Crossley, and Robert Joyce, “Inference with DifferenceinDifferences Revisited,” IZA Discussion Papers 7742, Institute for the Study of Labor (IZA) November 2013. Cameron, A.C., J.B. Gelbach, and D.L. Miller, “Bootstrapbased improvements for inference with clustered errors,” The Review of Economics and Statistics, 2008, 90 (3), 414–427. Canay, Ivan A., Joseph P. Romano, and Azeem M. Shaikh, “Randomization Tests under an Ap proximate Symmetry Assumption?,” 2014. Carvalho, Carlos V., Ricardo Mansini, and Marcelo C. Medeiros, “ArCo: An Artificial Counter factual Approach for Aggregate Data,” February 2015. Working Paper. Casella, G. and C. Goutis, “Frequentist PostData Inference,” International Statistical Review, 1995, 63, 325–344. Conley, Timothy G. and Christopher R. Taber, “Inference with “Difference in Differences with a Small Number of Policy Changes,” The Review of Economics and Statistics, February 2011, 93 (1), 113–125. Cox, D. R., “Some Problems Connected with Statistical Inference,” Ann. Math. Statist., 06 1958, 29 (2), 357–372. Cox, D. R., “Local Ancillarity,” Biometrika, 1980, 67, 279–86. Cox, D.R. and D.V. Hinkley, Theoretical Statistics, Taylor & Francis, 1979. Donald, Stephen G. and Kevin Lang, “Inference with DifferenceinDifferences and Other Panel Data,” The Review of Economics and Statistics, May 2007, 89 (2), 221–233. Efron, David V. Hinkley Bradley, “Assessing the Accuracy of the Maximum Likelihood Estimator: Observed Versus Expected Fisher Information,” Biometrika, 1978, 65 (3), 457–482. Ferman, Bruno and Cristine Campos de Xavier Pinto, “Revisiting the synthetic control estimator,” Textos para discusso 421, Escola de Economia de So Paulo, Getulio Vargas Foundation (Brazil) June 2016. Ferman, Bruno and Cristine Pinto, “Inference in DifferencesinDifferences with Few Treated Groups and Heteroskedas ticity,” MPRA Paper 67665, University Library of Munich, Germany November 2015. Fisher, R. A., “Two New Properties of Mathematical Likelihood,” Proceedings of the Royal Society of London A: Mathematical, Physical and Engineering Sciences, 1934, 144 (852), 285–307. Fisher, R. A., The design of experiments. 1935, Edinburgh: Oliver and Boyd, 1935. Fisher, R. A., “The comparison of sample with possibly unequal variances,” Annals of Eugenics, 1939, 9 (2), 380–385. Fraser, D.A.S., The structure of inference Wiley series in probability and mathematical statistics, Wiley, 1968. Gobillon, Laurent and Thierry Magnac, “Regional Policy Evaluation: Interactive Fixed Effects and Synthetic Controls,” PSE Working Papers halshs00849071, HAL July 2013. Hansen, Christian B., “Generalized least squares inference in panel and multilevel models with serial correlation and fixed effects,” Journal of Econometrics, October 2007, 140 (2), 670–694. Hausman, Jerry and Guido Kuersteiner, “Difference in difference meets generalized least squares: Higher order properties of hypotheses tests,” Journal of Econometrics, June 2008, 144 (2), 371–391. Hinkley, D. V., “Likelihood as Approximate Pivotal Distribution,” Biometrika, 1980, 67 (2), 287–292. Ibragimov, Rustam and Ulrich K. Mu¨ller, “Inference with Few Heterogenous Clusters,” 2013. Imbens, Guido W. and Michal Kolesar, “Robust Standard Errors in Small Samples: Some Practical Advice,” Working Paper 18478, National Bureau of Economic Research October 2012. Lehmann, E.L. and J.P. Romano, Testing Statistical Hypotheses Springer Texts in Statistics, Springer New York, 2008. Liang, KungYee and Scott L. Zeger, “Longitudinal data analysis using generalized linear models,” Biometrika, 1986, 73 (1), 13–22. MacKinnon, James G. and Matthew D. Webb, “DifferencesinDifferences Inference with Few Treated Clusters,” 2015. MacKinnon, James G. and Matthew D. Webb, “Wild Bootstrap Inference for Wildly Different Cluster Sizes,” Working Papers 1314, Queen’s University, Department of Economics February 2015. Mcullagh, P., “Local Sufficiency,” Biometrika, 1984, 71, 233–44. Mcullagh, P., “Conditional Inference and Cauchy Models,” Biometrika, 1992, 79 (2), 247–259. Moulton, Brent R., “Random group effects and the precision of regression estimates,” Journal of Econo metrics, August 1986, 32 (3), 385–397. Pengyuan, Wang, Traskin Mikhail, and Small Dylan S., “Robust Inferences from a BeforeandAfter Study with Multiple Unaffected Control Groups,” Journal of Causal Inference, 2013, 1 (2), 209–234. Pitman, E. J. G., “The Estimation of the Location and Scale Parameters of a Continuous Population of any Given Form,” Biometrika, 1938, 30 (34), 391–421. Powell, David, “Synthetic Control Estimation Beyond Case Studies: Does the Minimum Wage Reduce Employment?,” 2016. RAND Corporation, WR1142. Rosenbaum, Paul R., “Covariance Adjustment in Randomized Experiments and Observational Studies,” Statist. Sci., 08 2002, 17 (3), 286–327. Ruggles, Steven, Katie Genadek, Ronald Goeken, Josiah Grover, and Matthew Sobek, “Inte grated Public Use Microdata Series: Version 6.0 [Machinereadable database].,” 2015. Scheffe, H., “Practical solutions of the BehrensFisher problem,” Journal of the American Statistical As sociation., 1970, 65, 1501–1508. Wang, Y. Y., “Probabilities or the Type l errors of the Welch tests for the BehrensFisher problem,” Journal of the American Statistical Association., 1971, 66, 605–608. Webb, Matthew D., “Reworking Wild Bootstrap Based Inference for Clustered Errors,” Working Papers 1315, Queen’s University, Department of Economics November 2014. White, Halbert, “A HeteroskedasticityConsistent Covariance Matrix Estimator and a Direct Test for Heteroskedasticity,” Econometrica, May 1980, 48 (4), 817–838. Wooldridge, Jeffrey M., “ClusterSample Methods in Applied Econometrics,” American Economic Re view, 2003, 93 (2), 133–138. Yates, F., “Tests of Significance for 2 2 Contingency Tables,” Journal of the Royal Statistical Society. Series A (General), 1984, 147 (3), 426–463. Young, Alwyn, “Channeling Fisher: Randomization Tests and the Statistical Insignificance of Seemingly Significant Experimental Results,” 2015. 
URI:  https://mpra.ub.unimuenchen.de/id/eprint/73683 
Available Versions of this Item

Inference in DifferencesinDifferences with Few Treated Groups and Heteroskedasticity. (deposited 06 Nov 2015 15:26)

Inference in DifferencesinDifferences with Few Treated Groups and Heteroskedasticity. (deposited 08 Dec 2015 19:58)
 Inference in DifferencesinDifferences with Few Treated Groups and Heteroskedasticity. (deposited 14 Sep 2016 06:00) [Currently Displayed]

Inference in DifferencesinDifferences with Few Treated Groups and Heteroskedasticity. (deposited 08 Dec 2015 19:58)