New Evidence on Linear Regression and Treatment Effect Heterogeneity

Słoczyński, Tymon (2014): New Evidence on Linear Regression and Treatment Effect Heterogeneity.

Abstract

It is standard practice in applied work to rely on linear least squares regression to estimate the effect of a binary variable ("treatment") on some outcome of interest. In this paper I study the interpretation of the regression estimand when treatment effects are in fact heterogeneous. I show that the coefficient on treatment is identical to the outcome of the following three-step procedure: first, calculate the linear projection of treatment on the vector of other covariates ("propensity score"); second, calculate average partial effects for both groups of interest from a regression of outcome on treatment, the propensity score, and their interaction; third, calculate a weighted average of these two effects, with weights being inversely related to the unconditional probability that a unit belongs to a given group. Each of these steps is potentially problematic, but this last property – the reliance on implicit weights which are inversely related to the proportion of each group – can have particularly devastating consequences for applied work. To illustrate the severity of this issue, I perform Monte Carlo simulations as well as replicate two prominent applied papers: Berger, Easterly, Nunn and Satyanath (2013) on the effects of successful CIA interventions during the Cold War on imports from the US; and Martinez-Bravo (2014) on the effects of appointed officials on village-level electoral results in Indonesia. In both cases some of the conclusions change dramatically after allowing for heterogeneity in effects.

• Item Type: MPRA Paper
• Language: English
• Keywords: treatment effects; linear regression; ordinary least squares; heterogeneity
• Subjects: C - Mathematical and Quantitative Methods > C3 - Multiple or Simultaneous Equation Models ; Multiple Variables > C31 - Cross-Sectional Models ; Spatial Models ; Treatment Effect Models ; Quantile Regressions ; Social Interaction Models; C - Mathematical and Quantitative Methods > C2 - Single Equation Models ; Single Variables > C21 - Cross-Sectional Models ; Spatial Models ; Treatment Effect Models ; Quantile Regressions; C - Mathematical and Quantitative Methods > C0 - General > C01 - Econometrics

References

ALESINA, A., GIULIANO, P. & NUNN, N. (2013). On the origins of gender roles: Women and the plough. Quarterly Journal of Economics 128, 469–530.

ALMOND, D., CHAY, K. Y. & LEE, D. S. (2005). The costs of low birth weight. Quarterly Journal of Economics 120, 1031–1083.

ANGRIST, J. D. (1998). Estimating the labor market impact of voluntary military service using Social Security data on military applicants. Econometrica 66, 249–288.

ANGRIST, J. D., GRADDY, K. & IMBENS, G. W. (2000). The interpretation of instrumental variables estimators in simultaneous equations models with an application to the demand for fish. Review of Economic Studies 67, 499–527.

ANGRIST, J. D. & KRUEGER, A. B. (1999). Empirical strategies in labor economics. In: Handbook of Labor Economics (ASHENFELTER, O. & CARD, D., eds.), vol. 3A. North-Holland.

ANGRIST, J. D. & PISCHKE, J.-S. (2009). Mostly Harmless Econometrics: An Empiricist’s Companion. Princeton University Press.

ARONOW, P. M. & SAMII, C. (2014). Does regression produce representative estimates of causal effects? Unpublished.

BERGER, D., EASTERLY, W., NUNN, N. & SATYANATH, S. (2013). Commercial imperialism? Political influence and trade during the Cold War. American Economic Review 103, 863–896.

BITLER, M. P., GELBACH, J. B. & HOYNES, H. W. (2006). What mean impacts miss: Distributional effects of welfare reform experiments. American Economic Review 96, 988–1012.

BITLER, M. P., GELBACH, J. B. & HOYNES, H. W. (2008). Distributional impacts of the Self-Sufficiency Project. Journal of Public Economics 92, 748–765.

BLACK, D. A., SMITH, J. A., BERGER, M. C. & NOEL, B. J. (2003). Is the threat of reemployment services more effective than the services themselves? Evidence from random assignment in the UI system. American Economic Review 93, 1313–1327.

BLINDER, A. S. (1973). Wage discrimination: Reduced form and structural estimates. Journal of Human Resources 8, 436–455.

BOND, T. N. & LANG, K. (2013). The evolution of the black-white test score gap in grades K–3: The fragility of results. Review of Economics and Statistics 95, 1468–1479.

BOUSTAN, L. P. & COLLINS, W. J. (2013). The origins and persistence of black-white differences in women’s labor force participation. NBER Working Paper no. 19040.

BUSSO, M., DINARDO, J. & MCCRARY, J. (2013). New evidence on the finite sample properties of propensity score reweighting and matching estimators. Review of Economics and Statistics (forthcoming).

CHERNOZHUKOV, V., FERNANDEZ-VAL, I., HAHN, J. & NEWEY, W. (2013). Average and quantile effects in nonseparable panel models. Econometrica 81, 535–580.

CRUMP, R. K., HOTZ, V. J., IMBENS, G. W. & MITNIK, O. A. (2008). Nonparametric tests for treatment effect heterogeneity. Review of Economics and Statistics 90, 389–405.

DEATON, A. (1997). The Analysis of Household Surveys: A Microeconometric Approach to Development Policy. Johns Hopkins University Press.

DEATON, A. (2010). Instruments, randomization, and learning about development. Journal of Economic Literature 48, 424–455.

DIETERLE, S. & SNELL, A. (2014). Exploiting nonlinearities in the first stage regressions of IV procedures. Unpublished.

ELDER, T. E., GODDEERIS, J. H. & HAIDER, S. J. (2010). Unexplained gaps and Oaxaca–Blinder decompositions. Labour Economics 17, 284–290.

FORTIN, N., LEMIEUX, T. & FIRPO, S. (2011). Decomposition methods in economics. In: Handbook of Labor Economics (ASHENFELTER, O. & CARD, D., eds.), vol. 4A. North-Holland.

FREEDMAN, D. A. (2008a). On regression adjustments in experiments with several treatments. Annals of Applied Statistics 2, 176–196.

FREEDMAN, D. A. (2008b). On regression adjustments to experimental data. Advances in Applied Mathematics 40, 180–193.

FRISCH, R. & WAUGH, F. V. (1933). Partial time regressions as compared with individual trends. Econometrica 1, 387–401.

FRYER, R. G. & GREENSTONE, M. (2010). The changing consequences of attending historically black colleges and universities. American Economic Journal: Applied Economics 2, 116–148.

FRYER, R. G. & LEVITT, S. D. (2004). Understanding the black-white test score gap in the first two years of school. Review of Economics and Statistics 86, 447–464.

FRYER, R. G. & LEVITT, S. D. (2010). An empirical analysis of the gender gap in mathematics. American Economic Journal: Applied Economics 2, 210–240.

GIBBONS, C. E., SUAREZ SERRATO, J. C. & URBANCIC, M. B. (2014). Broken or fixed effects? NBER Working Paper no. 20342.

GITTLEMAN, M. & WOLFF, E. N. (2004). Racial differences in patterns of wealth accumulation. Journal of Human Resources 39, 193–227.

HECKMAN, J. J. (2001). Micro data, heterogeneity, and the evaluation of public policy: Nobel Lecture. Journal of Political Economy 109, 673–748.

HECKMAN, J. J., URZUA, S. & VYTLACIL, E. (2006). Understanding instrumental variables in models with essential heterogeneity. Review of Economics and Statistics 88, 389–432.

HECKMAN, J. J. & VYTLACIL, E. (2005). Structural equations, treatment effects, and econometric policy evaluation. Econometrica 73, 669–738.

HECKMAN, J. J. & VYTLACIL, E. J. (2007). Econometric evaluation of social programs, part II: Using the marginal treatment effect to organize alternative econometric estimators to evaluate social programs, and to forecast their effects in new environments. In: Handbook of Econometrics (HECKMAN, J. J. & LEAMER, E. E., eds.), vol. 6B. North-Holland.

HOLLAND, P. W. (1986). Statistics and causal inference. Journal of the American Statistical Association 81, 945–960.

HUMPHREYS, M. (2009). Bounds on least squares estimates of causal effects in the presence of heterogeneous assignment probabilities. Unpublished.

IMAI, K. & KIM, I. S. (2013). On the use of linear fixed effects regression estimators for causal inference. Unpublished.

IMBENS, G. W. (2014). Matching methods in practice: Three examples. NBER Working Paper no. 19959.

IMBENS, G. W. & ANGRIST, J. D. (1994). Identification and estimation of local average treatment effects. Econometrica 62, 467–475.

IMBENS, G. W. & WOOLDRIDGE, J. M. (2009). Recent developments in the econometrics of program evaluation. Journal of Economic Literature 47, 5–86.

KHWAJA, A., PICONE, G., SALM, M. & TROGDON, J. G. (2011). A comparison of treatment effects estimators using a structural model of AMI treatment choices and severity of illness information from hospital charts. Journal of Applied Econometrics 26, 825–853.

KLINE, P. (2011). Oaxaca-Blinder as a reweighting estimator. American Economic Review: Papers & Proceedings 101, 532–537.

KLINE, P. (2014). A note on variance estimation for the Oaxaca estimator of average treatment effects. Economics Letters 122, 428–431.

KOLESAR, M. (2013). Estimation in an instrumental variables model with treatment effect heterogeneity. Unpublished.

LALONDE, R. J. (1986). Evaluating the econometric evaluations of training programs with experimental data. American Economic Review 76, 604–620.

LANG, K. & MANOVE, M. (2011). Education and labor market discrimination. American Economic Review 101, 1467–1496.

LIN, W. (2013). Agnostic notes on regression adjustments to experimental data: Reexamining Freedman’s critique. Annals of Applied Statistics 7, 295–318.

LØKEN, K. V., MOGSTAD, M. & WISWALL, M. (2012). What linear estimators miss: The effects of family income on child outcomes. American Economic Journal: Applied Economics 4, 1–35.

MARTINEZ-BRAVO, M. (2014). The role of local officials in new democracies: Evidence from Indonesia. American Economic Review 104, 1244–1287.

OAXACA, R. (1973). Male-female wage differentials in urban labor markets. International Economic Review 14, 693–709.

RHODES, W. (2010). Heterogeneous treatment effects: What does a regression estimate? Evaluation Review 34, 334–361.

ROSENBAUM, P. R. & RUBIN, D. B. (1983). The central role of the propensity score in observational studies for causal effects. Biometrika 70, 41–55.

ROTHSTEIN, J. & WOZNY, N. (2013). Permanent income and the black-white test score gap. Journal of Human Resources 48, 509–544.

SCHOCHET, P. Z. (2010). Is regression adjustment supported by the Neyman model for causal inference? Journal of Statistical Planning and Inference 140, 246–259.

SOLON, G., HAIDER, S. J. & WOOLDRIDGE, J. (2013). What are we weighting for? NBER Working Paper no. 18859.

VOGL, T. S. (2013). Marriage institutions and sibling competition: Evidence from South Asia. Quarterly Journal of Economics 128, 1017–1072.

WOOLDRIDGE, J. M. (2005). Fixed-effects and related estimators for correlated random-coefficient and treatment-effect panel data models. Review of Economics and Statistics 87, 385–390.

WOOLDRIDGE, J. M. (2010). Econometric Analysis of Cross Section and Panel Data. MIT Press, 2nd ed.

YITZHAKI, S. (1996). On using linear regressions in welfare economics. Journal of Business & Economic Statistics 14, 478–486.