Multiple Testing and the Distributional Effects of Accountability Incentives in Education

Lehrer, Steven F. and Pohl, R. Vincent and Song, Kyungchul (2018): Multiple Testing and the Distributional Effects of Accountability Incentives in Education.

Abstract

Economic theory that underlies many empirical microeconomic applications predicts that treatment responses depend on individuals’ characteristics and location on the outcome distribution. Using data from a large-scale Pakistani school report card experiment, we consider tests for treatment effect heterogeneity that make corrections for multiple testing to avoid an overestimation of positive treatment effects. These tests uncover evidence of policy-relevant heterogeneous effects from information provision on child test scores. Further, our analysis reinforces the importance of preventing the inflation of false positive conclusions since over 65% of the estimated statistically significant quantile treatment effects become insignificant once these corrections are applied.

• Item Type: MPRA Paper
• Language: English
• Keywords: information, student performance, quantile treatment effects, multiple testing, bootstrap tests
• Subjects: C - Mathematical and Quantitative Methods > C1 - Econometric and Statistical Methods and Methodology: General > C12 - Hypothesis Testing: General; C - Mathematical and Quantitative Methods > C2 - Single Equation Models ; Single Variables > C21 - Cross-Sectional Models ; Spatial Models ; Treatment Effect Models ; Quantile Regressions; I - Health, Education, and Welfare > I2 - Education and Research Institutions > I21 - Analysis of Education; L - Industrial Organization > L1 - Market Structure, Firm Strategy, and Market Performance > L15 - Information and Product Quality ; Standardization and Compatibility

References

Abadie, Alberto. 2002. “Bootstrap Tests for Distributional Treatment Effects in Instrumental Variable Models.” Journal of the American Statistical Association 97 (457):284–292.

Abadie, Alberto, Joshua Angrist, and Guido Imbens. 2002. “Instrumental Variables Estimates of the Effect of Subsidized Training on the Quantiles of Trainee Earnings.” Econo-metrica 70 (1):91–117.

Andrabi, Tahir, Jishnu Das, and Asim Ijaz Khwaja. 2017. “Report Cards: The Impact of Providing School and Child Test Scores on Educational Markets.” American Economic Review 107 (6):1535–63.

Armstrong, Timothy B. and Shu Shen. 2015. “Inference on Optimal Treatment Assignments.” Cowles Foundation Discussion Paper 1927RR.

Becker, Gary S. 1995. Human Capital and Poverty Alleviation. World Bank, Human Resources Development and Operations Policy.

Bitler, Marianne P., Jonah B. Gelbach, and Hilary W. Hoynes. 2006. “What Mean Impacts Miss: Distributional Effects of Welfare Reform Experiments.” American Economic Review 96 (4):988–1012.

———. 2017. “Can Variation in Subgroups’ Average Treatment Effects Explain Treatment Effect Heterogeneity? Evidence from a Social Experiment.” Review of Economics and Statistics 99 (4):683–697.

Brinch, Christian N, Magne Mogstad, and Matthew Wiswall. 2017. “Beyond LATE with a discrete instrument.” Journal of Political Economy 125 (4):985–1039.

Camargo, Braz, Rafael Camelo, Sergio Firpo, and Vladimir Ponczek. 2014. “Information, Market Incentives, and Student Performance.” IZA Discussion Paper 7941.

———. 2018. “Information, Market Incentives, and Student Performance Evidence from a Regression Discontinuity Design in Brazil.” Journal of Human Resources 53 (2):414–444.

Carneiro, Pedro, Jishnu Das, and Hugo Reis. 2013. “Parental valuation of school attributes in developing countries: Evidence from Pakistan.” Unpublished manuscript.

Chernozhukov, Victor, Ivan Fernandez-Val, and Blaise Melly. 2013. “Inference on Counterfactual Distributions.” Econometrica 81 (6):2205–2268.

Crump, Richard K., V. Joseph Hotz, Guido W. Imbens, and Oscar A. Mitnik. 2008. “Non- parametric Tests for Treatment Effect Heterogeneity.” Review of Economics and Statistics 90 (3):389–405.

Deaton, Angus. 2010. “Instruments, Randomization, and Learning about Development.” Journal of Economic Literature 48 (2):424–455.

Dehejia, Rajeev H. 2005. “Program Evaluation as a Decision Problem.” Journal of Econometrics 125 (1-2):141–173.

Ding, Weili and Steven F Lehrer. 2007. “Do peers affect student achievement in China’s secondary schools?” The Review of Economics and Statistics 89 (2):300–312.

Djebbari, Habiba and Jeffrey Smith. 2008. “Heterogeneous Impacts in PROGRESA.” Journal of Econometrics 145 (1-2):64–80.

Figlio, David and Susanna Loeb. 2011. “School accountability.” In Handbook of the Economics of Education, vol. 3. Elsevier, 383–421.

Fink, Gunther, Margaret McConnell, and Sebastian Vollmer. 2014. “Testing for Heterogeneous Treatment Effects in Experimental Data: False Discovery Risks and Correction Procedures.” Journal of Development Effectiveness 6 (1):44–57.

Firpo, Sergio. 2007. “Efficient Semiparametric Estimation of Quantile Treatment Effects.” Econometrica 75 (1):259–276.

Friedlander, Daniel and Philip K. Robins. 1997. “The Distributional Impacts of Social Programs.” Evaluation Review 21 (5):531–553.

Friedman, Milton. 1955. The Role of Government in Education. Rutgers University Press New Brunswick, NJ.

Friesen, Jane, Mohsen Javdani, Justin Smith, and Simon Woodcock. 2012. “How do school report cards affect school choice decisions?” Canadian Journal of Economics/Revue canadienne d’ ́economique 45 (2):784–807.

Gibbons, Stephen and Stephen Machin. 2006. “Paying for primary schools: admission constraints, school popularity or congestion?” The Economic Journal 116 (510):C77–C92.

Gu, Jiaying and Shu Shen. 2017. “Oracle and Adaptive False Discovery Rate Controlling Method for One-Sided Testing: Theory and Application in Treatment Effect Evaluation.” The Econometrics Journal 21 (1):11–35.

Hastings, Justine, Thomas J Kane, and Douglas O Staiger. 2009. “Heterogeneous preferences and the efficacy of public school choice.” NBER Working Paper 12145.

Hastings, Justine S and Jeffrey M Weinstein. 2008. “Information, school choice, and academic achievement: Evidence from two experiments.” The Quarterly Journal of Economics 123 (4):1373–1414.

Heckman, James J. 2001. “Micro Data, Heterogeneity, and the Evaluation of Public Policy: Nobel Lecture.” Journal of Political Economy 109 (4):673–748.

Heckman, James J., Jeffrey Smith, and Nancy Clements. 1997. “Making The Most Out Of Programme Evaluations and Social Experiments: Accounting For Heterogeneity in Programme Impacts.” Review of Economic Studies 64 (4):487–535.

Heckman, James J. and Sergio Urzua. 2010. “Comparing IV with Structural Models: What Simple IV Can and Cannot Identify.” Journal of Econometrics 156 (1):27–37.

Heckman, James J. and Edward Vytlacil. 2005. “Structural Equations, Treatment Effects, and Econometric Policy Evaluation.” Econometrica 73 (3):669–738.

Holmstrom, Bengt. 1999. “Managerial incentive problems: A dynamic perspective.” The Review of Economic Studies 66 (1):169–182.

Hoxby, Caroline M. 2003. “School choice and school productivity. Could school choice be a tide that lifts all boats?” In The Economics of School Choice. University of Chicago Press, 287–342.

Imbens, Guido W. 2010. “Better LATE than Nothing: Some Comments on Deaton (2009) and Heckman and Urzua (2009).” Journal of Economic Literature 48 (2):399–423.

Kitagawa, Toru and Aleksey Tetenov. 2018. “Who Should Be Treated? Empirical Welfare Maximization Methods for Treatment Choice.” Econometrica 86 (2):591–616.

Koning, Pierre and Karen Van der Wiel. 2012. “School responsiveness to quality rankings: An empirical analysis of secondary education in the netherlands.” De Economist 160 (4):339– 355.

Lee, Soohyung and Azeem M. Shaikh. 2014. “Multiple Testing and Heterogeneous Treatment Effects: Re-Evaluating the Effect of PROGRESA on School Enrollment.” Journal of Applied Econometrics 29 (4):612–626.

List, John A., Azeem M. Shaikh, and Yang Xu. 2016. “Multiple Hypothesis Testing in Experimental Economics.” NBER Working Paper 21875.

Maier, Michael. 2011. “Tests For Distributional Treatment Effects Under Unconfoundedness.” Economics Letters 110 (1):49–51.

Manski, Charles F. 2004. “Statistical Treatment Rules for Heterogeneous Populations.” Econometrica 72 (4):1221–1246.

Milgrom, Paul and John Roberts. 1986. “Price and advertising signals of product quality.” Journal of Political Economy 94 (4):796–821.

Mizala, Alejandra and Miguel Urquiola. 2013. “School markets: The impact of information approximating schools’ effectiveness.” Journal of Development Economics 103:313–335.

Romano, Joseph P. and Azeem M. Shaikh. 2010. “Inference for the Identified Set in Partially Identified Econometric Models.” Econometrica 78 (1):169–211.

Romano, Joseph P., Azeem M. Shaikh, and Michael Wolf. 2010. “Hypothesis Testing in Econometrics.” Annual Review of Economics 2 (1):75–104.

Romano, Joseph P. and Michael Wolf. 2005a. “Exact and Approximate Stepdown Meth- ods for Multiple Hypothesis Testing.” Journal of the American Statistical Association 100 (469):94–108.

———. 2005b. “Stepwise Multiple Testing as Formalized Data Snooping.” Econometrica 73 (4):1237–1282.

Rothe, Christoph. 2010. “Nonparametric Estimation of Distributional Policy Effects.” Journal of Econometrics 155 (1):56–70.

Schneider, Mark, Gregory Elacqua, and Jack Buckley. 2006. “School choice in Chile: Is it class or the classroom?” Journal of Policy Analysis and Management 25 (3):577–601.

Shapiro, Carl. 1983. “Premiums for high quality products as returns to reputations.” The Quarterly Journal of Economics 98 (4):659–679.

Smith, Jeffrey A. and Petra E. Todd. 2005. “Does Matching Overcome LaLonde’s Critique of Nonexperimental Estimators?” Journal of Econometrics 125 (1-2):305–353.

Solon, Gary, Steven J. Haider, and Jeffrey M. Wooldridge. 2015. “What Are We Weighting For?” Journal of Human Resources 50 (2):301–316.

White, Halbert. 2000. “A Reality Check for Data Snooping.” Econometrica 68 (5):1097– 1126.

Wolinsky, Asher. 1983. “Prices as signals of product quality.” The Review of Economic Studies 50 (4):647–658.