Now, whose schools are really better (or weaker) than Germany's? A multiple testing approach

Hanck, Christoph (2008): Now, whose schools are really better (or weaker) than Germany's? A multiple testing approach.

Abstract

Using PIRLS (Progress in International Reading Literacy Study) data, we investigate which countries' schools can be be classified as significantly better or weaker than Germany's as regards the reading literacy of primary school children. The `standard' approach is to conduct separate tests for each country relative to the reference country (Germany) and to reject the null of equally good schools for all those countries whose $p$-value satisfies p_i< 0.05. We demonstrate that this approach ignores the multiple testing nature of the problem and thus overstates differences between schooling systems by producing unwarranted rejections of the null. We employ various multiple testing techniques to remedy this problem. The results suggest that the `standard' approach may overstate the number of significantly different countries by up to 30%.

• Item Type: MPRA Paper
• Language: English
• Keywords: PIRLS; Multiple Testing; Multi-Country Comparisons
• Subjects: C - Mathematical and Quantitative Methods > C1 - Econometric and Statistical Methods and Methodology: General > C12 - Hypothesis Testing: General; I - Health, Education, and Welfare > I2 - Education and Research Institutions > I21 - Analysis of Education

References

Benjamini, Yoav, and Yosef Hochberg, "Controlling the False Discovery Rate: A Practical and Powerful Approach to Multiple Testing," Journal of the Royal Statistical Society. Series B 57:1 (1995), 289-300.

Benjamini, Yoav, and Daniel Yekutieli, "The Control of the False Discovery Rate in Multiple Testing under Dependency," The Annals of Statistics 29:4 (2001), 1165-1188.

Bos, Wilfried, Sabine Hornberg, Karl-Heinz Arnold, Gabriele Faust, Lilian Fried, Eva-Maria Lankes, Knut Schwippert, and Renate Valtin (Eds.), IGLU 2006. Lesekompetenzen von Grundschulkindern in Deutschland im internationalen Vergleich (Münster: Waxmann, 2007).

Dudoit, Sandrine, and Mark J. van der Laan, Multiple Testing Procedures and Applications to Genomics, Springer Series in Statistics (Berlin: Springer, 2007).

Finner, Helmut, Thorsten Dickhaus, and Markus Roters, "On the False Discovery Rate and an Asymptotically Optimal Rejection Curve," The Annals of Statistics (to appear).

Hanushek, Eric A., and Ludger Wößmann, "Does Educational Tracking Affect Performance and Inequality? Differences-in-Differences Evidence Across Countries," The Economic Journal 116 (2006), C63-C76.

Holm, Sture, "A Simple Sequentially Rejective Multiple Test Procedure," Scandinavian Journal of Statistics 6:1 (1979), 65-70.

Hommel, Gerhard, "A Stagewise Rejective Multiple Tests Procedure Based on a Modified Bonferroni Test," Biometrika 75:2 (1988), 383-386.

J�urges, Hendrik, and Kerstin Schneider, "Fair ranking of teachers," Empirical Economics 32 (2007), 411-431.

Lehmann, Erich L., and Joseph P. Romano, Testing Statistical Hypotheses (New York: Springer, 2005), 3rd ed.

Mullis, Ina V.S., Michael O. Martin, and Ann M. Kennedy Pierre Foy, Progress in International Reading Literacy Study (Boston: TIMSS & PIRLS International Study Center, 2007).

Romano, Joseph P., and Michael Wolf, "Stepwise Multiple Testing as Formalized Data Snooping," Econometrica 73:4 (2005), 1237-1282.

Romano, Joseph P., and Michael Wolf, "Formalized Data Snooping Based on Generalized Error Rates," Econometric Theory 24 (2008), 404-447.

Sarkar, Sanat K., "Probability Inequalities for Ordered MTP2 Random Variables: A Proof of the Simes Conjecture," The Annals of Statistics 26:2 (1998), 494-504. Savin, N. Eugene, "Multiple Hypothesis Testing," in Z. Griliches, and M.D. Intriligator (Eds.), "Handbook of Econometrics," vol. 2, chap. 14 (Amsterdam: North-Holland Publishing, 1984), pp. 827-879.

Simes, R. John, "An Improved Bonferroni Procedure for Multiple Tests of Significance," Biometrika 73:3 (1986), 751-754.

Wößmann, Ludger, and Martin West, "Class-size effects in school systems around the world: Evidence from between-grade variation in TIMSS," European Economic Review 50 (2006), 695-736.