Assessing the quality of institutions’ rankings obtained through multilevel linear regression models

Arpino, Bruno and Varriale, Roberta (2009): Assessing the quality of institutions’ rankings obtained through multilevel linear regression models.

Abstract

The aim of this paper is to assess the quality of the ranking of institutions obtained with multilevel techniques in presence of different model misspecifications and data structures. Through a Monte Carlo simulation study, we find that it is quite hard to obtain a reliable ranking of the whole effectiveness distribution while, under various experimental conditions, it is possible to identify institutions with extreme performances. Ranking quality increases with increasing intra class correlation coefficient and/or overall sample size. Furthermore, multilevel models where the between and within cluster components of first-level covariates are distinguished perform significantly better than both multilevel models where the two effects are set to be equal and the fixed effect models.

• Item Type: MPRA Paper
• Language: English
• Keywords: Multilevel models, ranking of institutions, second-level residuals distribution
• Subjects: I - Health, Education, and Welfare > I2 - Education and Research Institutions; C - Mathematical and Quantitative Methods > C1 - Econometric and Statistical Methods and Methodology: General > C15 - Statistical Simulation Methods: General

References

Aassve, A. & Arpino, B. (2007). Dynamic Multi-Level Analysis of Households' Living Standards and Poverty: Evidence from Vietnam. Working Paper of Institute for Social and Economic Research, paper 2007-10. Colchester: University of Essex.

Adab, P., Rouse. A., Mohammed, M.A., & Marshall. T. (2002). Performance league tables: the NHS deserves better. BMJ, 324, 95–98.

Aitkin, M., & Longford, N. (1986). Statistical modelling in school effectiveness studies (with discussion). Journal of Royal Statistical Society A, 149, 1-42.

Chiandotto, C. & Varriale, R. (2005). Un modello multilivello per l’analisi dei tempi di conseguimento del titolo nell’Ateneo fiorentino, in C. Crocetta (Ed.), Modelli di analisi della transizione università-lavoro. Vol.7, Cleup, Padova.

Downey, D.B., von Hippel, P.T., &Hughes, M. (2008). Are “failing” schools really failing? Using seasonal comparison to evaluate school effectiveness. Sociology of Education, 81(3), 242-270

Goldstein, H. & Healy, M.J.R. (1995). The graphical presentation of a collection of means. Journal of the Royal Statistical Society, 158, 175-177.

Goldstein, H. & Spiegelhalter, D.J. (1996). League tables and their limitations: statistical issues in comparisons of institutional performance. Journal of the Royal Statistical Society: Series A, 159, 385-443.

Goldstein, H., Rasbash, J., Yang, M., Woodhouse, G., Pan, H., Nuttall, D., & Thomas, S. (1993). A multilevel analysis of school examination results. Oxford Review of Education, 19, 425-433.

Jürges, H, & Sneider, K. (2007). Fair ranking of teachers. Empirical Economics, 32, 411-431.

Ladd, H. F., & Walsh, R.P. (2002). Implementing value-added measures of school effectiveness: getting the incentives right. Economics of Education Review, 21, 1-17.

Leckie, G. & Goldstein, H. (2009). The Limitations of Using School League Tables to Inform School Choice. The Centre for Market and Public Organisation 09/208, Department of Economics, University of Bristol, UK.

Maas, C.J.M. and Hox. J.J. (2004). Robustness issues in multilevel regression analysis. Statistica Neerlandica, 58(2), 127–137.

Maddala, G.S. (1971). The use of variance components models in pooling cross section and time series data. Econometrica, 39 (2), 341-358.

Neuhaus, J.M. & Kalbfleisch, J.D. (1998). Between- and within-cluster covariate effects in the analysis of clustered data. Biometrics, 54, 638-645.

Rabe-Hesketh, S. & Skrondal, A. (2005). Multilevel and longitudinal modeling using Stata. Stata Press, College Station, TX.

Rampichini, C., Grilli, L., & Petrucci, A. (2004). Analysis of university course evaluations: from descriptive measures to multilevel models. Statistical Methods and Applications, 13(3), 357-373.

Raudenbush, S.W., & Willis, J.D. (1995). The estimation of school effects. Journal of Educational and Behavioral Statistics, 20, 307-335.

Shaw, C.D. & Costain, D.W. (1995). League tables for health care. Journal of the Royal Society of Medicine, 88, 54-57.

Skrondal, A. & Rabe-Hesketh, S. (2004). Generalized Latent Variable Modelling: Multilevel, Longitudinal and Structural Equation Models. Boca Raton, FL: Chapman & Hall/CRC.

Snijders, T., & Bosker, R.(1999). An Introduction to Basic and Advanced MultileveModeling. Sage, London.

Tekwe, C.D., Carter, R.L., Ma, C., Algina, J. Lucas, M.E., Roth, J. Ariet, M., Fisher, T., &

Resnick, M.B. (2004). An empirical comparison of statistical models for value-added assessment of school performance. Journal of Educational and Behavioral Statistics, 29 (1), 11-35

Wößmann, L. (2008). Efficiency and equity of European education and training policies. International Tax and Public Finance, 15, 199–230