Causal Inference on Education Policies: A Survey of Empirical Studies Using PISA, TIMSS and PIRLS

Cordero, José Manuel and Cristobal, Victor and Santín, Daniel (2017): Causal Inference on Education Policies: A Survey of Empirical Studies Using PISA, TIMSS and PIRLS.

Abstract

The identification of the causal effects of educational policies is the top priority in recent education economics literature. As a result, a shift can be observed in the strategies of empirical studies. They have moved from the use of standard multivariate statistical methods, which identify correlations or associations between variables only, to more complex econometric strategies, which can help to identify causal relationships. However, exogenous variations in databases have to be identified in order to apply causal inference techniques. This is a far from straightforward task. For this reason, this paper provides an extensive and comprehensive overview of the literature using quasi-experimental techniques applied to three well-known international large-scale comparative assessments, such as PISA, PIRLS or TIMSS, over the period 2004-2016. In particular, we review empirical studies employing instrumental variables, regression discontinuity designs, difference in differences and propensity score matching to the above databases. Additionally, we provide a detailed summary of estimation strategies, issues treated and profitability in terms of the quality of publications to encourage further potential evaluations. The paper concludes with some operational recommendations for prospective researchers in the field.

• Item Type: MPRA Paper
• Language: English
• Keywords: Survey, Education, Causal Inference, Selection-bias, International assessments.
• Subjects: C - Mathematical and Quantitative Methods > C4 - Econometric and Statistical Methods: Special Topics > C40 - General; C - Mathematical and Quantitative Methods > C5 - Econometric Modeling > C55 - Large Data Sets: Modeling and Analysis; I - Health, Education, and Welfare > I2 - Education and Research Institutions > I21 - Analysis of Education

References

Agasisti, T. and Murtinu, S. (2012). ‘Perceived’ competition and performance in Italian secondary schools: new evidence from OECD–PISA 2006. British Educational Research Journal, 38(5): 841-858.

Ammermüller, A. (2012). Institutional features of schooling systems and educational inequality: Cross‐Country evidence from PIRLS and PISA. German Economic Review, 14(2): 190-213.

Ammermüller, A. and Dolton, P. (2006). Pupil-teacher gender interaction effects on scholastic outcomes in England and the USA, ZEW Discussion Paper 06-060. Mannheim, Germany: Centre for European Economic Research.

Ammermueller, A. and Pischke, J. S. (2009). Peer effects in European primary schools: Evidence from the progress in international reading literacy study, Journal of Labor Economics, 27(3): 315-348.

Anghel, B., Cabrales, A., Sainz, J. and Sanz, I. (2015). Publicizing the results of standardized external tests: does it have an effect on school outcomes?. IZA Journal of European Labor Studies, 4(1), 1.

Angrist J.D. and Lavy V. (1999). Using Maimonides’ rule to estimate the effect of class size on scholastic achievement, Quarterly Journal of Economics, 114(2): 533-575.

Angrist, J. D. and Pischke, J. S. (2008). Mostly harmless econometrics: An empiricist's companion. Princeton university press.

Angrist, J. D. and Pischke, J. S. (2014). Mastering metrics: the path from cause to effect. Princeton University Press.

Arikan, S., van de Vijver, F. and Yagmur, K. (2016). Factors contributing to mathematics achievement differences of Turkish and Australian Students in TIMSS 2007 and 2011. Eurasia Journal of Mathematics, Science and Technology Education, 12: 2039-2059.

Bedard, K. and Dhuey, E. (2006). The persistence of early childhood maturity: International evidence of long-run age effects, The Quarterly Journal of Economics, 121(4): 1437-1472.

Bietenbeck, J. (2014). Teaching practices and cognitive skills. Labour Economics, 30: 143-153.

Blundell, R. and Dias, M. C. (2009). Alternative approaches to evaluation in empirical microeconomics. Journal of Human Resources, 44(3): 565-640.

Cattaneo, M.A., Oggenfuss, C. and Wolter, S.C. (2016). The more, the better? The impact of instructional time on student performance, CESIFO Working Paper, 5813.

Cho, I. (2012). The effect of teacher–student gender matching: Evidence from OECD countries. Economics of Education Review, 31(3): 54-67.

Choi, Á., Calero, J. and Escardíbul, J. O. (2012). Private tutoring and academic achievement in Korea: An approach through PISA-2006, KEDI Journal of Educational Policy, 9(2): 299-302.

Clotfelter, C.T., Ladd, H.F. and Vigdor, J.L. (2007). Teacher credentials and student achievement: Longitudinal analysis with student fixed effects, Economics of Education Review, 26(6): 673–82.

Cornelisz, I. (2013). Relative Private School Effectiveness in the Netherlands: A Reexamination of PISA 2006 and 2009 data, Procedia Economics and Finance, 5: 192-201.

Creemers, B. P. M. and Kyriakides, L. (2008). The dynamics of educational effectiveness, London, Rouledge.

Crespo-Cebada, E., Pedraja-Chaparro, F. and Santín, D. (2014). Does school ownership matter? An unbiased efficiency comparison for regions of Spain. Journal of Productivity Analysis, 41(1): 153-172.

De Witte, K. and López-Torres, L. (2015). Efficiency in education: a review of literature and a way forward. Journal of the Operational Research Society, forthcoming.

Denny, K. and Oppedisano, V. (2013). The surprising effect of larger class sizes: Evidence using two identification strategies, Labour Economics, 23: 57-65.

Dronkers, J. and Avram, S. (2010). A cross-national analysis of the relations of school choice and effectiveness differences between private-dependent and public schools. Educational Research and Evaluation, 16(2): 151-175.

Edwards, S. and Marin, A. G. (2015). Constitutional rights and education: An international comparative study, Journal of Comparative Economics, 43(4): 938-955.

Falck, O. and Woessmann, L. (2013). School competition and students’ entrepreneurial intentions: International evidence using historical Catholic roots of private schooling, Small Business Economics, 40(2): 459-478.

Felfe, C., Nollenberger, N. and Rodríguez-Planas, N. (2015). Can’t buy mommy’s love? Universal childcare and children’s long-term cognitive development, Journal of Population Economics, 28(2): 393-422.

Fisher, R. A. (1925). Statistical methods for research workers. London: Oliver and Boyd.

Gamboa, L., Rodríguez, M. and García, A. (2013). Differences in motivations and academic achievement. Lecturas de Economía, 78: 9-44.

García-Pérez, J. I., Hidalgo-Hidalgo, M. and Robles-Zurita, J. A. (2014). Does grade retention affect students’ achievement? Some evidence from Spain. Applied Economics, 46(12): 1373-1392.

Gee, K. and Cho, R.M. (2014). The effects of single-sex versus coeducational schools on adolescent peer victimization and perpetration. Journal of Adolescence, 3: 1237-1251.

Gertler, P. J., Martinez, S., Premand, P., Rawlings, L. B. and Vermeersch, C. M. (2016). Impact evaluation in practice. 2nd edition, World Bank Publications.

Green, A. D. and Pensiero, N. (2016). The effects of upper secondary education and training systems on skills inequality: A quasi-cohort analysis using PISA 2000 and the OECD survey of adult skills. British Educational Research Journal, 42(5): 756-779.

Gustafsson, J. E. (2007). Understanding causal influences on educational achievement through analysis of differences over time within countries. In T. Loveless (Ed.), Lessons learned: What international assessments tell us about math achievement, (pp. 37–63), Washington, DC: Brookings,.

Gustafsson, J. E. (2008). Effects of international comparative studies on educational quality on the quality of educational research. European Educational Research Journal, 7(1): 1-17.

Gustafsson, J.E. (2013). Causal inference in educational effectiveness research: a comparison of three methods to investigate effects of homework on student achievement, School effectiveness and School Improvement, 24(3): 275-295.

Hanushek, E. A. (1979). Conceptual and empirical issues in the estimation of educational production functions. Journal of Human Resources, 14(3): 351-388.

Hanushek, E.A., Link, S. and Woessman, L. (2013). Does school autonomy make sense everywhere? Panel estimates from PISA, Journal of Development Economics, 104: 212-232.

Hanushek, E. A., Piopiunik, M. and Wiederhold, S. (2014). The value of smarter teachers: International evidence on teacher cognitive skills and student performance, National Bureau of Economic Research, Working Paper nº 20727.

Hanushek, E. A. and Woessmann, L. (2006). Does educational tracking affect performance and inequality? Differences-in-differences evidence across countries. Economic Journal, 116: 63–76.

Hanushek, E. A. and Woessman, L. (2011). The economics of international differences in educational achievement. In Hanushek, E.A., Machin, S., Woessmann, L. (Eds.), Handbook of the economics of education (vol. 3), (pp. 89–200). Amsterdam: North Holland.

Hanushek, E. A. and Woessmann, L. (2014). Institutional Structures of the Education System and Student Achievement: A Review of Cross-country Economic Research. Educational Policy Evaluation through International Comparative Assessments, 145.

Heckman, J. J. (1976). The common structure of statistical models of truncation, sample selection and limited dependent variables and a simple estimator for such models. Annals of Economic and Social Measurement, 5: 475–492.

Heckman, J. (1979). Sample selection bias as an especification error, Econometrica, 47: 153-161.

Heckman, J. and Navarro, S. (2004). Using matching, instrumental variables, and control functions to estimate economic choice models. Review of Economics and Statistics, 86(1): 30-57.

Hogrebe, N. and Strietholt, R. (2016). Does non-participation in preschool affect children’s reading achievement? International evidence from propensity score analyses. Large-scale Assessments in Education, 4(8), in press.

Holland, P. W. (1986). Statistics and causal inference (with discussion). Journal of the American Statistical Association, 81: 945–960.

Imai, K. and Van Dyk, D. A. (2004). Causal inference with general treatment regimes: Generalizing the propensity score, Journal of the American Statistical Association, 99(467): 854–866.

Imbens, G.W. (2000). The role of the propensity score in estimating dose-response functions. Biometrika, 87(3): 706–710.

Imbens, G. W. and Lemieux, T. (2008). Regression discontinuity designs: A guide to practice, Journal of Econometrics, 142(2): 615-635.

Isphording, I. E., Piopiunik, M. and Rodríguez-Planas, N. (2016). Speaking in numbers: The effect of reading performance on math performance among immigrants, Economics Letters, 139: 52-56.

Jakubowski, M. (2010). Institutional Tracking and Achievement Growth: Exploring Difference-in-Differences Approach to PIRLS, TIMSS, and PISA Data. In Dronkers (ed.), Quality and Inequality of Education, (pp. 41-81), Springer Netherlands.

Jakubowski, M. (2015). Latent variables and propensity score matching: a simulation study with application to data from the Programme for International Student Assessment in Poland, Empirical Economics, 48(3): 1287-1325.

Jensen, P. and Rasmussen, A. W. (2011). The effect of immigrant concentration in schools on native and immigrant children's reading and math skills, Economics of Education Review, 30(6): 1503-1515.

Jiang, F. and McComas, W. F. (2015). The effects of inquiry teaching on student science achievement and attitudes: Evidence from propensity score analysis of PISA data, International Journal of Science Education, 37(3): 554-576.

Johansson, S. (2016). International large-scale assessments: what uses, what consequences?, Educational Research, 1-10. forthcoming.

Jürges, H. and Schneider, K. (2004). International differences in student achievement: An economic perspective, German Economic Review, 5(3): 357-380.

Jürges, H., Schneider, K. and Büchel, F. (2005). The effect of central exit examinations on student achievement: Quasi‐experimental evidence from TIMSS Germany. Journal of the European Economic Association, 3(5): 1134-1155.

Kamens, D.H. (2009). Globalization and the growth of international educational testing and national assessment, Comparative Education Review, 54(1): 5-25.

Kaplan, D. (2016). Causal inference with large-scale assessments in education from a Bayesian perspective: a review and synthesis. Large-scale Assessments in Education, 4(1): 1-24.

Khandker, S. R., Koolwal, G. B. and Samad, H. A. (2010). Handbook on impact evaluation: quantitative methods and practices. World Bank Publications.

Kiss, D. (2013). Are immigrants and girls graded worse? Results of a matching approach. Education Economics, 21(5): 447-463.

Klieme, E. (2013). The role of large-scale assessments in research on educational effectiveness and school development, in Von Davier, M., Gonzalez, E. and Kirsch, I. (eds.). The role of international large-scale assessments: perspectives from technology, economy, and educational research (pp. 115-147), Springer Netherlands.

Konstantopoulos, S. and Shen, T. (2016) Class size effects on mathematics achievement in Cyprus: evidence from TIMSS. Education Research and Evaluation, 22: 86–109.

Konstantopoulos, S. and Traynor, A. (2014). Class Size Effects on Reading Achievement Using PIRLS Data: Evidence from Greece. Teachers College Record, 116(2), n2.

Kuzmina, J. and Carnoy, M. (2016). The effectiveness of vocational versus general secondary education: Evidence from the PISA 2012 for countries with early tracking, International Journal of Manpower, 37(1): 2-24.

Lavrijsen, J. and Nicaise, I. (2015). New empirical evidence on the effect of educational tracking on social inequalities in reading achievement. European Educational Research Journal, 14(3-4): 206-221.

Lavrijsen, J. and Nicaise, I. (2016). Educational tracking, inequality and performance: New evidence from a differences-in-differences technique. Research in Comparative and International Education, 11(3): 334-349.

Lavy, V. (2015). Do differences in schools' instruction time explain international achievement gaps? Evidence from developed and developing countries. The Economic Journal, 125(588): 397-424.

Lee, B. (2014). The influence of school tracking systems on educational expectations: a comparative study of Austria and Italy, Comparative Education, 50(2): 206-228.

Lee, J. and Fish, R. (2010). International and interstate gaps in value-added math achievement: Multilevel instrumental variable analysis of age effect and grade effect. American Journal of Education, 117(1): 109-137.

Li, W. and Konstantopoulos, S. (2016). Class Size Effects on Fourth Grade Mathematics Achievement: Evidence From TIMSS 2011. Journal of Research on Educational Effectiveness, in press.

Luyten, H. (2006). An empirical assessment of the absolute effect of schooling: regression-discontinuity applied to TIMSS-95. Oxford Review of Education, 32(3): 397–429.

Luyten, H., Peschar, J. and Coe, R. (2008). Effects of Schooling on Reading Performance, Reading Engagement, and Reading Activities of 15-Year-Olds in England, American Educational Research Journal, 45(2): 319-342.

Luyten, H. and Veldkamp, B. (2011). Assessing effects of schooling with cross-sectional data: Between-grades differences addressed as a selection-bias problem. Journal of Research on Educational Effectiveness, 4(3): 264-288.

McCaffrey, D. F., Lockwood, J. R., Koretz, D. M. and Hamilton, L. S. (2003). Evaluating Value-Added Models for Teacher Accountability. Santa Mónica, CA: The RAND Corporation.

Morgan, S. L. and Winship, C. (2007). Counterfactuals and causal inference. Methods and principles for social research. Cambridge: University Press.

Neyman, J. (1923). Statistical problems in agriculture experiments. Journal of the Royal Statistical Society, Series B, 2: 107–180.

Pedraja-Chaparro, F., Santín, D. and Simancas, R. (2016). The impact of immigrant concentration in schools on grade retention in Spain: a difference-in-differences approach. Applied Economics, 48(21): 1978-1990.

Perelman, S. and Santin, D. (2011). Measuring educational efficiency at student level with parametric stochastic distance functions: an application to Spanish PISA results. Education Economics, 19(1): 29-49.

Pfeffermann, D. and Landsman, V. (2011). Are private schools better than public schools? Appraisal for Ireland by methods for observational studies. The Annals of Applied Statistics, 5(3): 1726-1751.

Piopiunik, M. (2014). The effects of early tracking on student performance: Evidence from a school reform in Bavaria. Economics of Education Review, 42: 12–33.

Pokropek, A. (2016). Introduction to instrumental variables and their application to large-scale assessment data. Large-scale Assessments in Education, 4(1), 1.

Ponzo, M. (2013). Does bullying reduce educational achievement? An evaluation using matching estimators. Journal of Policy Modeling, 35: 1057–1078.

Ponzo, M. and Scoppa, V. (2014). The long-lasting effects of school entry age: Evidence from Italian students, Journal of Policy Modeling, 36(3): 578-599.

Puhani, P. A. and Weber, A. M. (2008). Does the early bird catch the worm?. In Dutsman, C., Fitzenberg, B. and Machin, S. (Eds.). The economics of education and training (pp. 105-132), Physica-Verlag HD.

Rivkin, S.G., Hanushek, E.A. and Kain, J.F. (2005). Teachers, schools, and academic achievement, Econometrica, 73 (2): 417–58.

Rivkin, S. G. and Schiman, J. C. (2015). Instruction time, classroom quality, and academic achievement. The Economic Journal, 125(588): 425-448.

Robinson, J.P. (2014). Causal Inference and Comparative Analysis with Large-Scale Assessments. In Rutkowski, L., von Davier, M. and Rutkowski, D. (eds.). Handbook of International Large-Scale Assessment: Background, Technical Issues, and Methods of Data Analysis, Chapman & Hall/CRC Statistics in the Social and Behavioral Sciences, Boca Raton.

Rosenbaum, P. R. and Rubin, D. B. (1983). The central role of the propensity score in observational studies for causal effects. Biometrika, 70(1): 41-55.

Rosenbaum, P. R. (1986). Dropping out of high school in the United States: An observational study. Journal of Educational Statistics, 11(3): 207–224.

Rosen, M. and Gustafsson, J. E. (2016). Is computer availability at home causally related to reading achievement in grade 4? A longitudinal difference in differences approach to IEA data from 1991 to 2006. Large-scale Assessments in Education, 4(1), 1.

Rossi, P. H., Freeman, H. E. and Lipsey, M. W. (2004). Evaluation, a systematic approach, Thousand Oaks/London/New Delhi, SAGE Publications.

Rubin, D. B. (1974). Estimating causal effects of treatments in randomized and nonrandomized studies. Journal of Educational Psychology, 66: 688–701.

Rubin, D. B. (2008). Objective causal inference, design trumps analysis. The Annals of Applied Statistics, 2(3): 808–840.

Ruhose, J. and Schwerdt, G. (2015). Does Early Educational Tracking Increase Migrant-Native Achievement Gaps? Differences-In-Differences Evidence Across Countries, CESIFO Working Paper No. 5248.

Rutkowski, L., von Davier, M. and Rutkowski, D. (2014). Handbook of International Large-Scale Assessment: Background, Technical Issues, and Methods of Data Analysis, Chapman & Hall/CRC Statistics in the Social and Behavioral Sciences, Boca Raton.

Rutkowski, D. and Delandshere, G. (2016). Causal Inferences with Large Scale Assessment Data: Using a Validity Framework. Large-Scale Assessments in Education, 4(1): 1–18. doi:10.1186/s40536-016-0019-1.

Schlotter, M., Schwerdt, G. and Woessmann, L. (2011). Econometric methods for causal evaluation of education policies and practices: a non‐technical guide. Education Economics, 19(2): 109-137.

Schneeweis, N. and Winter-Ebmer, R. (2008). Peer effects in Austrian schools, Empirical Economics, 32: 387-409.

Schneider, B., Carnoy, M., Kilpatrick, J., Schmidt, W. H. and Shavelson, R. J. (2007). Estimating causal effects using experimental and observational designs. Washington, DC: American Educational Research Association.

Schütz, G. (2009). Does the quality of pre-primary education pay off in secondary school? An international comparison using PISA 2003. Ifo Institute for Economic Research, Ifo Working Paper, 68.

Schwerdt, G. and Wuppermann, A. (2011). Is traditional teaching really all that bad? A within-student between-subject approach. Economics of Education Review, 30: 365–379.

Shadish, W.R., Cook, T.D. and Campbell, D.T. (2002). Experimental and quasi-experimental designs for generalized causal inference. Boston: Houghton Mifflin.

Strietholt, R., Gustafsson, J. E., Rosen, M., and Bos, W. (2014). Outcomes and Causal Inference in International Comparative Assessments. In Stietholt, R., Bos, W., Gustafsson, J.E., Rosen, M. (eds.), Educational Policy Evaluation through International Comparative Assessments, Waxman, Münster, New York.

Stuart, E. A. (2007). Estimating causal effects using school-level data sets. Educational Researcher, 36(4): 187-198.

Tiumeneva, Y. A. and Kuzmina, J. V. (2015). The Difference That One Year of Schooling Makes for Russian Schoolchildren: Based on PISA 2009: Reading. Russian Education & Society, 57(4): 214-253.

Todd, P. E. and Wolpin, K. I. (2003). On the specification and estimation of the production function for cognitive achievement. The Economic Journal, 113 (485): 3-33.

Vardardottir, A. (2015). The impact of classroom peers in a streaming system. Economics of Education Review, 49: 110-128.

Vandenberghe, V. and Robin, S. (2004). Evaluating the effectiveness of private education across countries: a comparison of methods. Labour Economics, 11(4): 487-506.

Wilde, E. T. and Hollister, R. (2007). How close is close enough? Evaluating propensity score matching using data from a class size reduction experiment, Journal of Policy Analysis and Management, 26(3): 455-477.

Webbink, D. (2005). Causal effects in education. Journal of Economic Surveys, 19(4): 535-560.

West, M.R. and Woessmann, L. (2006). Which School Systems Sort Weaker Students into Smaller Classes? International Evidence, European Journal of Political Economy, 22 (4): 944-968.

West, M. R. and Woessmann, L. (2010). ‘Every Catholic Child in a Catholic School’: Historical Resistance to State Schooling, Contemporary Private Competition and Student Achievement across Countries. The Economic Journal, 120(546): 229-255.

Woessmann, L. (2005). Educational production in Europe, Economic Policy, 20(43): 446-504.

Woessmann, L. and West, M. (2006). Class-size effects in school systems around the world: Evidence from between-grade variation in TIMSS. European Economic Review, 50(3): 695-736.

Woessmann, L. (2007). International evidence on school competition, autonomy, and accountability: A review. Peabody Journal of Education, 82(2-3): 473-497.

Wooldridge, J. M. (2010). Econometric analysis of cross section and panel data. MIT press.