Are Public Schools Ready to Integrate Math Classes with Khan Academy?

Ferman, Bruno and Finamor, Lucas and Lima, Lycia (2019): Are Public Schools Ready to Integrate Math Classes with Khan Academy?

Abstract

We study the impacts of the program Khan Academy in Schools using a randomized control trial in Brazilian primary public schools. Once a week, teachers would take their students to the school's computer lab and teach using the computer-assisted learning platform, instead of their standard math classes. We find positive effects of the program on measures of attitudes towards math, which were not translated to a positive average treatment effect on students' math proficiency. We explore treatment heterogeneity by quality of implementation. This provides suggestive evidence that the program may have positive effects when there are no infrastructure problems and when the implementation modality is based on one computer per student. These results highlight the implementation challenges associated with educational tech-interventions in developing countries and help explain the mixed results found in the literature.

• Item Type: MPRA Paper
• Language: English
• Keywords: Computer-aided learning, Education Technology, Program Implementation
• Subjects: C - Mathematical and Quantitative Methods > C9 - Design of Experiments > C93 - Field Experiments; I - Health, Education, and Welfare > I2 - Education and Research Institutions > I21 - Analysis of Education; J - Labor and Demographic Economics > J2 - Demand and Supply of Labor > J24 - Human Capital ; Skills ; Occupational Choice ; Labor Productivity

References

Adams, D. L. (2016). Action research on the effectiveness of Khan Academy as a tier I intervention. PhD thesis, Miami University.

Aiken Jr, L. R. and Dreger, R. M. (1961). The effect of attitudes on performance in mathematics. Journal of Educational Psychology, 52(1):19.

Banerjee, A. V., Cole, S., Duflo, E., and Linden, L. (2007). Remedying education: Evidence from two randomized experiments in india. The Quarterly Journal of Economics, 122(3):1235–1264.

Bettinger, E., Fairlie, R. W., Kapuza, A., Kardanova, E., Loyalka, P., and Zakharov, A. (2020). Does edtech substitute for traditional learning? experimental estimates of the educational production function. Technical report, National Bureau of Economic Research.

Brito, M. R. F. (1998). Adaptação e validação de uma escala de atitudes em relacção a matemática. Zetetik ́e: Revista de Educaçao Matemática, 6(9).

Buchel, K., Jakob, M., Kuhnhanss, C., Steffen, D., and Brunetti, A. (2019). Expanding school time and the value of computer-assisted learning: Evidence from a randomized controlled trial in el salvador. Technical report.

Bulman, G. and Fairlie, R. W. (2016). Technology and education: Computers, software, and the internet. In Handbook of the Economics of Education, volume 5, pages 239–280. Elsevier.

Carrillo, P. E., Onofa, M., and Ponce, J. (2011). Information technology and student achievement: Evidence from a randomized experiment in ecuador.

Chu, L., Nautiyal, A., Rais, S., and Yamtich, H. (2018). Does khan academy work?

de Chaisemartin, C. and Ramirez-Cuellar, J. (2019). At What Level Should One Cluster Standard Errors in Paired Experiments, and in Stratified Experiments with Small Strata? arXiv e-prints, page arXiv:1906.00288.

Duflo, E., Banerjee, A., Finkelstein, A., Katz, L. F., Olken, B. A., and Sautmann, A. (2020). In praise of moderation: Suggestions for the scope and use of pre-analysis plans for rcts in economics. Working Paper 26993, National Bureau of Economic Research.

Ferman, B. (2019). A simple way to assess inference methods. arXiv e-prints, page arXiv:1912.08772.

Glewwe, P. and Muralidharan, K. (2016). Improving Education Outcomes in Developing Countries: Evidence, Knowledge Gaps, and Policy Implications, volume 5. Elsevier B.V., 1 edition.

Kelly, D. P. and Rutherford, T. (2017). Khan academy as supplemental instruction: A controlled study of a computer-based mathematics intervention. The International Review of Research in Open and Distributed Learning, 18(4).

Lai, F., Luo, R., Zhang, L., Huang, X., and Rozelle, S. (2015). Does computer-assisted learning improve learning outcomes? evidence from a randomized experiment in mi- grant schools in beijing. Economics of Education Review, 47:34–48.

Linden, L. L. (2008). Complement or substitute?: The effect of technology on student achievement in India. InfoDev Working Paper, Columbia University.

Manaus (2016). Assessing the use of technology and khan academy to improve educational outcomes in sacatepequez, guatemala. Technical report.

Mo, D., Zhang, L., Wang, J., Huang, W., Shi, Y., Boswell, M., and Rozelle, S. (2013). The persistence of gains in learning from computer assisted learning: Evidence from a randomized experiment in rural schools in shaanxi province. Technical report, REAP working paper.

Muralidharan, K., Singh, A., and Ganimian, A. J. (2019). Disrupting education? experi- mental evidence on technology-aided instruction in india. American Economic Review, 109(4):1426–60.

Phillips, D. and Cohen, J. (2015). Learning gets personal. how idaho students and teachers are embracing personalized learning through khan academy. Technical report.

Weeraratne, B. and Chin, B. (2018). Can khan academy e-learning video tutorials im- prove mathematics achievement in sri lanka?. International Journal of Education and Development using Information and Communication Technology, 14(3):93–112.

WorldBank (2018). World development report 2018 : Learning to realize education’s promise.

Yang, Y., Zhang, L., Zeng, J., Pang, X., Lai, F., and Rozelle, S. (2013). Computers and the academic performance of elementary school-aged girls in china’s poor communities. Computers & Education, 60(1):335–346.

Young, A. (2018). Channeling Fisher: Randomization Tests and the Statistical Insignificance of Seemingly Significant Experimental Results*. The Quarterly Journal of Economics, 134(2):557–598.