Online Maths for Social Science Assessment Design in the Age of Generative AI

Soobedar de Villeneuve, Zeenat (2024): Online Maths for Social Science Assessment Design in the Age of Generative AI.

Abstract

In the fast-changing educational world, Generative AI (GenAI) has brought about big changes, especially in online formative and summative assessments. Universities are concerned about the unethical GenAI use, compromising academic integrity. This study proposes and evaluates strategies to design questions in maths for social science education that are challenging for ChatGPT-3.5 to solve. Drawing on Bloom’s Taxonomy, a trend analysis of academic performances and focus group discussions, it proposes a transformed approach to assessment design: the SHARP (Strategic, Holistic, Adaptive, Reflective, Process) assessment cycle. This framework is iterative and integrates real-time feedback to ensure inclusivity and transparency, stemming from a Reflect-Rewrite-Retest-Review redesign approach, focusing on higher-order cognitive questions. A quantitative analysis between 2020 and 2024 reveals a significant increase in higher-order level questions (e.g. from 29% to 84% in a test) and a significant but not drastic drop in academic performance. The effectiveness of ChatGPT-challenging designs is corroborated by focus group discussions, highlighting the need for a balance between student accessibility and academic rigour. This study contributes to the literature by providing unique empirical evidence on the validity of the strategies and offering actionable steps for educators, policymakers and institutions to maintain academic integrity in maths for social science education.

• Item Type: MPRA Paper
• Language: English
• Keywords: ChatGPT; academic integrity; formative and summative assessments; maths questions; assessment design
• Subjects: A - General Economics and Teaching > A2 - Economic Education and Teaching of Economics > A21 - Pre-college; C - Mathematical and Quantitative Methods > C8 - Data Collection and Data Estimation Methodology ; Computer Programs > C80 - General; C - Mathematical and Quantitative Methods > C8 - Data Collection and Data Estimation Methodology ; Computer Programs > C89 - Other

References

Al-Ubaydli O, List JA (2015) On the Generalizability of Experimental Results in Economics. Handb Exp Econ Methodol :420–462. https://doi.org/10.1093/acprof:oso/9780195328325.003.0022 Amzalag M, Shapira N, Dolev N (2021) Two Sides of the Coin: Lack of Academic Integrity in Exams During the Corona Pandemic, Students’ and Lecturers’ Perceptions. J Acad Ethics 20:243–263 Anderson LW, Krathwohl DR (2001) A Taxonomy for Learning, Teaching, and Assessing: A Revision of Bloom’s Taxonomy of Educational Objectives. Longman Bain, Jennifer (2010) Integrating student voice: assessment for empowerment. Pract Res High Educ 4(1):14–29 Bitzenbauer P (2023) ChatGPT in physics education: A pilot study on easy-to-implement activities. Contemp Educ Technol 15(3). https://doi.org/10.30935/cedtech/13176 Bloom BS, Engelhart MD, Furst EJ, Hill WH, Krathwohl DR, others (1956) Taxonomy of educational objectives: The classification of educational goals. Handbook 1: Cognitive domain. Longman New York Braun V, Clarke V (2006) Using thematic analysis in psychology. Qual Res Psychol 3:77–101. https://doi.org/10.1191/1478088706qp063oa Brown GTL (2022) The past, present and future of educational assessment: A transdisciplinary perspective. Front Educ 7(November):1–8. https://doi.org/10.3389/feduc.2022.1060633 Bryman AE (2008) Of methods and methodology. Qual Res Organ Manag An Int J 3:159–168 Clarke O, Chan WYD, Bukuru S, Logan J, Wong R (2023) Assessing knowledge of and attitudes towards plagiarism and ability to recognize plagiaristic writing among university students in Rwanda. High Educ 85(2):247–263. https://doi.org/10.1007/S10734-022-00830-Y Clements D, Battista M (1990) Constructivist learning and teaching. Arith Teach 38:34–35. https://doi.org/10.5951/AT.38.1.0034 Digital Education Council (2024). Global AI Student Survey 2024. https://26556596.fs1.hubspotusercontent-eu1.net/hubfs/26556596/Digital%20Education%20Council%20Global%20AI%20Student%20Survey%202024.pdf. Accessed 24 Jan 2025. Dunne S, Lee D (2022) Designing assessment for neurodiverse students. In: LMJU Student Experience Proceedings (ed) 2022: Students at the Heart Conference. Liverpool John Moores University Eke DO (2023) ChatGPT and the rise of generative AI: Threat to academic integrity? J Responsible Technol 13:100060. https://doi.org/10.1016/J.JRT.2023.100060 Else H (2023) BY CHATGPT FOOL SCIENTISTS. Nature 613:423 Farrelly T, Baker N (2023) Generative Artificial Intelligence: Implications and Considerations for Higher Education Practice. Educ. Sci. 13 Flavell JH (1979) Metacognition and Cognitive Monitoring: A New Area of Cognitive-Developmental Inquiry. Am Psychol 34:906–911 Gikandi JW, Morrow D, Davis NE (2011) Online formative assessment in higher education: A review of the literature. Comput Educ 57(4):2333–2351. https://doi.org/10.1016/j.compedu.2011.06.004 Henderson M, Awdry R, Chung J, Ashford C, Bryant M, Mundy M, Ryan K (2022) Online exams: exploring student experience and integrity behaviours as we return to campus. ASCILITE Publ :e22095. https://doi.org/10.14742/apubs.2022.95 Hersh W, Fultz Hollis K (2024) Results and implications for generative AI in a large introductory biomedical and health informatics course. npj Digit Med 7(1):1–7. https://doi.org/10.1038/s41746-024-01251-0 Holden OL, Norris ME, Kuhlmeier VA (2021) Academic Integrity in Online Assessment: A Research Review . Front. Educ. 6 Lancaster T, Cotarlan C (2021) Contract cheating by STEM students through a file sharing website: a Covid-19 pandemic perspective. Int J Educ Integr 17(1):1–17. https://doi.org/10.1007/s40979-021-00070-0 Liu D, Bridgeman A (2023) How can I update assessments to deal with ChatGPT and other generative AI? In: 23 January 2023. https://educational-innovation.sydney.edu.au/teaching@sydney/how-can-i-update-assessments-to-deal-with-chatgpt-and-other-generative-ai/ Lumivero (2023) NVivo Lye CY, Lim L (2024) Generative Artificial Intelligence in Tertiary Education: Assessment Redesign Principles and Considerations. Educ Sci 14(6). https://doi.org/10.3390/educsci14060569 Nikolic S, Daniel S, Haque R, Belkina M, Hassan GM, Grundy S, Lyden S, Neal P, Sandison C (2023) ChatGPT versus engineering education assessment: a multidisciplinary and multi-institutional benchmarking and analysis of this generative artificial intelligence tool to investigate assessment integrity. Eur J Eng Educ 48(4):559–614. https://doi.org/10.1080/03043797.2023.2213169 Paul R, Elder L (2013) Critical Thinking: Tools for Taking Charge of Your Learning and Your Life: Pearson New International Edition. Pearson Education Phillips AJ, Briggs JC, Jensen JL (2019) Beyond Bloom’s: Students’ Perception of Bloom’s Taxonomy and its Convolution with Cognitive Load. J Psychol Res 1(01):24–32. https://doi.org/10.30564/jpr.v1i01.421 Piaget J (1976) To Understand is to Invent: The Future of Education. Penguin Books Rasul T, Nair S, Kalendra D, Robin M, Santini F de O, Ladeira WJ, Sun M, Day I, Rather RA, Heathcote L (2023) The role of ChatGPT in higher education: Benefits, challenges, and future research directions. J Appl Learn Teach 6(1):41–56. https://doi.org/10.37074/jalt.2023.6.1.29 Reardon SF, Kalogrides D, Fahle EM, Podolsky A, Zárate RC (2018) The Relationship Between Test Item Format and Gender Achievement Gaps on Math and ELA Tests in Fourth and Eighth Grades. Educ Res 47(5):284–294. https://doi.org/10.3102/0013189X18762105 Reedy A, Pfitzner D, Rook L, Ellis L (2021) Responding to the COVID-19 emergency: student and academic staff perceptions of academic integrity in the transition to online exams at three Australian universities. Int J Educ Integr 17(1):9. https://doi.org/10.1007/s40979-021-00075-9 Roe J, Perkins M, Ruelle D (2024) Understanding Student and Academic Staff Perceptions of AI Use in Assessment and Feedback. arXiv Prepr arXiv240615808 :1–13 Salinas-Navarro DE, Vilalta-Perdomo E, Michel-Villarreal R, Montesinos L (2024) Using Generative Artificial Intelligence Tools to Explain and Enhance Experiential Learning for Authentic Assessment. Educ Sci 14(1). https://doi.org/10.3390/educsci14010083 Sallam M, Salim NA, Barakat M, Ala’a B (2023) ChatGPT applications in medical, dental, pharmacy, and public health education: A descriptive study highlighting the advantages and limitations. Narra J 3(1) Seo K, Tang J, Roll I, Fels SS, Yoon D (2021) The impact of artificial intelligence on learner–instructor interaction in online learning. Int J Educ Technol High Educ 18 Silverman D (2016) Qualitative Research. SAGE Publications St-Onge C, Ouellet K, Lakhal S, Dubé T, Marceau M (2022) COVID-19 as the tipping point for integrating e-assessment in higher education practices. Br J Educ Technol 53(2):349–366. https://doi.org/10.1111/bjet.13169 Stamov Roßnagel C, Lo Baido K, Fitzallen N (2021) Revisiting the relationship between constructive alignment and learning approaches: A perceived alignment perspective. PLoS One 16(8):1–13. https://doi.org/10.1371/journal.pone.0253949 Su J, Yang W (2023) Unlocking the Power of ChatGPT: A Framework for Applying Generative AI in Education. ECNU Rev Educ 6(3):355–366. https://doi.org/10.1177/20965311231168423 Sweller J, Ayres P, Kalyuga S (2011) Cognitive Load Theory. Springer New York Tan TF, Thirunavukarasu AJ, Campbell JP, Keane PA, Pasquale LR, Abramoff MD, Kalpathy-Cramer J, Lum F, Kim JE, Baxter SL, Ting DSW (2023) Generative Artificial Intelligence Through ChatGPT and Other Large Language Models in Ophthalmology: Clinical Applications and Challenges. Ophthalmol Sci 3(4):100394. https://doi.org/https://doi.org/10.1016/j.xops.2023.100394 UCL Assessment Working Group (2020) Designing Effective Online Assessment Van Dis EAM, Bollen J, Zuidema W, Van Rooij R, Bockting CL (2023) ChatGPT: five priorities for research. Nature 614(7947):224–226 Vellanki S, Mond S, Khan Z (2023) Promoting Academic Integrity in Remote/Online Assessment – EFL Teachers’ Perspectives. Teach English as a Second or Foreign Lang 26:1–20. https://doi.org/10.55593/ej.26104a7 Vygotsky LS (1980) Mind in Society: The Development of Higher Psychological Processes. Harvard University Press Weimer M (2018) Multiple-Choice Tests: Revisiting the Pros and Cons. Fac. Focus