Evaluating the Role of ChatGPT as a Legal Educational Companion
Main Article Content
Abstract
This study evaluates the efficacy of ChatGPT as a study companion in legal education, addressing ongoing debates regarding its pedagogical benefits and potential risks to critical thinking. Guided by the IDEE framework, a mixed-methods approach (quantitative and thematic qualitative analysis) was adopted using a pre-test/post-test design involving 64 first-year law students. Student performance was measured through a 10-question multiple-choice quiz, administered without ChatGPT in the pre-test and with ChatGPT in the post-test, with supplementary qualitative feedback collected via an open-ended questionnaire. The findings reveal a statistically significant improvement in overall student performance, particularly in facilitating progression from failing to passing and good grades. However, no significant evidence was found to suggest that ChatGPT supports the attainment of the highest academic grades (A or A–). This suggests that ChatGPT is effective in supporting lower-order cognitive processes but has limited impact on higher-order reasoning required for advanced legal analysis. These findings should be interpreted with caution due to methodological limitations, including the absence of a control group and potential test familiarity effects. Overall, ChatGPT functions best as a supplementary learning tool that supports foundational understanding, while human instruction remains essential for developing critical thinking and nuanced legal judgement.
Article Details
This work is licensed under a Creative Commons Attribution 4.0 International License.
References
Ajevski, M., Barker, K., Gilbert, A., Hardie, L., & Ryan, F. (2023). ChatGPT and the future of legal education and practice. The Law Teacher, 57, 352–364. https://doi.org/10.1080/03069400.2023.2207426
Alarie, B. (2023, January 2). The rise of the robotic tax analyst. Tax Notes Federal, 178, 57.
Anderson, L. W., & Krathwohl, D. R. (Eds.). (2001). A taxonomy for learning, teaching, and assessing: A revision of Bloom’s taxonomy of educational objectives. Longman.
Ball, C. (2023, January 27). ChatGPT proves a mediocre law student. Ball in Your Court. https://craigball.net/2023/01/27/chatgpt-proves-a-mediocre-law-student/
Biggs, J., & Tang, C. (2011). Teaching for quality learning at university (4th ed.). Open University Press.
Braun, V., & Clarke, V. (2006). Using thematic analysis in psychology. Qualitative Research in Psychology, 3(2), 77–101. https://doi.org/10.1191/1478088706qp063oa
Campbell, D. T., & Stanley, J. C. (1963). Experimental and quasi-experimental designs for research. Houghton Mifflin.
Chan, C. K. Y. (2023). A comprehensive AI policy education framework for university teaching and learning. International Journal of Educational Technology in Higher Education, 20(1), 38. https://doi.org/10.1186/s41239-023-00408-3
Choi, J. H., Hickman, K. E., Monahan, A. B., & Schwarcz, D. (2022). ChatGPT goes to law school. Journal of Legal Education, 71(3), 387–400. https://doi.org/10.2139/ssrn.4335905
Cotton, D. R. E., Cotton, P. A., & Shipway, J. R. (2024). Chatting and cheating: Ensuring academic integrity in the era of ChatGPT. Innovations in Education and Teaching International, 61(2), 228–239. https://doi.org/10.1080/14703297.2023.2190148
Creswell, J. W., & Plano Clark, V. L. (2018). Designing and conducting mixed methods research (3rd ed.). SAGE Publications.
Dimitrov, D. M., & Rumrill, P. D. (2003). Pretest-posttest designs and measurement of change. Work, 20(2), 159–165. https://doi.org/10.3233/WOR-2003-00285
Dwivedi, Y. K., Kshetri, N., Hughes, L., Slade, E. L., Jeyaraj, A., Kar, A. K., Baabdullah, A. M., Koohang, A., Raghavan, V., Ahuja, M., Albanna, H., Albashrawi, M., Al-Busaidi, K. A., Balakrishnan, J., Barlette, Y., Basu, S., Bose, I., Brooks, L., Buhalis, D., … Wright, R. (2023). So what if ChatGPT wrote it? Multidisciplinary perspectives on opportunities, challenges and implications of generative conversational AI for research, practice and policy. International Journal of Information Management, 71, 102642. https://doi.org/10.1016/j.ijinfomgt.2023.102642
Etikan, I., Musa, S. A., & Alkassim, R. S. (2016). Comparison of convenience sampling and purposive sampling. American Journal of Theoretical and Applied Statistics, 5(1), 1–4. https://doi.org/10.11648/j.ajtas.20160501.11
Frank, A. (2024). Artificial intelligence as a scaffolding tool for student learning: Implications for higher education. Frontiers in Education, 9, 1793940. https://doi.org/10.3389/feduc.2024.1793940
Hargreaves, S. (2023). “Words are flowing out like endless rain into a paper cup”: ChatGPT & Law School Assessments. Legal Education Review, 33(1), 69–105. https://doi.org/10.53300/001c.83297
Hu, R., & Lin, X. (2025). Incorporating ChatGPT into online discussions in a literacy course: Impact and students’ perceptions. International Journal of Artificial Intelligence (AI) in Teaching and Learning, 1(1), 1–18. https://doi.org/10.4018/IJAITL.366589
Hwang, G. -J., Xie, H., Wah, B. W., & Gašević, D. (2020). Vision, challenges, roles and research issues of Artificial Intelligence in Education. Computers and Education: Artificial Intelligence, 1, 100001. https://doi.org/10.1016/j.caeai.2020.100001
Irfan, M., Murray, L., & Ali, S. (2023). Integration of artificial intelligence in academia: A case study of critical teaching and learning in higher education. Global Social Sciences Review, 8(1), 352–364. https://doi.org/10.31703/gssr.2023(VIII-I).32
Javaid, M., Haleem, A., Singh, R. P., Khan, S., & Khan, I. H. (2023). Unlocking the opportunities through ChatGPT Tool towards ameliorating the education system. BenchCouncil Transactions on Benchmarks, Standards & Evaluations, 3(2), 1–12. https://doi.org/10.1016/j.tbench.2023.100115
Kasneci, E., Sessler, K., Küchemann, S., Bannert, M., Dementieva, D., Fischer, F., Gasser, U., Groh, G., Günnemann, S., Hüllermeier, E., Krusche, S., Kutyniok, G., Michaeli, T., Nerdel, C., Pfeffer, J., Poquet, O., Sailer, M., Schmidt, A., Seidel, T., . . . Kasneci, G. (2023). ChatGPT for good? On opportunities and challenges of large language models for education. Learning and Individual Differences, 103, 102274. https://doi.org/10.1016/j.lindif.2023.102274
Keith, M., Keiller, E., Windows-Yule, C., Kings, I., & Robbins, P. (2025). Harnessing generative AI in chemical engineering education: Implementation and evaluation of the large language model ChatGPT v3.5. Education for Chemical Engineers, 51, 20–33. https://doi.org/10.1016/j.ece.2025.01.002
Kiryakova, G., & Angelova, N. (2023). ChatGPT: A challenging tool for the university professors in their teaching practice. Education Sciences, 13(10), 1056. https://doi.org/10.3390/educsci13101056
Klingensmith, M. W. (2023). Let’s talk, ChatGPT: What will the judiciary’s future look like? Florida Bar Journal, 97(3), 26–33.
Lee, H. (2024). The rise of ChatGPT: Exploring its potential in medical education. Anatomical Sciences Education, 17, 926–931. https://doi.org/10.1002/ase.2270
Lo, C. K. (2023). What is the impact of ChatGPT on education? A rapid review of the literature. Education Sciences, 13(4), 410. https://doi.org/10.3390/educsci13040410
Lock, S. (2022, December 5). What is AI chatbot phenomenon ChatGPT and could it replace humans? The Guardian. https://www.theguardian.com/technology/2022/dec/05/what-is-ai-chatbot-phenomenon-chatgpt-and-could-it-replace-humans
Miller, M. (n.d.). Artificial intelligence (AI). Ditch That Textbook. https://ditchthattextbook.com/ai/
OECD. (2021). AI and the future of skills, Volume 1: Capabilities and assessments. OECD Publishing. https://doi.org/10.1787/5ee71f34-en
Okonkwo, C. W., & Ade-Ibijola, A. (2021). Chatbots applications in education: A systematic review. Computers and Education: Artificial Intelligence, 2, 100033. https://doi.org/10.1016/j.caeai.2021.100033
Oltz, T. P. (2023). ChatGPT, professor of law. University of Illinois Journal of Law, Technology & Policy, 2023(1), 207–221.
OpenAI. (n.d.). GPT-4 is OpenAI’s most advanced system, producing safer and more useful responses. https://openai.com/product/gpt-4
Perlman, A. M. (2023, March/April). The implications of ChatGPT for legal services and society. The Practice. https://clp.law.harvard.edu/knowledge-hub/magazine/issues/generative-ai-in-the-legal-profession/the-implications-of-chatgpt-forlegal-services-and-society/
Riermaier, P. (2023). ChatGPT and other AI technologies in the study and practice of law. Penn Carey Law, University of Pennsylvania. https://www.law.upenn.edu/live/news/15538-chatgpt-and-the-law
Roediger, H. L., III, & Butler, A. C. (2011). The critical role of retrieval practice in long-term retention. Trends in Cognitive Sciences, 15(1), 20–27. https://doi.org/10.1016/j.tics.2010.09.003
Shadish, W. R., Cook, T. D., & Campbell, D. T. (2002). Experimental and quasi-experimental designs for generalized causal inference. Houghton Mifflin.
Skulmowski, A., & Xu, K. M. (2022). Understanding cognitive load in digital and online learning: A new perspective on extraneous cognitive load. Educational Psychology Review, 34(1), 171–196. https://doi.org/10.1007/s10648-021-09624-7
Stokel-Walker, C. (2023). ChatGPT listed as an author on research papers: Many scientists disapprove. Nature, 613(7945), 620–621. https://doi.org/10.1038/d41586-023-00107-z
Stott, F. A., & Stott, D. M. (2023). A perspective on the use of ChatGPT in tax education. In T. G. Calderon (Ed.), Advances in accounting education: Teaching and curriculum innovations (Vol. 27, pp. 145–153). Emerald Publishing Limited. https://doi.org/10.1108/S1085-462220230000027007
Su, J., & Yang, W. (2023). Unlocking the power of ChatGPT: A framework for applying generative AI in education. ECNU Review of Education, 20965311231168423.
Sullivan, W. M., Colby, A., Wegner, J. W., Bond, L., & Shulman, L. S. (2007). Educating lawyers: Preparation for the profession of law. Jossey-Bass.
Sweller, J. (1988). Cognitive load during problem solving: Effects on learning. Cognitive Science, 12(2), 257–285. https://doi.org/10.1207/s15516709cog1202_4
Taylor, S. T. (2023). Taylor’s perspective: Pros & cons of ChatGPT for law firms according to ... ChatGPT. Of Counsel, 42(4), 3–4.
Terwiesch, C. (2023). Would ChatGPT get a Wharton MBA? A prediction based on its performance in the operations management course. https://mackinstitute.wharton.upenn.edu/wp-content/uploads/2023/01/Christian-Terwiesch-Chat-GTP.pdf
Tran, T., Nguyen, H., Pham, L., & Le, D. (2025). Artificial intelligence as a “more knowledgeable other”: Exploring generative AI in learning contexts. npj Digital Medicine. Advance online publication. https://doi.org/10.1038/s41746-025-01823-8
Twabu, T. (2025). Artificial intelligence and cognitive load: Implications for learning and instruction. Discover Education, 4, Article 592. https://doi.org/10.1007/s44217-025-00592-6
van Gog, T., & Sweller, J. (2015). Not new, but nearly forgotten: The testing effect decreases with increased complexity. Educational Psychology Review, 27(2), 247–264. https://doi.org/10.1007/s10648-015-9310-x
Ventayen, R. J. M. (2023). ChatGPT by OpenAI: Students’ viewpoint on cheating using artificial intelligence-based application. Social Science Research Network. https://doi.org/10.2139/ssrn.4361548
Ward, S. F. (2023). STUDY AIDS: Some law schools already are using ChatGPT to teach legal research and writing. ABA Journal, 109(3), 38–39.
Weiler, J. H. H. (2023). Editorial: ChatGPT and law exams; On my way in IV: “Aren’t you exclusive?!” On the pros and cons of writing letters of reference for only one candidate in an academic hiring process; In this issue; In this issue - reviews. European Journal of International Law, 34(2), 281–290. https://doi.org/10.1093/ejil/chad029
Yang, C., Luo, L., Vadillo, M. A., Yu, R., & Shanks, D. R. (2021). Testing (quizzing) boosts classroom learning: A systematic and meta-analytic review. Psychological Bulletin, 147(4), 399–435. https://doi.org/10.1037/bul0000309
Yin, J., Goh, T. -T., Yang, B., & Xiaobin, Y. (2021). Conversation technology with microlearning: The impact of chatbot-based learning on students’ learning motivation and performance. Journal of Educational Computing Research, 59(1), 154–177. https://doi.org/10.1177/0735633120952067
Zhai, X. (2021). Practices and theories: How can machine learning assist in innovative assessment practices in science education? Journal of Science Education and Technology, 30(2), 139–149. https://doi.org/10.1007/s10956-021-09901-8
Zhai, X. (2022). ChatGPT user experience: Implications for education. Social Science Research Network. https://doi.org/10.2139/ssrn.4312418