XAI as Compliance and Design Strategy for Automated Grading Systems
Main Article Content
Abstract
Ensuring explainability in Artificial Intelligence (AI) systems requires the careful integration of legal compliance and technical design. This paper proposes a compliance and design strategy for operationalising explainability obligations under the European Union’s Artificial Intelligence Act (AI Act), and the General Data Protection Regulation (GDPR), using automated grading systems (AGSs) in education as a case study. We argue that legal requirements and technical approaches should be coordinated from the earliest stages of system development to ensure trustworthy AI deployment. Our analysis combines legal analysis on the provisions of the AI Act’s relating to “high-risk” educational AI and the GDPR’s provisions on solely automated decision-making with a technical assessment of explainability methods including SHAP, LIME, feature sensitivity analysis, and interpretable models, and evaluate their appropriateness for educational contexts. An illustrative analysis of automated grading systems (AGSs) as a representative educational AI use case highlights how risks can emerge in automated grading and how targeted explainability approaches may mitigate these risks. Legal analysis reveals a duty to provide meaningful, fair and transparent explanations. A compliance and design strategy is proposed that integrates human-in-the-loop (HITL) mechanisms, collaborative design processes with educators, and stakeholder-centric explanation techniques. This interdisciplinary strategy demonstrates how technical methods and legal standards can work together to reduce risks, improve accountability, and strengthen trust in AGSs.
Article Details
This work is licensed under a Creative Commons Attribution 4.0 International License.
References
Arrieta, A. B., Díaz-Rodríguez, N., Del Ser, J., Bennetot, A., Tabik, S., Barbado, A., Herrera, F. (2020). Explainable artificial intelligence (XAI): Concepts, taxonomies, opportunities and challenges toward responsible AI. Information Fusion, 58, 82–115. https://doi.org/10.1016/j.inffus.2019.12.012
Bellas, F., Ooge, J., Roddeck, L., Rashheed, H., Prifti Skenduli, M., Masdoum, F., Zainuddin, N., Niewint, J., Costello, E., Kralj, L., Dcosta, D., Katsamori, D., Neethling, D., Maat, S., Saurabh, R., Alasgarova, R., Radaelli, E., Stamatescuù, A., Blazic, A., . . . Obae, C. (2025). Explainable AI in education: Fostering human oversight and shared responsibility (Technical Report). European Digital Education Hub; Publications Office of the European Union. https://data.europa.eu/doi/10.2797/6780469
Burger, C., Chen, L., & Le, T. (2023). Are your explanations reliable? Investigating the stability of LIME in explaining text classifiers by marrying XAI and adversarial attack. arXiv. https://doi.org/10.48550/arXiv.2305.12351
Caruana, R., Lou, Y., Gehrke, J., Koch, P., Sturm, M., & Elhadad, N. (2015). Intelligible models for healthcare: Predicting pneumonia risk and hospital 30-day readmission. In Proceedings of the 21st ACM SIGKDD International Conference on Knowledge Discovery and Data Mining (pp. 1721–1730). ACM. https://doi.org/10.1145/2783258.2788613
Dimari, A., Tyagi, N., Davanageri, M., Kukreti, R., Yadav, R., & Dimari, H. (2024). AI-based automated grading systems for open book examination system: Implications for assessment in higher education. In 2024 International Conference on Knowledge Engineering and Communication Systems (ICKECS) (pp. 1–7). IEEE. https://doi.org/10.1109/ICKECS61492.2024.10616490
Edwards, L., & Veale, M. (2017). Slave to the algorithm? Why a “right to an explanation” is probably not the remedy you are looking for. Duke Law & Technology Review, 16(1), 18–84. https://doi.org/10.2139/ssrn.2972855
European Commission. (2026). Guidelines on the ethical use of artificial intelligence and data in teaching and learning for educators. Publications Office of the European Union. https://op.europa.eu/s/Aeo7
Figueras, C., Rossitto, C., & Cerratto Pargman, T. (2024). Doing responsibilities with automated grading systems: An empirical multi-stakeholder exploration. In Proceedings of the 13th Nordic Conference on Human-Computer Interaction (pp. 1–13). ACM. https://doi.org/10.1145/3679318.3685378
Fiok, K., Farahani, F. V., Karwowski, W., & Ahram, T. (2022). Explainable artificial intelligence for education and training. The Journal of Defense Modeling and Simulation, 19(2), 133–144. https://doi.org/10.1177/15485129211028651
Fülöp, T., & Poindl, P. (2025). Article 27: Fundamental rights impact assessment for high-risk AI systems. In C. N. Pehlivan, N. Forgo, & P. Valcke (Eds.), The EU Artificial Intelligence (AI) Act (pp. 571–590). Wolters Kluwer.
Hacker, P., Krestel, R., Grundmann, S., & Naumann, F. (2020). Explainable AI under contract and tort law: Legal incentives and technical challenges. Artificial Intelligence and Law, 28(4), 415–439. https://doi.org/10.1007/s10506-020-09260-6
Holmes, W., Bialik, M., & Fadel, C. (2022). Artificial intelligence in education: Promises and implications for teaching and learning. Center for Curriculum Redesign.
Information Commissioner’s Office, & The Alan Turing Institute. (2020). Explaining decisions made with AI. https://ico.org.uk/for-organisations/uk-gdpr-guidance-and-resources/artificial-intelligence/explaining-decisions-made-with-artificial-intelligence/
Kilian, R., Jäck, L., & Ebel, D. (2025). European AI standards: Technical standardization and implementation challenges under the EU AI Act. SSRN. https://doi.org/10.2139/ssrn.5155591
Laux, J., & Ruschemeier, H. (2025). Automation bias in the AI Act: On the legal implications of attempting to de-bias human oversight of AI. SSRN. https://ssrn.com/abstract=5117560
Mantelero, A., Guzmán, C., García, E., Ortiz, R., & Mor, M. A. (2025). FRIA model: Guide and use cases. FRIA methodology for AI design and development. Catalan Data Protection Authority.
McConville, M., & Chui, W. H. (2017). Research methods for law (2nd ed.). Edinburgh University Press.
Miller, T. (2019) Explanation in artificial intelligence: Insights from the social sciences. Artificial Intelligence, 267, 1–38. https://doi.org/10.1016/j.artint.2018.07.007
Miller, T. (2023). Explainable AI is dead, long live explainable AI! Hypothesis-driven decision support using evaluative AI. In FAccT ‘23: Proceedings of the 2023 ACM Conference on Fairness, Accountability, and Transparency (pp. 333–342). ACM. https://doi.org/10.1145/3593013.3594001
Mittelstadt, B., Russell, C., & Wachter, S. (2019). Explaining explanations in AI. In Proceedings of the Conference on Fairness, Accountability, and Transparency (FAT 2019) (pp. 279–288). ACM. https://doi.org/10.1145/3287560.3287574
National Institute of Standards and Technology. (2023). Artificial intelligence risk management framework (AI RMF 1.0). https://doi.org/10.6028/NIST.AI.100-1
OECD. (2019). Recommendation of the Council on Artificial Intelligence. OECD/LEGAL/0449. OECD Legal Instruments. https://oecd.ai/en/assets/files/OECD-LEGAL-0449-en.pdf
OECD. (2024). Recommendation of the Council on Artificial Intelligence (OECD/LEGAL/0449, amended 2024). OECD Legal Instruments. https://legalinstruments.oecd.org/en/instruments/OECD-LEGAL-0449
Onatere-Ubrurhe, J. O., & Ubrurhe, O. (2025). Leveraging artificial intelligence to redesign TVET assessment systems for enhancing creativity and innovation in technical education. International Journal of Vocational and Technical Education Research, 11(2), 1–20.
Pasquale, F. (2015). The black box society: The secret algorithms that control money and information. Harvard University Press.
Ravi, P., Masla, J., Kakoti, G., Lin, G. C., Anderson, E., Taylor, M., Ostrowski, A. K., Breazeal, C., Klopfer, E., & Abelson, H. (2025). Co-designing large language model tools for project-based learning with K-12 educators. In Proceedings of the 2025 CHI Conference on Human Factors in Computing Systems (CHI ‘25) (pp. 138:1–138:25). ACM. https://doi.org/10.1145/3706598.3713971
Roddeck, L., Abu-Rasheed, H., Ooge, J., & Bellas, F. (2025). Educational stakeholders need a legal guide on explainable AI [Paper presentation]. XAI-Ed: Pedagogy-Founded Explainable AI for Transparent, User-Centred AI in Education — Workshop at the 26th International Conference on Artificial Intelligence in Education (AIED 2025), Palermo, Italy.
Rozenblit, L., & Keil, F (2002) The misunderstood limits of folk science: An illusion of explanatory depth. Cognitive Science, 26, 521–562. https://doi.org/10.1207/s15516709cog2605_1
Russell, S., & Norvig, P. (2022). Artificial intelligence: A modern approach (4th ed.). Pearson.
Saeed, W., & Omlin, C. (2023). Explainable AI (XAI): A systematic meta-survey of current challenges and future opportunities. Knowledge-Based Systems, 263, 110273. https://doi.org/10.1016/j.knosys.2023.110273
Selbst, A. D. (2021). An institutional view of algorithmic impact assessments. Harvard Journal of Law & Technology, 35(1), 117–194. https://jolt.law.harvard.edu/assets/articlePDFs/v35/Selbst-An-Institutional-View-of-Algorithmic-Impact-Assessments.pdf
Selwyn, N. (2019). Should robots replace teachers? AI and the future of education. Polity Press.
Tseng, E.-Q., Huang, P.-C., Hsu, C., Wu, P.-Y., Ku, C.-T., & Kang, Y. (2024). CodEv: An automated grading framework leveraging large language models for consistent and constructive feedback. In 2024 IEEE International Conference on Big Data (BigData) (pp. 5442–5449). IEEE. https://doi.org/10.1109/BigData62323.2024.10825949
UNESCO. (2021). Recommendation on the ethics of artificial intelligence. https://unesdoc.unesco.org/ark:/48223/pf0000381137
Wachter, S. (2024). Limitations and loopholes in the EU AI Act and AI liability directives: What this means for the European Union, the United States, and beyond. Yale Journal of Law & Technology, 26(3), 671–718.
Wachter, S., Mittelstadt, B., & Russell, C. (2017). Counterfactual explanations without opening the black box: Automated decisions and the GDPR. Harvard Journal of Law & Technology, 31(2), 841–887.
Williamson, B., & Eynon, R. (2020). Historical threads, missing links, and future directions in AI in education. Learning, Media and Technology, 45(3), 223–235. https://doi.org/10.1080/17439884.2020.1798995
Wood, S. N. (2017). Generalized additive models: An introduction with R (2nd ed.). Chapman & Hall/CRC. https://doi.org/10.1201/9781315370279