Assessing the Authenticity of Artificial Intelligence Appraisals of Clinical Psychiatric Scenarios: A Preliminary Investigation
Main Article Content
Abstract
This short communication explores the utility of generative artificial intelligence (GenAI) systems in helping students prepare for clinical examinations. Four clinical scenarios from an objective structured clinical examination (OSCE) in psychiatry were presented to GPT-4o mini. This GenAI system was asked to respond to four questions related to the clinical psychiatric scenarios. Two questions were designed to be straightforward and contextually clear, whereas two incorporated irrelevant information. The GenAI responses demonstrated noticeable difficulty in accurately and authentically interpreting the scenarios, particularly when extraneous content was introduced. While the answers to the first two questions were reasonably accurate, the responses to the remaining two were unacceptable, highlighting the system’s inability to decipher ambiguous or unclear input. Currently, GenAI systems appear to perform well at solving clinical cases when the data are straightforward and rule-based. However, GPT-4o mini demonstrated difficulty in interpreting scenarios complicated by emotional nuances or distracting elements.
Article Details
This work is licensed under a Creative Commons Attribution 4.0 International License.
References
Lindgren S, Argullos JLP, Millan JR. Assessment of clinical competence of medical students: future perspectives for Spanish faculties. Med Clin Pract. 2024;7(2):100424. https://doi.org/10.1016/j.mcpsp.2023.100424
Tervalon M. Components of culture in health for medical students’ education. Acad Med. 2003;78(6):570–6. https://doi.org/10.1097/00001888-200306000-00005
Badyal DK, Sharma M. Internal assessment in new MBBS curriculum: methods and logistics. Int J Appl Basic Med Res. 2020;10(2):68–75. https://doi.org/10.4103/ijabmr.IJABMR_18_20
Wass V, Van der Vleuten C, Shatzer J, Jones R. Assessment of clinical competence. Lancet. 2001;357(9260):945–9. https://doi.org/10.1016/S0140-6736(00)04221-5
Mitchell ML, Henderson A, Groves M, Dalton M, Nulty D. The objective structured clinical examination (OSCE): optimising its value in the undergraduate nursing curriculum. Nurse Educ Today. 2009;29(4):398–404. https://doi.org/10.1016/j.nedt.2008.10.007
Olivares XG, de León DA. Objective structured clinical examination (OSCE) in psychology. In: Hosseini S, Beitzel B, Köhler M, Sanjakdar F, editors. Technological advancements and innovations in future of education. Singapore: Springer; 2024. p. 53.
Sheen J, McGillivray J, Gurtman C, Boyd L. Assessing the clinical competence of psychology students through objective structured clinical examinations (OSCEs): student and staff views. Aust Psychol. 2015;50(1):51–9. https://doi.org/10.1111/ap.12095
Vitands B. Artificial intelligence behind improved clinical skills assessment and teaching. Medical School News [cited 2025 Jun 24]. Available from: https://med.umn.edu/news/artificialintelligence-behind-improved-clinical-skills-assessment-and-teaching
Ciotti M, Ciccozzi M, Terrinoni A, Jiang WC, Wang CB, Bernardini S. The COVID-19 pandemic. Crit Rev Clin Lab Sci. 2020;57(6):365–88. https://doi.org/10.1080/10408363.2020.1783198
Mishra GV, Luharia AA, Naqvi W, Sood A. Artificial intelligence in OSCE: innovations and implications for medical education assessment–a systematic review. In: 2024 2nd DMIHER International Conference on Artificial Intelligence in Healthcare, Education and Industry (IDICAIEI). IEEE; 2024. p. 1–5. https://doi.org/10.1109/IDICAIEI61867.2024.10842789
Misra SM, Suresh S. Artificial intelligence and objective structured clinical examinations: using ChatGPT to revolutionise clinical skills assessment in medical education. J Med Educ Curric Dev. 2024;11:23821205241263475. https://doi.org/10.1177/23821205241263475
Soong TK, Ho CM. Artificial intelligence in medical OSCEs: reflections and future developments. Adv Med Educ Pract. 2021;12:167–73. https://doi.org/10.2147/AMEP.S287926
Janumpally R, Nanua S, Ngo A, Youens K. Generative artificial intelligence in graduate medical education. Front Med (Lausanne). 2025;11:1525604. https://doi.org/10.3389/fmed.2024.1525604
McCoy L, Ganesan N, Rajagopalan V, McKell D, Niño DF, Swaim MC. A training needs analysis for AI and generative AI in medical education: perspectives of faculty and students. J Med Educ Curric Dev. 2025;12:23821205251339226. https://doi.org/10.1177/23821205251339226
Mousavi Baigi SF, Sarbaz M, Ghaddaripouri K, Ghaddaripouri M, Mousavi AS, Kimiafar K. Attitudes, knowledge, and skills towards artificial intelligence among healthcare students: a systematic review. Health Sci Rep. 2023;6(3):e1138. https://doi.org/10.1002/hsr2.1138
Tangadulrat P, Sono S, Tangtrakulwanich B. Using ChatGPT for clinical practice and medical education: cross-sectional survey of medical students’ and physicians’ perceptions. JMIR Med Educ. 2023;9:e50658. https://doi.org/10.2196/50658
Sami A, Tanveer F, Sajwani K, Kiran N, Javed MA, Ozsahin DU, et al. Medical students’ attitudes toward AI in education: perception, effectiveness, and its credibility. BMC Med Educ. 2025;25(1):82. https://doi.org/10.1186/s12909-025-06574-x
Webster CS, Jowsey T. Artificial intelligence must operate ethically in health care and not be prone to racist or sexist biases. Anesth Analg. 2025;140(5):1099–104. https://doi.org/10.1213/ANE.0000000000007131
OpenAI. ChatGPT: get answers, find inspiration, be more productive. 2025. Available from: https://openai.com/chatgpt/overview/
Hodges B, Regehr G, Hanson M, McNaughton N. Validation of an objective structured clinical examination in psychiatry. Acad Med. 1998;73(8):910–2. https://doi.org/10.1097/00001888-199808000-00021
Koo TK, Li MY. A guideline of selecting and reporting intraclass correlation coefficients for reliability research. J Chiropr Med. 2016;15(2):155–63. https://doi.org/10.1016/j.jcm.2016.02.012
Hodges BD, Hollenberg E, McNaughton N, Hanson MD, Regehr G. The psychiatry OSCE: a 20-year retrospective. Acad Psychiatry. 2014;38(1):26–34. https://doi.org/10.1007/s40596-013-0008-8
Shafti SS. Association between emotional intelligence and objective structured examinations: a study on psychiatric residents. Educ Res Med Sci. 2017;6(1):23–8. https://doi.org/10.22110/erms.2017.51478
Nori H, King N, McKinney SM, Carignan D, Horvitz E. Capabilities of GPT-4o mini on medical challenge problems. arXiv. 2023;arXiv:2303.13375. https://doi.org/10.48550/arXiv.2303.13375
Arkoudas K. GPT-4o mini can’t reason. 2023;arXiv:2308.03762. https://doi.org/10.48550/arXiv.2308.03762
Mahbub T, Dghaym D, Shankarnarayanan A, Syed T, Shapsough S, Zualkernan I. Can GPT-4
aid in detecting ambiguities, inconsistencies, and incompleteness in requirements analysis?
A comprehensive case study. IEEE Access. 2024;12:12345–56. https://doi.org/10.1109/ACCESS.2024.3350000
He R, Sarwal V, Qiu X, Zhuang Y, Zhang L, Liu Y, et al. Generative AI models in time-varying biomedical data: scoping review. J Med Internet Res. 2025;27:e59792. https://doi.org/10.2196/59792
Prather J, Leinonen J, Kiesler N, Gorson Benario J, Lau S, MacNeil S, et al. Beyond the hype: a comprehensive review of current trends in generative AI research, teaching practices, and tools. In: 2024 Working Group Reports on Innovation and Technology in Computer Science Education. New York: Association for Computing Machinery; 2025. p. 300–38. https://doi.
org/10.1145/3689187.3709614
Ng BY, Li J, Tong X, Ye K, Yenne G, Chandrasekaran V, et al. Analyzing security and privacy challenges in generative AI usage guidelines for higher education. arXiv. 2025; arXiv:2506.20463. https://doi.org/10.48550/arXiv.2506.20463