Development, Testing, and International Uptake of a ‘Just-in-Time’ Clinical Skills Training Application for Medical Student Education
Main Article Content
Abstract
Medical student education may incorporate adjunctive teaching methods to enhance procedural skill competencies. A ‘Just-in-Time Training’ (JITT) smartphone application (app) was developed in New Zealand to enhance the undergraduate clinical skills teaching programme. The Clinical Skills app was developed at the University of Otago Christchurch Simulation Centre. Content was based on the taught undergraduate programme and includes checklists and photo/video guides for 19 clinical procedures. Usability has been assessed annually using the System Usability Scale (SUS) since 2022, and mean scores > 73/100 represent good usability. The app has been open source since 2017 for iPhone Operating System (iOS) and 2019 for Android users. Cumulative downloads and daily user numbers are recorded. The SUS was completed by 251 medical students over three years; the mean score increased from 74/100 in 2022 to 87/100 in 2024 (p < 0.001). The cumulative international downloads between 2017–2024 were 706,607. In 2023, there were 7,437 international downloads, increasing by 104% to 15,193 users in July 2024, with users located in 155 countries, mainly New Zealand (22%), and China (18%). The increasing downloads of the JITT Clinical Skills app suggest that it is a popular learning tool to supplement clinical skills teaching.
Article Details
This work is licensed under a Creative Commons Attribution 4.0 International License.
References
Sawyer T, White M, Zaveri P, Chang T, Ades A, French H, et al. Learn, see, practice, prove, do, maintain: an evidence-based pedagogical framework for procedural skill training in medicine. Acad Med. 2015;90(8):1025–33. https://doi.org/10.1097/ACM.0000000000000734
Khan WU, Twomey J, Ryan E, Martin T, Kamal M, Lok Boris Cheng P, et al. Barriers and enablers to achieving clinical procedure competency-based outcomes in a national paediatric training/residency program—a multi-centered qualitative study. BMC Med Educ. 2023;23(1):1–10. https://doi.org/10.1186/s12909-023-04928-4
Kodikara K, Seneviratne T, Godamunne P, Premaratna R. Challenges in learning procedural skills: student perspectives and lessons learned for curricular design. Teach Learn Med. 2023:1–19. https://doi.org/10.1080/10401334.2023.2226633
Topps D, Helmer J, Ellaway R. YouTube as a platform for publishing clinical skills training videos. Acad Med. 2013;88(2):192–7. https://doi.org/10.1097/ACM.0b013e31827c5352
Zante B. Impact of number of critical care procedural skill repetitions on supervision level and teaching style. PLoS One. 2023;18(1):1–13. https://doi.org/10.1371/journal.pone.0280207
Toy S, McKay Robert SF, Walker JL, Johnson S, Arnett JL. Using learner-centered, simulation-based training to improve medical students’ procedural skills. J Med Educ Curric Dev. 2017;4:1–6. https://doi.org/10.1177/2382120516684829
Sullivan M, Nyquist J, Etcheverry J, Nally M, Schaff P, Abbott A, et al. The development of a comprehensive school-wide simulation-based procedural skills curriculum for medical students. J Surg Educ. 2010;67(5):309–15. https://doi.org/10.1016/j.jsurg.2010.07.010
Seidman PA, Maloney LM, Olvet DM, Chandran L. Preclinical simulation training of medical students results in better procedural skills performance in end of the year three objective structured clinical evaluation assessments. Med Sci Educ. 2017;27(1):89–96. https://doi.org/10.1007/s40670-017-0371-y
Petrica A, Lungeanu D, Ciuta A, Marza AM, Botea M-O, Mederle OA. Using 360-degree video for teaching emergency medicine during and beyond the COVID-19 pandemic. Ann Med. 2021;53(1):1520–30. https://doi.org/10.1080/07853890.2021.1970219
Alvarez Aa, Manthey D, Promes SB, Haas M, Santen SA, Wagner J, et al. Applying the master adaptive learner framework to just-in-time training of procedures. AEM Educ Train. 2024;8(S1):17–23. https://doi.org/10.1002/aet2.10953
Barnes BE. Creating the practice-learning environment: using information technology to support a new model of continuing medical education. Acad Med. 1998;73(3):278–81. https://doi.org/10.1097/00001888-199803000-00016
Chueh H, Barnett GO. “Just-in-time” clinical information. Acad Med. 1997;72(6):512–7. https://doi.org/10.1097/00001888-199706000-00016
Cheng Y-T, Liu DR, Wang VJ. Teaching splinting techniques using a just-in-time training instructional video. Pediatr Emerg Care. 2017;33(3):166–70. https://doi.org/10.1097/PEC.0000000000000390
Branzetti JB, Adedipe AA, Gittinger MJ, Rosenman ED, Brolliar S, Chipman AK, et al. Randomised controlled trial to assess the effect of a just-in-time training on procedural performance: a proofof-concept study to address procedural skill decay. BMJ Qual. Saf. 2017;26(11):1–11. https://doi.org/10.1136/bmjqs-2017-006656
Fennessy T, Parekh K, Walsh R. Just-in-time procedure guides in emergency medicine. West. J. Emerg. Med. 2022;23(3):1–18. https://doi.org/10.5811/westjem.2022.2.53655
McKay J, Wasserman M, Monuteaux MC, Hirsch AW, Nagler J. Just-in-time procedural training for pediatric emergency medicine trainees: a randomised educational interventional trial. AEM Educ Train. 2023;7(3):1–12. https://doi.org/10.1002/aet2.10886
University of Otago, Christchurch Simulation Centre (UOCSC) [Internet]. Christchurch, New Zealand: University of Otago; 2024 [cited 2025 June 1]. Available from: https://www.otago.ac.nz/christchurch/services/simulationcentre
Simulation Centre. Clinical skills (Version 1.8.8). Google Play Store; Christchurch: University of Otago; 2018.
Simulation Centre. Clinical skills (Version 2.1.8). Apple App Store. Christchurch: University of Otago; 2018.
Dziuban CD, Hartman JL, Moskal PD. Blended learning. Center for applied research. Res Bull. 2004;2004(7):1–12.
Ozdamli F. Pedagogical framework of m-learning. Procedia - Soc Behav Sci. 2012;31:927–31. https://doi.org/10.1016/j.sbspro.2011.12.171
Prados-Carmona A, Fuentes-Jimenez F, Rafael Roman de los R, García-Rios A, Rioja-Bravo J, Herruzo-Gomez E, et al. A pilot study on the feasibility of developing and implementing a mobile app for the acquisition of clinical knowledge and competencies by medical students transitioning from preclinical to clinical years. Int J Env Res Pub He. 2022;19(5):1–17. https://doi.org/10.3390/ijerph19052777
Rowe M, Frantz J, Bozalek V. The role of blended learning in the clinical education of healthcare students: a systematic review. Med Teach. 2012;34(4):216–21. https://doi.org/10.3109/0142159X.2012.642831
Li A, Zain NM, Abu Wardeh Y, Yusuf A, Deng H. Effectiveness of online and offline blended teaching methods using a mobile application on surgical nursing apprenticeship: a clustered randomised controlled trial. Nurse Educ Pract. 2025;84:1–8. https://doi.org/10.1016/j.nepr.2025.104320
Wang X, Sazalli N. Mobile applications integrated in blended learning: a systematic literature review. Int J Commun Netw Inf Secur. 2023;15(4):1–10. https://doi.org/10.17762/ijcnis.v15i4.6309
Feriandi Y, Wasliman I, Hanafiah N, Budiman B, Putra AR, Rachmadini N, et al. Blended e-learning on clinical skills learning from the medical student’s perspectives. J Phys: Conf Ser. 2020;1469(1):1–6. https://doi.org/10.1088/1742-6596/1469/1/012055
International Organization for Standardization. ISO 9241-210:2019. Ergonomics of human-system interaction. Part 210, human-centred design for interactive systems. Geneva, Switzerland: International Organization for Standardization; 2019.
Xcode [Internet]. California, US: Apple Inc; [cited 25 June 2025]. Available from: https://developer.
apple.com/xcode/
TestFlight [Internet]. California, US: Apple Inc; [cited 25 June 2025]. Available from: https://developer.apple.com/testflight/
Rose S, Spinks N, Canhoto A. Management research: applying the principles. New York: Routledge; 2015. https://doi.org/10.4324/9781315819198
Hsieh H-F, Shannon S. Three approaches to qualitative content analysis. Qual Health Res. 2005;15:1277–88. https://doi.org/10.1177/1049732305276687
Kondracki NL, Wellman NS, Amundson DR. Content analysis: review of methods and their applications in nutrition education. J Nutr Educ Behav. 2002;34(4):224–30. https://doi.org/10.1016/S1499-4046(06)60097-3
Vaismoradi M, Turunen H, Bondas T. Content analysis and thematic analysis: implications for conducting a qualitative descriptive study. Nurs Health Sci. 2013;15(3):398–405. https://doi.org/10.1111/nhs.12048
Grbich C. Qualitative data analysis: an introduction. 2nd ed. Thousand Oaks, CA: SAGE Publications; 2013. https://doi.org/10.4135/9781529799606
Brooke J. SUS: a “quick and dirty” usability scale. In: Jordan P, Thomas B, Weerdmeester B, McClelland I, editors. Usability evaluation in industry. London: Taylor and Francis; 1996. p. 189–94.
Bangor A, Kortum P, Miller J. Determining what individual SUS scores mean: adding an adjective rating scale. J Usability Stud. 2009;4(3):114–23.
Payne KFB, Wharrad H, Watts K. Smartphone and medical related app use among medical students and junior doctors in the United Kingdom (UK): a regional survey. BMC Med Inform Decis Mak. 2012;12(1):1–11. https://doi.org/10.1186/1472-6947-12-121.
Robinson T, Cronin T, Ibrahim H, Jinks M, Molitor T, Newman J, et al. Smartphone use and acceptability among clinical medical students: a questionnaire-based study. J Med Syst. 2013;37(3):1–7. https://doi.org/10.1007/s10916-013-9936-5
Masters K, Ellaway RH, Topps D, Archibald D, Hogue RJ. Mobile technologies in medical education: AMEE guide no. 105. Med Teach. 2016;38(6):537–49. https://doi.org/10.3109/0142159X.2016.1141190
Aldekhyyel R, Almulhem J, Binkheder S, Almulhem M, Mohamed E, Aldekhyyel S, et al. User perceptions and use of decision support medical apps among medical students: cross-sectional study. Stud Health Technol Inform. 2024;310:1216–20. https://doi.org/10.3233/SHTI231158
Alameddine M, Soueidan H, Makki M, Tamim H, Hitti E. The use of smart devices by care providers in emergency departments: cross-sectional survey design. JMIR mHealth uHealth. 2019;7(6):1–18. https://doi.org/10.2196/13614
Kerry G, Gokani S, Rasasingam D, Zargaran A, Ash J, Mittal A. Patient perception of smartphone usage by doctors. Smart Homecare Technol TeleHealth. 2017;4:31–4. https://doi.org/10.2147/SHTT.S122792
Mannino RG, Arconada Alvarez SJ, Greenleaf M, Parsell M, Mwalija C, Lam WA. Navigating the complexities of mobile medical app development from idea to launch, a guide for clinicians and biomedical researchers. BMC Med. 2023;21(1):1–11. https://doi.org/10.1186/s12916-023-02833-7
Indika A, Lee C, Wang H, Lisoway J, Peruma A, Kazman R. Exploring accessibility trends and
challenges in mobile app development: a study of stack overflow questions. 2024:1–10. https://
doi.org/10.24251/HICSS.2025.885
Senjam SS, Manna S, Bascaran C. Smartphones-based assistive technology: accessibility features and apps for people with visual impairment, and its usage, challenges, and usability testing. Clin Optom. 2021;13:311–22. https://doi.org/10.2147/OPTO.S336361
Cai Y, Nazeha N, Perera S, Thiery AH, Girard MJA, Lee CE, et al. A decision-support tool for funding health innovations at a tertiary academic medical center. Int J Technol Assess. Health Care. 2023;39(1):1–9. https://doi.org/10.1017/S0266462323000028
Oliveira EY, Crosewski NI, Silva ALM, Ribeiro CTD, de Oliveira CM, Fogaça RTH, et al. Profile of educational technology use by medical students and evaluation of a new mobile application designed for the study of human physiology. J Med Syst. 2019;43(10):1–7. https://doi.org/10.1007/s10916-019-1438-7
Gladman T, Tylee G, Gallagher S, Mair J, Rennie SC, Grainger R. A tool for rating the value of health education mobile apps to enhance student learning (MARuL): Development and usability study. JMIR mHealth uHealth. 2020;8(7):1–18. https://doi.org/10.2196/18015
Gladman T, Tylee G, Gallagher S, Mair J, Grainger R. Measuring the quality of clinical skills mobile apps for student learning: systematic search, analysis, and comparison of two measurement scales. JMIR mHealth uHealth. 2021;9(4):1–18. https://doi.org/10.2196/25377
Khan H, Syed W, Golani S, Kadir MM, Ashraf M. Perceived usefulness of smartphone medical apps as theoretical and clinical learning aids among medical students in Pakistan: a cross-sectional study. Cureus. 2023;15(7):1–10. https://doi.org/10.7759/cureus.41682