多模态AI评估系统MAPLES在医学OSCE考核中的首次前瞻性部署
Prospective deployment of multimodal AI grading for medical student OSCEs
UT Southwestern开发MAPLES多模态AI评分系统,于2025年秋季对222名医学生进行OSCE考核,涵盖笔记、音频和视频。系统在低分复核中AI与标准化患者评价者一致率达85.7–92.4%,医生复核616例分歧时与AI一致率76.0%。AI初筛使人工评分环节减少92.3%。
- Article
- Open access
- Published: 07 October 2026
- Huong-Tra Ngo ORCID: orcid.org/0000-0001-5016-33911 na1,
- Ameer Hamza Shakur1 na1,
- Michael Holcomb ORCID: orcid.org/0000-0002-0595-14761,
- Shinyoung Kang1,
- David Hein ORCID: orcid.org/0000-0002-8625-95281,
- Judah Gruen2,
- Hunter Schuler1,
- Philip Jarrett ORCID: orcid.org/0000-0003-4534-85563,
- Thomas Dalton ORCID: orcid.org/0000-0002-8351-83362,
- Krystle Campbell ORCID: orcid.org/0000-0001-6920-78923,4,
- Daniel Scott4,5 &
- …
- Andrew R. Jamieson ORCID: orcid.org/0000-0001-5416-93791
npj Digital Medicine (2026) Cite this article
We’re sharing this article early to provide faster access to peer-reviewed, accepted research. It is citable and carries a permanent DOI. This version is subject to further edits and will be replaced automatically by the final Version of Record. All legal disclaimers apply.
Abstract
Objective Structured Clinical Examinations (OSCEs) are central to assessing medical student clinical competence, but human grading imposes substantial burden. We report, to our knowledge, the first prospective single-center deployment of an integrated multimodal artificial intelligence (AI) system for OSCE grading in undergraduate medical education, spanning notes, audio, and video. We present MAPLES (Multimodal Assessment Pipeline for Learning Encounter Scoring), a rubric-driven zero-shot multimodal LLM system deployed in Fall 2025 for 222 students (72,907 retained item-level scores). In the routed low-scoring review set, tolerant AI–standardized patient evaluator (SPE) agreement was 85.7–92.4% by modality. In a selected set of 616 disagreements adjudicated with visible score provenance, physician scores matched AI on 76.0% of items and the SPE on 19.0%. Human scoring passes fell by 92.3% versus a modeled single-pass manual comparator. Together, these findings show that multimodal AI first-pass scoring can be embedded in routine OSCE operations, with human review on the low-scoring tail and physician adjudication of selected disagreements.
Acknowledgements
We are grateful to the standardized patient evaluators and Simulation Center staff who conducted and reviewed the OSCEs and the AI supervisory committee for oversight. Support for this work was provided by UT Southwestern institutional funds (Office of the President).
Author information
Author notes
These authors contributed equally: Huong-Tra Ngo, Ameer Hamza Shakur.
Authors and Affiliations
Lyda Hill Department of Bioinformatics, UT Southwestern Medical Center, Dallas, TX, USA
Huong-Tra Ngo, Ameer Hamza Shakur, Michael Holcomb, Shinyoung Kang, David Hein, Hunter Schuler & Andrew R. Jamieson
Department of Internal Medicine, UT Southwestern Medical Center, Dallas, TX, USA
Judah Gruen & Thomas Dalton
Department of Emergency Medicine, UT Southwestern Medical Center, Dallas, TX, USA
Philip Jarrett & Krystle Campbell
Simulation Center, UT Southwestern Medical Center, Dallas, TX, USA
Krystle Campbell & Daniel Scott
Department of Surgery, UT Southwestern Medical Center, Dallas, TX, USA
Daniel Scott
Authors
- Huong-Tra Ngo
- Ameer Hamza Shakur
- Michael Holcomb
- Shinyoung Kang
- David Hein
- Judah Gruen
- Hunter Schuler
- Philip Jarrett
- Thomas Dalton
- Krystle Campbell
- Daniel Scott
- Andrew R. Jamieson
Corresponding author
Correspondence to Andrew R. Jamieson.
Ethics declarations
Competing interests
The authors declare the following competing interests. The Board of Regents of The University of Texas System is the applicant for four pending patent applications related to this manuscript: (i) U.S. non-provisional patent application 19/699,629, filed 5 June 2026 and claiming priority to provisional application 63/819,115 (inventors: Andrew Jamieson, Michael Holcomb, Gaudenz Danuser, Daniel Scott, Krystle Campbell, and Thomas Dalton), covering context-aware grading of learner responses; (ii) PCT/US2026/022629, filed 7 April 2026 (inventors: Andrew Jamieson and Michael Holcomb), covering audio-visual segmentation and rubric-based OSCE grading; (iii) U.S. provisional patent application 64/140,783, filed 25 August 2026 (inventors: Andrew Jamieson and Shinyoung Kang), covering AI-assisted rubric authoring and refinement; and (iv) U.S. provisional patent application 64/136,854, filed 19 August 2026 (inventors: Andrew Jamieson, Michael Holcomb, Thomas Dalton, and Ngoc Le Van Horn), covering multimodal clinical-skills assessment and review workflows. The remaining authors declare no competing interests.
Additional information
Publisher’s note Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
Rights and permissions
Open Access This article is licensed under a Creative Commons Attribution-NonCommercial-NoDerivatives 4.0 International License, which permits any non-commercial use, sharing, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if you modified the licensed material. You do not have permission under this licence to share adapted material derived from this article or parts of it. The images or other third party material in this article are included in the article’s Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article’s Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit http://creativecommons.org/licenses/by-nc-nd/4.0/.
About this article
Cite this article
Ngo, HT., Shakur, A.H., Holcomb, M. et al. Prospective deployment of multimodal AI grading for medical student OSCEs. npj Digit. Med. (2026). https://doi.org/10.1038/s41746-026-03352-4
Received: 24 July 2026
Accepted: 21 September 2026
Published: 07 October 2026
DOI: https://doi.org/10.1038/s41746-026-03352-4
来源:Nature 健康护理主题 · nature.com