五种大型语言模型在传染病相关问题上的表现
Performance of five large language models (LLM) on infectious diseases related questions
五种大型语言模型(ChatGPT-4.1、ChatGPT-4o-mini、Claude、Copilot、Gemini)在药学课程传染病相关问题上的准确率范围为89.4%至91.2%,其中ChatGPT-4o-mini表现最佳,Claude和Copilot表现最差。模型在药理学和治疗学方面表现最佳,而在微生物学和化学方面表现较差。图像解释能力也较弱。
We’re sharing this article early to provide faster access to peer-reviewed, accepted research. It is citable and carries a permanent DOI. This version is subject to further edits and will be replaced automatically by the final Version of Record. All legal disclaimers apply.
Abstract
As there is growing interest in utilizing large language models (LLMs) in healthcare education, there are still concerns about their accuracy. The primary study question is to evaluate the accuracy of five LLMs in answering infectious disease (ID) related questions. We compared five LLMs—ChatGPT-4.1, ChatGPT-4o-mini, Claude, Copilot, and Gemini—on 227 exam items from an infectious diseases (ID) course in a Doctor of Pharmacy program. Items were categorized by topic, Bloom’s taxonomy, and format. LLM accuracy across different categories was summarized using descriptive statistics. The Chi-Square test was used to compare results among the different LLMs. Agreement across responses from different LLMs was investigated via Cohen’s Kappa Coefficient. The overall accuracy across all models ranged from 89.4% to 91.2% with ChatGPT 4o-mini scoring highest and Claude and Copilot scoring lowest. LLMs had highest accuracy answering pharmacology questions (93.6%), followed by therapeutics (93.0%), PK/PD (91.1%), microbiology (87.5%), and medicinal chemistry (84%). Medicinal chemistry and microbiology performed significantly worse compared to therapeutics (p < 0.05). Questions categorized as Evaluate/Create performed significantly worse compared to Remember/Understand category (85.6% vs. 92.0%, p < 0.05). The accuracy of questions with a picture was significantly worse than those with no picture (69% vs 91%, respectively). Inter-model agreement was moderate (κ = 0.497–0.613). Contemporary LLMs show high accuracy on ID coursework, with weaknesses in chemistry/microbiology content, higher-order reasoning, and image interpretation; careful integration and guidance are warranted.
Funding
No funding was received for this project.
Author information
Authors and Affiliations
University of the Pacific School of Pharmacy, Stockton, CA, 95211, USA
Jenana Maker, Myo Kyong Kim & Xin Guo
Authors
- Jenana Maker
- Myo Kyong Kim
- Xin Guo
Corresponding author
Correspondence to Jenana Maker.
Ethics declarations
Competing interests
The authors declare no competing interests.
Ethics, consent to participate, and consent to publish
Not applicable.
Additional information
Publisher's note
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
Rights and permissions
Open Access This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made. The images or other third party material in this article are included in the article’s Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article’s Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit http://creativecommons.org/licenses/by/4.0/.
About this article
Cite this article
Maker, J., Kim, M.K. & Guo, X. Performance of five large language models (LLM) on infectious diseases related questions. Sci Rep (2026). https://doi.org/10.1038/s41598-026-78926-z
Received: 15 December 2025
Accepted: 29 October 2026
Published: 07 November 2026
DOI: https://doi.org/10.1038/s41598-026-78926-z
Keywords
来源:Nature 健康护理主题 · nature.com