跳到正文
原文
Nature 健康护理主题·· 13 天前AI 评分40

五种大型语言模型在传染病相关问题上的表现

Performance of five large language models (LLM) on infectious diseases related questions

AI 导读

五种大型语言模型(ChatGPT-4.1、ChatGPT-4o-mini、Claude、Copilot、Gemini)在药学课程传染病相关问题上的准确率范围为89.4%至91.2%,其中ChatGPT-4o-mini表现最佳,Claude和Copilot表现最差。模型在药理学和治疗学方面表现最佳,而在微生物学和化学方面表现较差。图像解释能力也较弱。

正文

We’re sharing this article early to provide faster access to peer-reviewed, accepted research. It is citable and carries a permanent DOI. This version is subject to further edits and will be replaced automatically by the final Version of Record. All legal disclaimers apply.

Abstract

As there i​s growing interest in utilizing large language models (LLMs) in healthcare education, there are still concerns about their accuracy. The primary study question is to evaluate the accuracy of five LLMs in answering infectious disease (ID) related questions. We compared five LLMs—ChatGPT-4.1, ChatGPT-4o-mini, Claude, Copilot, and Gemini—on 227 exam items from an infectious diseases (ID) course in a Doctor of Pharmacy program. Items were categorized by topic, Bloom’s taxonomy, and format. LLM accuracy across different categories was summarized using descriptive statistics. The Chi-Square test was used to compare results among the different LLMs. Agreement across responses from different LLMs was investigated via Cohen’s Kappa Coefficient. The overall accuracy across all models ranged from 89.4% to 91.2% with ChatGPT 4o-mini scoring highest and Claude and Copilot scoring lowest. LLMs had highest accuracy answering pharmacology questions (93.6%), followed by therapeutics (93.0%), PK/PD (91.1%), microbiology (87.5%), and medicinal chemistry (84%). Medicinal chemistry and microbiology performed significantly worse compared to therapeutics (p < 0.05). Questions categorized as Evaluate/Create performed significantly worse compared to Remember/Understand category (85.6% vs. 92.0%, p < 0.05). The accuracy of questions with a picture was significantly worse than those with no picture (69% vs 91%, respectively). Inter-model agreement was moderate (κ = 0.497–0.613). Contemporary LLMs show high accuracy on ID coursework, with weaknesses in chemistry/microbiology content, higher-order reasoning, and image interpretation; careful integration and guidance are warranted.

Funding

No funding was received for this project.

Author information

Authors and Affiliations

  1. University of the Pacific School of Pharmacy, Stockton, CA, 95211, USA

    Jenana Maker, Myo Kyong Kim & Xin Guo

Authors

  1. Jenana Maker
  2. Myo Kyong Kim
  3. Xin Guo

Corresponding author

Correspondence to Jenana Maker.

Ethics declarations

Competing interests

The authors declare no competing interests.

Ethics, consent to participate, and consent to publish

Not applicable.

Additional information

Publisher's note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Open Access This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made. The images or other third party material in this article are included in the article’s Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article’s Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit http://creativecommons.org/licenses/by/4.0/.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Maker, J., Kim, M.K. & Guo, X. Performance of five large language models (LLM) on infectious diseases related questions. Sci Rep (2026). https://doi.org/10.1038/s41598-026-78926-z

Download citation

  • Received: 15 December 2025

  • Accepted: 29 October 2026

  • Published: 07 November 2026

  • DOI: https://doi.org/10.1038/s41598-026-78926-z

Keywords

来源:Nature 健康护理主题 · nature.com