跳到正文
原文
Nature 健康护理主题·· 2026-10-13AI 评分60

中医大语言模型评估:提示策略与模型依赖效应影响可靠部署

Model-dependent prompt effects challenge reliable deployment of medical LLMs in traditional Chinese medicine

AI 导读

该发表于npj Digital Medicine的研究评估了16种大语言模型在7种提示策略下的中医执照考试表现,无提示工程下准确率在35.4%至85.9%间波动。成对分析显示提示策略的模型交互作用导致表现分歧,提示优化使某模型准确率从49.7%升至87.4%,却降低其他模型表现,研究团队据此指出中医LLM部署需同时验证模型选择与提示策略。

正文

npj Digital Medicine (2026) Cite this article

We’re sharing this article early to provide faster access to peer-reviewed, accepted research. It is citable and carries a permanent DOI. This version is subject to further edits and will be replaced automatically by the final Version of Record. All legal disclaimers apply.

Abstract

Although contemporary large language models (LLMs) have achieved satisfactory results in Western medical knowledge, their performance on traditional Chinese medicine (TCM) remains underexplored, and the determinants of performance variation are poorly understood. We evaluated 16 LLMs under seven prompt strategies on 475 questions from the 2025 Taiwan national TCM licensing examination, a public benchmark of TCM knowledge. Without prompt engineering, accuracy ranged from 35.4% to 85.9%, with approximately half passing. Among the evaluated model versions, adjusted accuracy differed substantially when models were grouped by family (for example, Claude-family models, 84% vs. GPT-4, 43%); these family-grouped comparisons were descriptive because only 16 individual models were evaluated. Paired prompt analyses showed modest average absolute effects, ranging from a 2.03-percentage-point decrease with implicit CoT to a 3.04-percentage-point increase with explicit CoT. However, prompt–model interactions produced substantial model-level divergence: prompt optimization reversed failing performance in one model (87.4% vs. 49.7%) while reducing accuracy in others. No single prompt strategy yielded consistent benefits across the evaluated models. Among five selected lower-performing models evaluated under explicit CoT prompting, misunderstanding of foundational TCM concepts was the most frequent error. These findings indicate that reliable LLM deployment in TCM requires empirical validation of both model choice and prompt strategy: model selection strongly shapes baseline capability, while prompt-model interactions can materially alter performance.

Acknowledgements

Not applicable.

Funding

This project was funded by the Ministry of Health and Welfare (MOHW113-CMAP-M-113-000002-D, MOHW114-CMAP-M-113-000005-C and MOHW115-CMAP-M-113-000005-C), and the National Science and Technology Council in Taiwan (NSTC114-2320-B-182-008 and 115-2320-B-182-008). The funders had no role in study design, data collection, analysis, interpretation, manuscript preparation, or the decision to submit the work for publication.

Author information

Authors and Affiliations

  1. Division of Chinese Acupuncture and Traumatology, Center of Traditional Chinese Medicine, Chang Gung Memorial Hospital, Taoyuan, Taiwan

    Liang-Wei Tseng

  2. Division of Chinese Internal Medicine, Center for Traditional Chinese Medicine, Chang Gung Memorial Hospital, Taoyuan, Taiwan

    Yi-Chin Lu & Hsing-Yu Chen

  3. Google International LLC Taiwan Branch, Taipei, Taiwan

    Liang-Chi Tseng

  4. School of Medicine, Faculty of Medicine, National Yang-Ming Chiao Tung University, Taipei, Taiwan

    Yu-Chun Chen

  5. Institute of Hospital and Health Care Administration, National Yang-Ming Chiao Tung University, Taipei, Taiwan

    Yu-Chun Chen

  6. Taipei Veterans General Hospital, Yuli Branch, Yuli Township, Hualien, Taiwan

    Yu-Chun Chen

  7. School of Traditional Chinese Medicine, College of Medicine, Chang Gung University, Taoyuan, Taiwan

    Hsing-Yu Chen

Authors

  1. Liang-Wei Tseng
  2. Yi-Chin Lu
  3. Liang-Chi Tseng
  4. Yu-Chun Chen
  5. Hsing-Yu Chen

Corresponding author

Correspondence to Hsing-Yu Chen.

Ethics declarations

Competing interests

The authors declare no competing interests.

Additional information

Publisher’s note Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Open Access This article is licensed under a Creative Commons Attribution-NonCommercial-NoDerivatives 4.0 International License, which permits any non-commercial use, sharing, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if you modified the licensed material. You do not have permission under this licence to share adapted material derived from this article or parts of it. The images or other third party material in this article are included in the article’s Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article’s Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit http://creativecommons.org/licenses/by-nc-nd/4.0/.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Tseng, LW., Lu, YC., Tseng, LC. et al. Model-dependent prompt effects challenge reliable deployment of medical LLMs in traditional Chinese medicine. npj Digit. Med. (2026). https://doi.org/10.1038/s41746-026-03397-5

Download citation

  • Received: 14 May 2026

  • Accepted: 30 September 2026

  • Published: 13 October 2026

  • DOI: https://doi.org/10.1038/s41746-026-03397-5

来源:Nature 健康护理主题 · nature.com