中医大语言模型评估:提示策略与模型依赖效应影响可靠部署
Model-dependent prompt effects challenge reliable deployment of medical LLMs in traditional Chinese medicine
该发表于npj Digital Medicine的研究评估了16种大语言模型在7种提示策略下的中医执照考试表现,无提示工程下准确率在35.4%至85.9%间波动。成对分析显示提示策略的模型交互作用导致表现分歧,提示优化使某模型准确率从49.7%升至87.4%,却降低其他模型表现,研究团队据此指出中医LLM部署需同时验证模型选择与提示策略。
- Article
- Open access
- Published: 13 October 2026
- Liang-Wei Tseng1,
- Yi-Chin Lu2,
- Liang-Chi Tseng3,
- Yu-Chun Chen ORCID: orcid.org/0000-0002-4124-19494,5,6 &
- …
- Hsing-Yu Chen2,7
npj Digital Medicine (2026) Cite this article
We’re sharing this article early to provide faster access to peer-reviewed, accepted research. It is citable and carries a permanent DOI. This version is subject to further edits and will be replaced automatically by the final Version of Record. All legal disclaimers apply.
Abstract
Although contemporary large language models (LLMs) have achieved satisfactory results in Western medical knowledge, their performance on traditional Chinese medicine (TCM) remains underexplored, and the determinants of performance variation are poorly understood. We evaluated 16 LLMs under seven prompt strategies on 475 questions from the 2025 Taiwan national TCM licensing examination, a public benchmark of TCM knowledge. Without prompt engineering, accuracy ranged from 35.4% to 85.9%, with approximately half passing. Among the evaluated model versions, adjusted accuracy differed substantially when models were grouped by family (for example, Claude-family models, 84% vs. GPT-4, 43%); these family-grouped comparisons were descriptive because only 16 individual models were evaluated. Paired prompt analyses showed modest average absolute effects, ranging from a 2.03-percentage-point decrease with implicit CoT to a 3.04-percentage-point increase with explicit CoT. However, prompt–model interactions produced substantial model-level divergence: prompt optimization reversed failing performance in one model (87.4% vs. 49.7%) while reducing accuracy in others. No single prompt strategy yielded consistent benefits across the evaluated models. Among five selected lower-performing models evaluated under explicit CoT prompting, misunderstanding of foundational TCM concepts was the most frequent error. These findings indicate that reliable LLM deployment in TCM requires empirical validation of both model choice and prompt strategy: model selection strongly shapes baseline capability, while prompt-model interactions can materially alter performance.
Acknowledgements
Not applicable.
Funding
This project was funded by the Ministry of Health and Welfare (MOHW113-CMAP-M-113-000002-D, MOHW114-CMAP-M-113-000005-C and MOHW115-CMAP-M-113-000005-C), and the National Science and Technology Council in Taiwan (NSTC114-2320-B-182-008 and 115-2320-B-182-008). The funders had no role in study design, data collection, analysis, interpretation, manuscript preparation, or the decision to submit the work for publication.
Author information
Authors and Affiliations
Division of Chinese Acupuncture and Traumatology, Center of Traditional Chinese Medicine, Chang Gung Memorial Hospital, Taoyuan, Taiwan
Liang-Wei Tseng
Division of Chinese Internal Medicine, Center for Traditional Chinese Medicine, Chang Gung Memorial Hospital, Taoyuan, Taiwan
Yi-Chin Lu & Hsing-Yu Chen
Google International LLC Taiwan Branch, Taipei, Taiwan
Liang-Chi Tseng
School of Medicine, Faculty of Medicine, National Yang-Ming Chiao Tung University, Taipei, Taiwan
Yu-Chun Chen
Institute of Hospital and Health Care Administration, National Yang-Ming Chiao Tung University, Taipei, Taiwan
Yu-Chun Chen
Taipei Veterans General Hospital, Yuli Branch, Yuli Township, Hualien, Taiwan
Yu-Chun Chen
School of Traditional Chinese Medicine, College of Medicine, Chang Gung University, Taoyuan, Taiwan
Hsing-Yu Chen
Authors
- Liang-Wei Tseng
- Yi-Chin Lu
- Liang-Chi Tseng
- Yu-Chun Chen
- Hsing-Yu Chen
Corresponding author
Correspondence to Hsing-Yu Chen.
Ethics declarations
Competing interests
The authors declare no competing interests.
Additional information
Publisher’s note Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
Rights and permissions
Open Access This article is licensed under a Creative Commons Attribution-NonCommercial-NoDerivatives 4.0 International License, which permits any non-commercial use, sharing, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if you modified the licensed material. You do not have permission under this licence to share adapted material derived from this article or parts of it. The images or other third party material in this article are included in the article’s Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article’s Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit http://creativecommons.org/licenses/by-nc-nd/4.0/.
About this article
Cite this article
Tseng, LW., Lu, YC., Tseng, LC. et al. Model-dependent prompt effects challenge reliable deployment of medical LLMs in traditional Chinese medicine. npj Digit. Med. (2026). https://doi.org/10.1038/s41746-026-03397-5
Received: 14 May 2026
Accepted: 30 September 2026
Published: 13 October 2026
DOI: https://doi.org/10.1038/s41746-026-03397-5
来源:Nature 健康护理主题 · nature.com