ChatGPT、Claude、DeepSeek、Doubao 回答活动性 Charcot 神经骨关节病患者提问的安全性与指南一致性对比研究
Safety and guideline concordance of public large language model responses to patient-facing questions on active charcot neuro-osteoarthropathy
活动性 Charcot 神经骨关节病(CNO)患者提问测试中,ChatGPT 的信息质量、指南一致性与安全性相对优于 Claude、DeepSeek 和 Doubao。
Abstract
Active Charcot neuro-osteoarthropathy (CNO) is a time-sensitive condition in which incomplete or misleading patient information may delay offloading, clinical assessment, and appropriate imaging. This study compared the informational quality, guideline concordance, safety, source verifiability, and readability of public large language model responses to patient-facing questions on active CNO. Twenty standardized questions covering 10 clinical domains were derived from six guideline or consensus sources and locked before response collection. ChatGPT, Claude, DeepSeek, and Doubao were evaluated under their recorded native public-facing configurations, with one first-complete response retained per question, yielding 80 responses. Two clinicians independently rated platform-masked responses using DISCERN, EQIP, GQS, the JAMA benchmark, a question-specific guideline-concordance scale, and a prespecified safety-error taxonomy; discordant assessments were adjudicated by a senior clinician. Sources were manually verified, and response length and six readability indices were assessed. Platform comparisons were paired by question with multiplicity-adjusted testing. Interrater agreement was high. All informational-quality outcomes differed across platforms (all global Friedman P < 0.001). ChatGPT achieved the highest median DISCERN, EQIP, GQS, JAMA, and guideline-concordance scores; median guideline-concordance scores were 5.0, 3.5, 3.0, and 2.0 for ChatGPT, Claude, DeepSeek, and Doubao, respectively (P < 0.001). Any safety-critical error occurred in 0/20 (0.0%), 14/20 (70.0%), 16/20 (80.0%), and 20/20 (100.0%) responses for ChatGPT, Claude, DeepSeek, and Doubao, respectively. Moderate or major errors occurred in 0.0%, 10.0%, 45.0%, and 60.0%, whereas major errors occurred only with Doubao (3/20, 15.0%). Authentic sources with auditable claim–source linkage were identified in all ChatGPT responses and in none from the other platforms. Unsupported clinically meaningful numerical claims occurred in 0%, 60%, 75%, and 65% of responses, respectively. Response length varied substantially across platforms. Although pooled analyses showed inverse associations between response length and all five quality-related measures, within-platform and adjusted sensitivity analyses did not show a consistent negative association. Under the prespecified study conditions, the tested ChatGPT public-facing configuration showed relative advantages in informational quality, guideline concordance, source verifiability, and safety across this fixed benchmark. These findings represent a single-response, time-specific snapshot of the recorded public-product configurations rather than a stable ranking of underlying models. However, fluent or lengthy responses were not necessarily clinically useful, and some responses from the other platforms omitted or obscured clinically important actions relevant to timely care. The findings support further prospective evaluation of whether well-governed public LLMs can contribute to disease-related health literacy and timely care-seeking.
Funding
This work was supported by the National Natural Science Foundation of China (No. 82374490); the Natural Science Foundation of Henan Province (No. 222300420486); the Henan University Science and Technology Innovation Team Program (No. 24IRTSTHN040); the Scientific Research Projects of Traditional Chinese Medicine in Henan Province (Nos. 2023ZYZD06, 2021ZY2010, and 2019ZY2035); the Zhengzhou Science and Technology Benefit to the People Program (No. 2023KJHM0009); the Key Research and Development Program of Henan Province (No. 241111311700); the Young and Middle-aged Health Science and Technology Innovation Leading Talents Program of Henan Province (No. LJRC2024020); the Key Project of the Henan Province Traditional Chinese Medicine Scientific Research Special Program (No. 2025ZY1016); and the Henan College Students’ Innovation and Entrepreneurship Training Program (Nos. 202410471006, 202510471007, and 202510471049). The funders had no role in the study design, data collection, analysis, interpretation of the results, preparation of the manuscript, or decision to submit the manuscript for publication.
Author information
Author notes
These authors contributed equally: Bingqi Wei, Yijing Li and Qian Ge.
Authors and Affiliations
School of Orthopedics, Henan University of Chinese Medicine, Zhengzhou, 450002, China
Bingqi Wei, Yijing Li, Qian Ge, Jintao Shen, Shengchao Wu & Shangzeng Wang
Department of Orthopaedics, Henan Provincial Hospital of TCM, Zhengzhou, 450002, China
Bingqi Wei, Yijing Li, Qian Ge, Jintao Shen, Shengchao Wu & Shangzeng Wang
Authors
- Bingqi Wei
- Yijing Li
- Qian Ge
- Jintao Shen
- Shengchao Wu
- Shangzeng Wang
Corresponding author
Correspondence to Shangzeng Wang.
Ethics declarations
Competing interests
The authors declare no competing interests.
Ethics approval and consent to participate
This study analysed only publicly accessible large language model outputs and information contained in published guideline or consensus documents. The term “patient-facing” describes the wording and intended audience of the standardized questions and does not indicate that patients or members of the public participated in the study. Researchers A, B, and C acted solely as investigators conducting source review, benchmark development, platform-masked response rating, and adjudication; they were members of the research team and were not enrolled or evaluated as research participants. No patients or other human research participants were recruited, and no personal, clinical, identifiable, patient-record, or human-tissue data were collected or analysed. No animal subjects were involved. Therefore, institutional ethics approval and informed consent were not required.
Additional information
Publisher’s note
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
Rights and permissions
Open Access This article is licensed under a Creative Commons Attribution-NonCommercial-NoDerivatives 4.0 International License, which permits any non-commercial use, sharing, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if you modified the licensed material. You do not have permission under this licence to share adapted material derived from this article or parts of it. The images or other third party material in this article are included in the article’s Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article’s Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit http://creativecommons.org/licenses/by-nc-nd/4.0/.
About this article
Cite this article
Wei, B., Li, Y., Ge, Q. et al. Safety and guideline concordance of public large language model responses to patient-facing questions on active charcot neuro-osteoarthropathy. Sci Rep (2026). https://doi.org/10.1038/s41598-026-76331-0
Received: 31 July 2026
Accepted: 13 October 2026
Published: 28 October 2026
DOI: https://doi.org/10.1038/s41598-026-76331-0
Keywords
来源:Nature 健康护理主题 · nature.com