Dr.Copilot 多语言检索增强型临床 AI 框架的实证评估
Real-world evaluation of a multilingual agentic retrieval-augmented clinical AI framework for evidence-grounded medical decision support
Dr.Copilot 多语言检索增强型临床 AI 框架在 70 天部署中记录 956 个会话线程及 4,226 次工具调用事件。在独立 75 例外部验证中,Dr.Copilot 加权得分 4.557,显著优于 GPT-5.5 的 4.217(p = 0.00023),且临床重要错误率更低(5/75 vs. 16/75,p = 0.00098)。
- Article
- Open access
- Published: 14 October 2026
- Mohammad Hossein Imani ORCID: orcid.org/0009-0006-7773-88061,
- Mohammad Hasannejad2,
- Amir Hossein Imani3 &
- …
- Mojtaba Shafiekhani4
Scientific Reports (2026) Cite this article
We’re sharing this article early to provide faster access to peer-reviewed, accepted research. It is citable and carries a permanent DOI. This version is subject to further edits and will be replaced automatically by the final Version of Record. All legal disclaimers apply.
Abstract
Large language models may support clinical information tasks, but hallucination, limited evidence traceability, and inconsistent workflow integration remain barriers to clinical deployment. We evaluated Dr.Copilot, a multilingual retrieval-augmented clinical AI framework combining hybrid evidence retrieval, specialized tools, and model-directed orchestration. During a 70-day deployment, 835 registered users generated 956 conversation threads; the analytics log contained 22,574 events and 4,226 tool-use events. In the original 64-prompt expert benchmark, Dr.Copilot and GPT-5.5 had comparable weighted scores (4.607 vs. 4.561; paired difference 0.046, 95% CI -0.064 to 0.140; p = 0.103). In a new independent 75-case, fixed-backbone external validation, Dr.Copilot scored 4.557 versus 4.217 for GPT-5.5 (difference 0.340, 95% CI 0.181 to 0.519; p = 0.00023), with fewer clinically important errors (5/75 vs. 16/75; p = 0.00098). A 30-case four-arm GPT-5.4 ablation found the highest mean score with standard RAG (4.813), while additional prespecified-tool and model-directed routing conditions did not yield uniform further gains. Targeted safety, pediatric, multilingual, citation, temporal, and abstention analyses identified both strengths and important failure modes. These findings support retrieval as the clearest incremental contributor in this evaluation, while underscoring that agentic tool orchestration requires direct validation and does not itself establish clinical benefit or patient-safety improvement.
Acknowledgements
The authors sincerely thank the internal medicine specialist and clinical pharmacist who independently and blindly evaluated the benchmark responses.
Funding
This study was supported by the Vice-Chancellorship for Research and Technology, Shiraz University of Medical Sciences (SUMS).
Author information
Authors and Affiliations
Student Research Committee, School of Medicine, Shiraz University of Medical Sciences, Shiraz, Iran
Mohammad Hossein Imani
Student Research Committee, School of Pharmacy, Shiraz University of Medical Sciences, Shiraz, Iran
Mohammad Hasannejad
Department of Radiology, Shiraz University of Medical Sciences, Shiraz, Iran
Amir Hossein Imani
Department of Clinical Pharmacy, Faculty of Pharmacy, Shiraz University of Medical Sciences, Shiraz, Iran
Mojtaba Shafiekhani
Authors
- Mohammad Hossein Imani
- Mohammad Hasannejad
- Amir Hossein Imani
- Mojtaba Shafiekhani
Corresponding author
Correspondence to Mojtaba Shafiekhani.
Ethics declarations
Competing interests
Mohammad Hossein Imani and Mohammad Hasannejad are members of the Dr.Copilot development team. Amir Hossein Imani and Mojtaba Shafiekhani declare no competing interests.
Additional information
Publisher’s note
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
Rights and permissions
Open Access This article is licensed under a Creative Commons Attribution-NonCommercial-NoDerivatives 4.0 International License, which permits any non-commercial use, sharing, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if you modified the licensed material. You do not have permission under this licence to share adapted material derived from this article or parts of it. The images or other third party material in this article are included in the article’s Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article’s Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit http://creativecommons.org/licenses/by-nc-nd/4.0/.
About this article
Cite this article
Imani, M.H., Hasannejad, M., Imani, A.H. et al. Real-world evaluation of a multilingual agentic retrieval-augmented clinical AI framework for evidence-grounded medical decision support. Sci Rep (2026). https://doi.org/10.1038/s41598-026-75369-4
Received: 26 June 2026
Accepted: 06 October 2026
Published: 14 October 2026
DOI: https://doi.org/10.1038/s41598-026-75369-4
Keywords
来源:Nature 健康护理主题 · nature.com