静默评估揭示AI生成临床消息自动指标的语境依赖性有效性
Silent evaluation reveals context-dependent validity of automated metrics for AI-generated clinical messages
一项研究显示,评估AI生成患者沟通的自动指标(如BLEU、BERTScore)具有语境依赖性。研究利用GPT生成草稿,在30名初级保健医生中分组实施干预(显示草稿,n=21,073)与非干预(不显示草稿,n=45,127)。尽管两组语义得分相近,医生更偏好独立撰写的内容(4.24/5),且BERTScore仅在干预组与医生质量评分相关,表明自动指标无法完全衡量AI生成内容的独立临床价值。
- Article
- Open access
- Published: 21 October 2026
- Gavin Hui1,2,3,4,
- Laura Prichard5,
- Taylor Martin6,
- Sitaram Vangala2,4,
- Joshua Khalili2,
- Sun M. Yoo2,
- Hawkin E. Woo2 &
- …
- Paul J. Lukac4,5,7
npj Digital Medicine (2026) Cite this article
We’re sharing this article early to provide faster access to peer-reviewed, accepted research. It is citable and carries a permanent DOI. This version is subject to further edits and will be replaced automatically by the final Version of Record. All legal disclaimers apply.
Abstract
Evaluation of AI-generated patient communication lacks standardized frameworks, and automated metrics may not reflect clinical utility. We leveraged a staged rollout of GPT-generated draft responses (GDRs) to patient portal messages. Thirty primary care physicians received drafts from September 2023 to August 2024 under two conditions: an intervention group where AI drafts were displayed and edited (n = 21,073), and a non-displayed group where drafts were generated but never shown to physicians, who composed responses independently (n = 45,127). We evaluated messages using BLEU, ROUGE, BERTScore, and Flesch Reading Ease, and then correlated automated metrics with blinded physician ratings of effectiveness, misunderstanding potential, and harm. In the intervention group, 2264 (11%) drafts were incorporated into final messages with high semantic alignment (BERTScore F1 = 0.93). In the non-displayed group, independent physician messages showed moderate similarity to unseen drafts (BERTScore F1 = 0.83). Despite comparable semantic scores, physicians preferred human-authored messages (4.24/5). BERTScore correlated with physician quality ratings in the intervention group but not in the non-displayed. Automated metrics demonstrated context-dependent validity; they assessed editing fidelity but failed to evaluate whether AI-generated content matches independent human performance. Rigorous evaluation of clinical generative AI requires validation of metric meaning within specific use contexts.
Acknowledgements
The authors thank the UCLA Health Clinical Informatics team and participating physicians for their support of this project.
Author information
Authors and Affiliations
Division of Hematology & Oncology, Department of Medicine, University of California, Los Angeles, CA, USA
Gavin Hui
Department of Medicine, David Geffen School of Medicine, University of California, Los Angeles, CA, USA
Gavin Hui, Sitaram Vangala, Joshua Khalili, Sun M. Yoo & Hawkin E. Woo
UCLA Health Jonsson Comprehensive Cancer Center, University of California, Los Angeles, CA, USA
Gavin Hui
Innovations and Outcomes Validation of AI (INOVAi) Center, Division of General Internal Medicine and Health Services Research, David Geffen School of Medicine, University of California, Los Angeles, CA, USA
Gavin Hui, Sitaram Vangala & Paul J. Lukac
UCLA Health Information Technology, UCLA Health, University of California, Los Angeles, CA, USA
Laura Prichard & Paul J. Lukac
Preventative Medicine, Gillett Health, Newport, CA, USA
Taylor Martin
Department of Pediatrics, David Geffen School of Medicine, University of California, Los Angeles, CA, USA
Paul J. Lukac
Authors
- Gavin Hui
- Laura Prichard
- Taylor Martin
- Sitaram Vangala
- Joshua Khalili
- Sun M. Yoo
- Hawkin E. Woo
- Paul J. Lukac
Corresponding author
Correspondence to Gavin Hui.
Ethics declarations
Competing interests
The authors declare no competing interests.
Additional information
Publisher’s note Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
Rights and permissions
Open Access This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made. The images or other third party material in this article are included in the article’s Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article’s Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit http://creativecommons.org/licenses/by/4.0/.
About this article
Cite this article
Hui, G., Prichard, L., Martin, T. et al. Silent evaluation reveals context-dependent validity of automated metrics for AI-generated clinical messages. npj Digit. Med. (2026). https://doi.org/10.1038/s41746-026-03211-2
Received: 21 October 2025
Accepted: 28 August 2026
Published: 21 October 2026
DOI: https://doi.org/10.1038/s41746-026-03211-2
来源:Nature 健康护理主题 · nature.com