跳到正文
原文
Nature 健康护理主题·· 2026-10-21AI 评分55

静默评估揭示AI生成临床消息自动指标的语境依赖性有效性

Silent evaluation reveals context-dependent validity of automated metrics for AI-generated clinical messages

AI 导读

一项研究显示,评估AI生成患者沟通的自动指标(如BLEU、BERTScore)具有语境依赖性。研究利用GPT生成草稿,在30名初级保健医生中分组实施干预(显示草稿,n=21,073)与非干预(不显示草稿,n=45,127)。尽管两组语义得分相近,医生更偏好独立撰写的内容(4.24/5),且BERTScore仅在干预组与医生质量评分相关,表明自动指标无法完全衡量AI生成内容的独立临床价值。

正文

npj Digital Medicine (2026) Cite this article

We’re sharing this article early to provide faster access to peer-reviewed, accepted research. It is citable and carries a permanent DOI. This version is subject to further edits and will be replaced automatically by the final Version of Record. All legal disclaimers apply.

Abstract

Evaluation of AI-generated patient communication lacks standardized frameworks, and automated metrics may not reflect clinical utility. We leveraged a staged rollout of GPT-generated draft responses (GDRs) to patient portal messages. Thirty primary care physicians received drafts from September 2023 to August 2024 under two conditions: an intervention group where AI drafts were displayed and edited (n = 21,073), and a non-displayed group where drafts were generated but never shown to physicians, who composed responses independently (n = 45,127). We evaluated messages using BLEU, ROUGE, BERTScore, and Flesch Reading Ease, and then correlated automated metrics with blinded physician ratings of effectiveness, misunderstanding potential, and harm. In the intervention group, 2264 (11%) drafts were incorporated into final messages with high semantic alignment (BERTScore F1 = 0.93). In the non-displayed group, independent physician messages showed moderate similarity to unseen drafts (BERTScore F1 = 0.83). Despite comparable semantic scores, physicians preferred human-authored messages (4.24/5). BERTScore correlated with physician quality ratings in the intervention group but not in the non-displayed. Automated metrics demonstrated context-dependent validity; they assessed editing fidelity but failed to evaluate whether AI-generated content matches independent human performance. Rigorous evaluation of clinical generative AI requires validation of metric meaning within specific use contexts.

Acknowledgements

The authors thank the UCLA Health Clinical Informatics team and participating physicians for their support of this project.

Author information

Authors and Affiliations

  1. Division of Hematology & Oncology, Department of Medicine, University of California, Los Angeles, CA, USA

    Gavin Hui

  2. Department of Medicine, David Geffen School of Medicine, University of California, Los Angeles, CA, USA

    Gavin Hui, Sitaram Vangala, Joshua Khalili, Sun M. Yoo & Hawkin E. Woo

  3. UCLA Health Jonsson Comprehensive Cancer Center, University of California, Los Angeles, CA, USA

    Gavin Hui

  4. Innovations and Outcomes Validation of AI (INOVAi) Center, Division of General Internal Medicine and Health Services Research, David Geffen School of Medicine, University of California, Los Angeles, CA, USA

    Gavin Hui, Sitaram Vangala & Paul J. Lukac

  5. UCLA Health Information Technology, UCLA Health, University of California, Los Angeles, CA, USA

    Laura Prichard & Paul J. Lukac

  6. Preventative Medicine, Gillett Health, Newport, CA, USA

    Taylor Martin

  7. Department of Pediatrics, David Geffen School of Medicine, University of California, Los Angeles, CA, USA

    Paul J. Lukac

Authors

  1. Gavin Hui
  2. Laura Prichard
  3. Taylor Martin
  4. Sitaram Vangala
  5. Joshua Khalili
  6. Sun M. Yoo
  7. Hawkin E. Woo
  8. Paul J. Lukac

Corresponding author

Correspondence to Gavin Hui.

Ethics declarations

Competing interests

The authors declare no competing interests.

Additional information

Publisher’s note Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Open Access This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made. The images or other third party material in this article are included in the article’s Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article’s Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit http://creativecommons.org/licenses/by/4.0/.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Hui, G., Prichard, L., Martin, T. et al. Silent evaluation reveals context-dependent validity of automated metrics for AI-generated clinical messages. npj Digit. Med. (2026). https://doi.org/10.1038/s41746-026-03211-2

Download citation

  • Received: 21 October 2025

  • Accepted: 28 August 2026

  • Published: 21 October 2026

  • DOI: https://doi.org/10.1038/s41746-026-03211-2

来源:Nature 健康护理主题 · nature.com