跳到正文
原文
Nature 健康护理主题·· 10 小时前AI 评分35

Research through Evaluation for Large Language Models in Patient-Clinician Communications

Research through evaluation for large language models in patient-clinician communications

AI 导读

一项研究提出了一种名为 Research through Evaluation (RtE) 的方法,通过与临床医生共同开发评估标准,评估大型语言模型(LLMs)生成的患者咨询回复。

正文

Scientific Reports (2026) Cite this article

We’re sharing this article early to provide faster access to peer-reviewed, accepted research. It is citable and carries a permanent DOI. This version is subject to further edits and will be replaced automatically by the final Version of Record. All legal disclaimers apply.

Abstract

Large language models (LLMs) are increasingly used to draft replies to patient messages, but the methods available for evaluating such drafts were developed for other kinds of systems, and expert evaluators are difficult to recruit at the scale that LLM output requires. We propose Research through Evaluation (RtE), an iterative approach in which the evaluation metrics and the grading rubric are developed together with the clinicians who apply them. We applied RtE in a retrospective comparative study of 158 prostate cancer patient inquiries, comparing replies written by the clinical care team with replies generated by a GPT-4-based in-basket bot. Three rounds of human evaluation with clinical professionals (Rounds 1–3) were followed by a fourth round in which four LLM graders scored the same responses with the final rubric. Across the human rounds, the design moved from disclosed-source comparison to randomized single-blind grading, Conciseness was replaced by Clarity and Trustworthiness by Empathy, and items on editing effort and time to answer were added. In the final clinician evaluation, clinical care team replies scored higher on Completeness, Correctness and Clarity, in-basket bot replies scored higher on Empathy, and in-basket bot replies more often needed major editing. All four LLM graders rated the in-basket bot replies more favourably than the clinical care team replies, and their absolute agreement with one another was low (Krippendorff's alpha 0.07–0.18) compared with that of the clinician graders (0.53–0.81). RtE produced a documented evaluation instrument in which every revision can be traced to the round that prompted it, and the results indicate that zero-shot LLM graders cannot yet replace clinician evaluation in this setting.

Funding

This research was supported by the National Institutes of Health (NIH) R01EB293388, National Cancer Institute (NCI) R01CA280134, Schmidt Sciences, Cornell University Center for Racial Justice and Equitable Futures, Cornell Center for Social Sciences (CCSS), and the Kemper Marley Foundation. The funders had no role in study design, data collection and analysis, decision to publish, or preparation of the manuscript.

Author information

Authors and Affiliations

  1. Department of Radiation Oncology, Mayo Clinic, Phoenix, AZ, USA

    Yuexing Hao, Jason Holmes, Nathan Y. Yu & Wei Liu

  2. Department of Radiation Oncology, Mayo Clinic, Rochester, MN, USA

    Jared Hobson, Alexandra Bennett, Elizabeth L. McKone, Daniel K. Ebner, David M. Routman, Satomi Shiraishi, Samir H. Patel, Chris L. Hallemeier, Brooke E. Ball & Mark Waddle

  3. Department of Electrical Engineering and Computer Science, Massachusetts Institute of Technology, Cambridge, MA, USA

    Yuexing Hao & Marzyeh Ghassemi

  4. Cornell University, Ithaca, NY, USA

    Saleh Kalantari

Authors

  1. Yuexing Hao
  2. Jason Holmes
  3. Jared Hobson
  4. Alexandra Bennett
  5. Elizabeth L. McKone
  6. Daniel K. Ebner
  7. David M. Routman
  8. Satomi Shiraishi
  9. Samir H. Patel
  10. Nathan Y. Yu
  11. Chris L. Hallemeier
  12. Brooke E. Ball
  13. Saleh Kalantari
  14. Marzyeh Ghassemi
  15. Mark Waddle
  16. Wei Liu

Corresponding authors

Correspondence to Yuexing Hao or Wei Liu.

Ethics declarations

Competing interests

The authors declare no competing interests.

Additional information

Publisher's note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Open Access This article is licensed under a Creative Commons Attribution-NonCommercial-NoDerivatives 4.0 International License, which permits any non-commercial use, sharing, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if you modified the licensed material. You do not have permission under this licence to share adapted material derived from this article or parts of it. The images or other third party material in this article are included in the article’s Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article’s Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit http://creativecommons.org/licenses/by-nc-nd/4.0/.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Hao, Y., Holmes, J., Hobson, J. et al. Research through evaluation for large language models in patient-clinician communications. Sci Rep (2026). https://doi.org/10.1038/s41598-026-80958-4

Download citation

  • Received: 16 May 2026

  • Accepted: 12 November 2026

  • Published: 20 November 2026

  • DOI: https://doi.org/10.1038/s41598-026-80958-4

来源:Nature 健康护理主题 · nature.com