Research through Evaluation for Large Language Models in Patient-Clinician Communications
Research through evaluation for large language models in patient-clinician communications
一项研究提出了一种名为 Research through Evaluation (RtE) 的方法,通过与临床医生共同开发评估标准,评估大型语言模型(LLMs)生成的患者咨询回复。
- Article
- Open access
- Published: 20 November 2026
- Yuexing Hao1,3,
- Jason Holmes1,
- Jared Hobson2,
- Alexandra Bennett2,
- Elizabeth L. McKone2,
- Daniel K. Ebner2,
- David M. Routman2,
- Satomi Shiraishi2,
- Samir H. Patel2,
- Nathan Y. Yu1,
- Chris L. Hallemeier2,
- Brooke E. Ball2,
- Saleh Kalantari4,
- Marzyeh Ghassemi3,
- Mark Waddle2 &
- …
- Wei Liu1
Scientific Reports (2026) Cite this article
We’re sharing this article early to provide faster access to peer-reviewed, accepted research. It is citable and carries a permanent DOI. This version is subject to further edits and will be replaced automatically by the final Version of Record. All legal disclaimers apply.
Abstract
Large language models (LLMs) are increasingly used to draft replies to patient messages, but the methods available for evaluating such drafts were developed for other kinds of systems, and expert evaluators are difficult to recruit at the scale that LLM output requires. We propose Research through Evaluation (RtE), an iterative approach in which the evaluation metrics and the grading rubric are developed together with the clinicians who apply them. We applied RtE in a retrospective comparative study of 158 prostate cancer patient inquiries, comparing replies written by the clinical care team with replies generated by a GPT-4-based in-basket bot. Three rounds of human evaluation with clinical professionals (Rounds 1–3) were followed by a fourth round in which four LLM graders scored the same responses with the final rubric. Across the human rounds, the design moved from disclosed-source comparison to randomized single-blind grading, Conciseness was replaced by Clarity and Trustworthiness by Empathy, and items on editing effort and time to answer were added. In the final clinician evaluation, clinical care team replies scored higher on Completeness, Correctness and Clarity, in-basket bot replies scored higher on Empathy, and in-basket bot replies more often needed major editing. All four LLM graders rated the in-basket bot replies more favourably than the clinical care team replies, and their absolute agreement with one another was low (Krippendorff's alpha 0.07–0.18) compared with that of the clinician graders (0.53–0.81). RtE produced a documented evaluation instrument in which every revision can be traced to the round that prompted it, and the results indicate that zero-shot LLM graders cannot yet replace clinician evaluation in this setting.
Funding
This research was supported by the National Institutes of Health (NIH) R01EB293388, National Cancer Institute (NCI) R01CA280134, Schmidt Sciences, Cornell University Center for Racial Justice and Equitable Futures, Cornell Center for Social Sciences (CCSS), and the Kemper Marley Foundation. The funders had no role in study design, data collection and analysis, decision to publish, or preparation of the manuscript.
Author information
Authors and Affiliations
Department of Radiation Oncology, Mayo Clinic, Phoenix, AZ, USA
Yuexing Hao, Jason Holmes, Nathan Y. Yu & Wei Liu
Department of Radiation Oncology, Mayo Clinic, Rochester, MN, USA
Jared Hobson, Alexandra Bennett, Elizabeth L. McKone, Daniel K. Ebner, David M. Routman, Satomi Shiraishi, Samir H. Patel, Chris L. Hallemeier, Brooke E. Ball & Mark Waddle
Department of Electrical Engineering and Computer Science, Massachusetts Institute of Technology, Cambridge, MA, USA
Yuexing Hao & Marzyeh Ghassemi
Cornell University, Ithaca, NY, USA
Saleh Kalantari
Authors
- Yuexing Hao
- Jason Holmes
- Jared Hobson
- Alexandra Bennett
- Elizabeth L. McKone
- Daniel K. Ebner
- David M. Routman
- Satomi Shiraishi
- Samir H. Patel
- Nathan Y. Yu
- Chris L. Hallemeier
- Brooke E. Ball
- Saleh Kalantari
- Marzyeh Ghassemi
- Mark Waddle
- Wei Liu
Corresponding authors
Correspondence to Yuexing Hao or Wei Liu.
Ethics declarations
Competing interests
The authors declare no competing interests.
Additional information
Publisher's note
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
Rights and permissions
Open Access This article is licensed under a Creative Commons Attribution-NonCommercial-NoDerivatives 4.0 International License, which permits any non-commercial use, sharing, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if you modified the licensed material. You do not have permission under this licence to share adapted material derived from this article or parts of it. The images or other third party material in this article are included in the article’s Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article’s Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit http://creativecommons.org/licenses/by-nc-nd/4.0/.
About this article
Cite this article
Hao, Y., Holmes, J., Hobson, J. et al. Research through evaluation for large language models in patient-clinician communications. Sci Rep (2026). https://doi.org/10.1038/s41598-026-80958-4
Received: 16 May 2026
Accepted: 12 November 2026
Published: 20 November 2026
DOI: https://doi.org/10.1038/s41598-026-80958-4
来源:Nature 健康护理主题 · nature.com