跳到正文
原文
Nature 健康护理主题·· 2026-10-16AI 评分54

一种用于大规模临床信息提取的多阶段验证框架

A multi-stage validation framework for trustworthy large-scale clinical information extraction using large language models

AI 导读

研究人员提出了一种多阶段验证框架,利用大型语言模型从未结构化的健康记录中提取临床信息。该框架结合了提示校准、规则过滤、语义接地、法官模型评估、专家审查和预测效度分析,提高了提取的准确性和可信度。

正文

npj Digital Medicine (2026) Cite this article

We’re sharing this article early to provide faster access to peer-reviewed, accepted research. It is citable and carries a permanent DOI. This version is subject to further edits and will be replaced automatically by the final Version of Record. All legal disclaimers apply.

Abstract

Large language models (LLMs) show promise for extracting clinically meaningful information from unstructured health records, yet real-world deployment is constrained by limited scalable, trustworthy validation approaches. Conventional evaluation relies on annotation-intensive reference standards and incomplete structured data, limiting population-scale use. We propose a multi-stage validation framework for LLM-based clinical information extraction that enables rigorous assessment under weak supervision by integrating complementary validation strategies. The framework combines prompt calibration, rule-based filtering, semantic grounding, judge-LLM evaluation, expert review, and predictive validity analysis to quantify uncertainty without exhaustive manual annotation. We applied this framework to extract substance use disorder (SUD) diagnoses across 11 substance categories from 919,783 clinical notes. Rule-based filtering and semantic grounding removed 14.59% of LLM-positive extractions that were unsupported or structurally implausible. For high-uncertainty cases, the judge LLM showed substantial agreement with subject matter expert. Using judge evaluations as references, the primary LLM achieved an F1 score of 0.80. LLM-extracted SUD diagnoses predicted subsequent engagement in SUD specialty care more accurately than structured-data baselines. By unifying complementary validation strategies, this work presents a generalizable human-in-the-loop framework for establishing the trustworthiness of LLM-based clinical information extraction at scale while minimizing the need for exhaustive manual annotation.

Acknowledgements

This manuscript has been authored by UT-Battelle, LLC, under contract DE-AC05-00OR22725 with the US Department of Energy (DOE). The US government retains and the publisher, by accepting the article for publication, acknowledges that the US government retains a nonexclusive, paid-up, irrevocable, worldwide license to publish or reproduce the published form of this manuscript, or allow others to do so, for US government purposes. DOE will provide public access to these results of federally sponsored research in accordance with the DOE Public Access Plan (https://www.energy.gov/doe-public-access-plan). The initiative aims to improve care for Veterans within the Veterans Health Administration (VHA) and involves models specifically tailored to the VHA system. The results are not generalizable outside of VHA, and the studies are formally considered non-research by the VHA. Oak Ridge National Laboratory received Institutional Review Board (IRB) approval for the secondary use of patient data, which complies with internal policies and standards. The authors acknowledge the broader partnership and express gratitude to the Veterans receiving care at the U.S. Department of Veterans Affairs (VA).

Funding

This initiative is sponsored by the U.S. Department of Veterans Affairs (VA) and utilizes VA-funded resources from the Knowledge Discovery Infrastructure (KDI) at Oak Ridge National Laboratory under the Department of Energy (DOE) Office of Science. This work is a collaboration between VA and DOE. VA has provided data and biomedical expertise to guide the study.

Author information

Authors and Affiliations

  1. Oak Ridge National Laboratory, Oak Ridge, TN, USA

    Maria Mahbub, Josh Arnold, Caitlin Rizy, Sudarshan Srinivasan & Ioana Danciu

  2. Program Evaluation and Resource Center, Office of Mental Health and Office of Suicide Prevention, Department of Veterans Affairs, Menlo Park, CA, USA

    Gregory M. Dams, Elliot M. Fielstein, Elizabeth M. Oliva & Jodie Trafton

  3. Vanderbilt University Medical Center, Nashville, TN, USA

    Elliot M. Fielstein

  4. VA Maryland Health Care System, Baltimore, MD, USA

    Minu A. Aghevli

  5. VA Desert Pacific Healthcare Network, Long Beach, CA, USA

    Kamonica L. Craig

  6. VA Connecticut Health Care System, West Haven, CT, USA

    Joseph Erdos

  7. Yale School of Medicine, New Haven, CT, USA

    Joseph Erdos

Authors

  1. Maria Mahbub
  2. Gregory M. Dams
  3. Josh Arnold
  4. Caitlin Rizy
  5. Sudarshan Srinivasan
  6. Elliot M. Fielstein
  7. Minu A. Aghevli
  8. Kamonica L. Craig
  9. Elizabeth M. Oliva
  10. Joseph Erdos
  11. Jodie Trafton
  12. Ioana Danciu

Corresponding author

Correspondence to Maria Mahbub.

Ethics declarations

Competing interests

The authors declare no competing interests.

Additional information

Publisher’s note Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Open Access This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made. The images or other third party material in this article are included in the article’s Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article’s Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit http://creativecommons.org/licenses/by/4.0/.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Mahbub, M., Dams, G.M., Arnold, J. et al. A multi-stage validation framework for trustworthy large-scale clinical information extraction using large language models. npj Digit. Med. (2026). https://doi.org/10.1038/s41746-026-03341-7

Download citation

  • Received: 27 March 2026

  • Accepted: 19 September 2026

  • Published: 16 October 2026

  • DOI: https://doi.org/10.1038/s41746-026-03341-7

来源:Nature 健康护理主题 · nature.com