一种用于大规模临床信息提取的多阶段验证框架
A multi-stage validation framework for trustworthy large-scale clinical information extraction using large language models
研究人员提出了一种多阶段验证框架,利用大型语言模型从未结构化的健康记录中提取临床信息。该框架结合了提示校准、规则过滤、语义接地、法官模型评估、专家审查和预测效度分析,提高了提取的准确性和可信度。
- Article
- Open access
- Published: 16 October 2026
- Maria Mahbub ORCID: orcid.org/0000-0002-3422-96501,
- Gregory M. Dams ORCID: orcid.org/0000-0002-8546-014X2,
- Josh Arnold1,
- Caitlin Rizy1,
- Sudarshan Srinivasan ORCID: orcid.org/0000-0003-1222-75121,
- Elliot M. Fielstein2,3,
- Minu A. Aghevli4,
- Kamonica L. Craig5,
- Elizabeth M. Oliva2,
- Joseph Erdos6,7,
- Jodie Trafton2 &
- …
- Ioana Danciu ORCID: orcid.org/0000-0002-0164-14031
npj Digital Medicine (2026) Cite this article
We’re sharing this article early to provide faster access to peer-reviewed, accepted research. It is citable and carries a permanent DOI. This version is subject to further edits and will be replaced automatically by the final Version of Record. All legal disclaimers apply.
Abstract
Large language models (LLMs) show promise for extracting clinically meaningful information from unstructured health records, yet real-world deployment is constrained by limited scalable, trustworthy validation approaches. Conventional evaluation relies on annotation-intensive reference standards and incomplete structured data, limiting population-scale use. We propose a multi-stage validation framework for LLM-based clinical information extraction that enables rigorous assessment under weak supervision by integrating complementary validation strategies. The framework combines prompt calibration, rule-based filtering, semantic grounding, judge-LLM evaluation, expert review, and predictive validity analysis to quantify uncertainty without exhaustive manual annotation. We applied this framework to extract substance use disorder (SUD) diagnoses across 11 substance categories from 919,783 clinical notes. Rule-based filtering and semantic grounding removed 14.59% of LLM-positive extractions that were unsupported or structurally implausible. For high-uncertainty cases, the judge LLM showed substantial agreement with subject matter expert. Using judge evaluations as references, the primary LLM achieved an F1 score of 0.80. LLM-extracted SUD diagnoses predicted subsequent engagement in SUD specialty care more accurately than structured-data baselines. By unifying complementary validation strategies, this work presents a generalizable human-in-the-loop framework for establishing the trustworthiness of LLM-based clinical information extraction at scale while minimizing the need for exhaustive manual annotation.
Acknowledgements
This manuscript has been authored by UT-Battelle, LLC, under contract DE-AC05-00OR22725 with the US Department of Energy (DOE). The US government retains and the publisher, by accepting the article for publication, acknowledges that the US government retains a nonexclusive, paid-up, irrevocable, worldwide license to publish or reproduce the published form of this manuscript, or allow others to do so, for US government purposes. DOE will provide public access to these results of federally sponsored research in accordance with the DOE Public Access Plan (https://www.energy.gov/doe-public-access-plan). The initiative aims to improve care for Veterans within the Veterans Health Administration (VHA) and involves models specifically tailored to the VHA system. The results are not generalizable outside of VHA, and the studies are formally considered non-research by the VHA. Oak Ridge National Laboratory received Institutional Review Board (IRB) approval for the secondary use of patient data, which complies with internal policies and standards. The authors acknowledge the broader partnership and express gratitude to the Veterans receiving care at the U.S. Department of Veterans Affairs (VA).
Funding
This initiative is sponsored by the U.S. Department of Veterans Affairs (VA) and utilizes VA-funded resources from the Knowledge Discovery Infrastructure (KDI) at Oak Ridge National Laboratory under the Department of Energy (DOE) Office of Science. This work is a collaboration between VA and DOE. VA has provided data and biomedical expertise to guide the study.
Author information
Authors and Affiliations
Oak Ridge National Laboratory, Oak Ridge, TN, USA
Maria Mahbub, Josh Arnold, Caitlin Rizy, Sudarshan Srinivasan & Ioana Danciu
Program Evaluation and Resource Center, Office of Mental Health and Office of Suicide Prevention, Department of Veterans Affairs, Menlo Park, CA, USA
Gregory M. Dams, Elliot M. Fielstein, Elizabeth M. Oliva & Jodie Trafton
Vanderbilt University Medical Center, Nashville, TN, USA
Elliot M. Fielstein
VA Maryland Health Care System, Baltimore, MD, USA
Minu A. Aghevli
VA Desert Pacific Healthcare Network, Long Beach, CA, USA
Kamonica L. Craig
VA Connecticut Health Care System, West Haven, CT, USA
Joseph Erdos
Yale School of Medicine, New Haven, CT, USA
Joseph Erdos
Authors
- Maria Mahbub
- Gregory M. Dams
- Josh Arnold
- Caitlin Rizy
- Sudarshan Srinivasan
- Elliot M. Fielstein
- Minu A. Aghevli
- Kamonica L. Craig
- Elizabeth M. Oliva
- Joseph Erdos
- Jodie Trafton
- Ioana Danciu
Corresponding author
Correspondence to Maria Mahbub.
Ethics declarations
Competing interests
The authors declare no competing interests.
Additional information
Publisher’s note Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
Rights and permissions
Open Access This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made. The images or other third party material in this article are included in the article’s Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article’s Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit http://creativecommons.org/licenses/by/4.0/.
About this article
Cite this article
Mahbub, M., Dams, G.M., Arnold, J. et al. A multi-stage validation framework for trustworthy large-scale clinical information extraction using large language models. npj Digit. Med. (2026). https://doi.org/10.1038/s41746-026-03341-7
Received: 27 March 2026
Accepted: 19 September 2026
Published: 16 October 2026
DOI: https://doi.org/10.1038/s41746-026-03341-7
来源:Nature 健康护理主题 · nature.com