跳到正文
原文
Nature 健康护理主题·· 2026-10-10AI 评分39

FHIR-AgentEval:用于评估临床大语言模型智能体及内存增强配置的模块化基准测试沙箱

FHIR-AgentEval: a modular sandbox for benchmarking clinical LLM agents with memory-augmented configurations

AI 导读

Boston Children’s Hospital 团队在 npj Digital Medicine 发布 FHIR-AgentEval,该模块化沙箱包含 43 个任务,用于在 HL7 FHIR 环境下评估临床大语言模型智能体。消融研究显示,引入长期内存可稳定提升任务成功率并减少策略性失败。在保留任务测试中,最佳内存配置较基准成功率提高 9.1%。

正文

npj Digital Medicine (2026) Cite this article

We’re sharing this article early to provide faster access to peer-reviewed, accepted research. It is citable and carries a permanent DOI. This version is subject to further edits and will be replaced automatically by the final Version of Record. All legal disclaimers apply.

Abstract

Healthcare data exchange increasingly relies on HL7 FHIR, but FHIR’s implementation complexity creates barriers for clinical workflows. Large language model (LLM) agents could bridge this gap by translating natural language requests into structured FHIR operations, yet their reliability remains unproven. We present FHIR-AgentEval, an extensible evaluation sandbox comprising 43 modular tasks for benchmarking LLM agents on realistic appointment management and genetic testing workflows. Each task executes against a resettable FHIR server with custom deterministic validation of both agent responses and resulting server state. We run an ablation study of five agent configurations, varying access to an on-demand FHIR R4 specifications server and long-term memory trained with or without specification grounding. Across four experimental settings, memory consistently improves task success and reduces strategic failures such as incorrect tool selection and resource-type confusion. On held-out tasks, the best memory configuration improves success by 9.1% over baseline, offering a potential pathway toward more robust clinical deployment.

Acknowledgements

This study was supported by National Human Genome Research Institute grant R01HG012655.

Author information

Authors and Affiliations

  1. Division of Genetics and Genomics, Department of Pediatrics, Boston Children’s Hospital, Harvard Medical School, Boston, MA, USA

    Youssef Mokssit, Kamalakkannan Ravi, Mengshu Nie, Junyoung Kim & Cong Liu

  2. Computational Health Informatics Program, Boston Children’s Hospital, Boston, MA, USA

    Cong Liu

Authors

  1. Youssef Mokssit
  2. Kamalakkannan Ravi
  3. Mengshu Nie
  4. Junyoung Kim
  5. Cong Liu

Corresponding author

Correspondence to Cong Liu.

Ethics declarations

Competing interests

The authors declare no competing interests.

Additional information

Publisher’s note Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Open Access This article is licensed under a Creative Commons Attribution-NonCommercial-NoDerivatives 4.0 International License, which permits any non-commercial use, sharing, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if you modified the licensed material. You do not have permission under this licence to share adapted material derived from this article or parts of it. The images or other third party material in this article are included in the article’s Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article’s Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit http://creativecommons.org/licenses/by-nc-nd/4.0/.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Mokssit, Y., Ravi, K., Nie, M. et al. FHIR-AgentEval: a modular sandbox for benchmarking clinical LLM agents with memory-augmented configurations. npj Digit. Med. (2026). https://doi.org/10.1038/s41746-026-03348-0

Download citation

  • Received: 31 January 2026

  • Accepted: 21 September 2026

  • Published: 10 October 2026

  • DOI: https://doi.org/10.1038/s41746-026-03348-0

来源:Nature 健康护理主题 · nature.com