FHIR-AgentEval:用于评估临床大语言模型智能体及内存增强配置的模块化基准测试沙箱
FHIR-AgentEval: a modular sandbox for benchmarking clinical LLM agents with memory-augmented configurations
Boston Children’s Hospital 团队在 npj Digital Medicine 发布 FHIR-AgentEval,该模块化沙箱包含 43 个任务,用于在 HL7 FHIR 环境下评估临床大语言模型智能体。消融研究显示,引入长期内存可稳定提升任务成功率并减少策略性失败。在保留任务测试中,最佳内存配置较基准成功率提高 9.1%。
- Article
- Open access
- Published: 10 October 2026
npj Digital Medicine (2026) Cite this article
We’re sharing this article early to provide faster access to peer-reviewed, accepted research. It is citable and carries a permanent DOI. This version is subject to further edits and will be replaced automatically by the final Version of Record. All legal disclaimers apply.
Abstract
Healthcare data exchange increasingly relies on HL7 FHIR, but FHIR’s implementation complexity creates barriers for clinical workflows. Large language model (LLM) agents could bridge this gap by translating natural language requests into structured FHIR operations, yet their reliability remains unproven. We present FHIR-AgentEval, an extensible evaluation sandbox comprising 43 modular tasks for benchmarking LLM agents on realistic appointment management and genetic testing workflows. Each task executes against a resettable FHIR server with custom deterministic validation of both agent responses and resulting server state. We run an ablation study of five agent configurations, varying access to an on-demand FHIR R4 specifications server and long-term memory trained with or without specification grounding. Across four experimental settings, memory consistently improves task success and reduces strategic failures such as incorrect tool selection and resource-type confusion. On held-out tasks, the best memory configuration improves success by 9.1% over baseline, offering a potential pathway toward more robust clinical deployment.
Acknowledgements
This study was supported by National Human Genome Research Institute grant R01HG012655.
Author information
Authors and Affiliations
Division of Genetics and Genomics, Department of Pediatrics, Boston Children’s Hospital, Harvard Medical School, Boston, MA, USA
Youssef Mokssit, Kamalakkannan Ravi, Mengshu Nie, Junyoung Kim & Cong Liu
Computational Health Informatics Program, Boston Children’s Hospital, Boston, MA, USA
Cong Liu
Authors
- Youssef Mokssit
- Kamalakkannan Ravi
- Mengshu Nie
- Junyoung Kim
- Cong Liu
Corresponding author
Correspondence to Cong Liu.
Ethics declarations
Competing interests
The authors declare no competing interests.
Additional information
Publisher’s note Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
Rights and permissions
Open Access This article is licensed under a Creative Commons Attribution-NonCommercial-NoDerivatives 4.0 International License, which permits any non-commercial use, sharing, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if you modified the licensed material. You do not have permission under this licence to share adapted material derived from this article or parts of it. The images or other third party material in this article are included in the article’s Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article’s Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit http://creativecommons.org/licenses/by-nc-nd/4.0/.
About this article
Cite this article
Mokssit, Y., Ravi, K., Nie, M. et al. FHIR-AgentEval: a modular sandbox for benchmarking clinical LLM agents with memory-augmented configurations. npj Digit. Med. (2026). https://doi.org/10.1038/s41746-026-03348-0
Received: 31 January 2026
Accepted: 21 September 2026
Published: 10 October 2026
DOI: https://doi.org/10.1038/s41746-026-03348-0
来源:Nature 健康护理主题 · nature.com