跳到正文
原文
Nature 健康护理主题·· 2026-10-06AI 评分52

SPIRIT-CONSORT-ELM:基于要素级标注与大型语言模型的随机对照试验报告质量评估

SPIRIT-CONSORT-ELM: element-level annotated dataset and large language model approach for assessing randomized controlled trial reporting

AI 导读

研究人员扩展了临床试验数据集SPIRIT-CONSORT-ELM,为150篇文章添加了要素级标注,并将其转化为包含119个问题的机器阅读理解任务。开发了一种结合PubMedBERT证据检索与GPT-5问答的自动化评估管道,该管道在150篇文章测试集上取得了0.822的F1分数和0.796的Gwet’s AC1一致性,为评估随机对照试验报告质量建立了细粒度的基准。

正文

Abstract

Randomized controlled trials (RCTs) are central to assessing the benefits and harms of interventions, but incomplete reporting undermines their verifiability and usefulness. Although SPIRIT and CONSORT reporting guidelines promote complete reporting of RCT protocols and results publications, many RCTs remain incompletely reported. Automated manuscript checking could help improve reporting completeness before publication. We previously developed SPIRIT-CONSORT-TM, a corpus of 200 articles (100 protocol-results publication pairs) annotated with 83 checklist items from SPIRIT 2013 and CONSORT 2010, and trained models for item-level assessment. However, checklist items may comprise multiple constituent elements, which prior work did not capture or evaluate. Here, we extend the corpus with element-level annotations (SPIRIT-CONSORT-ELM) and formulate assessment as a machine reading comprehension task operationalized through 119 questions targeting specific reporting elements. Two annotators independently assessed 50 articles (25 pairs), with discrepancies resolved through discussion; one annotator assessed the remaining 150 articles. We then developed an automated pipeline combining PubMedBERT-based evidence retrieval with GPT-5-based question answering. Inter-annotator agreement was high (Gwet’s AC1: 0.782), and the pipeline achieved high performance (F1: 0.822, Gwet’s AC1: 0.796). Component analyses demonstrated the importance of evidence retrieval quality and modest benefits from illustrative in-context examples. SPIRIT-CONSORT-ELM provides a benchmark for fine-grained assessment of RCT reporting completeness, while the automated pipeline establishes a robust baseline and shows potential for supporting authors, reviewers, and editors.

Similar content being viewed by others

Acknowledgements

This work was supported by the National Library of Medicine of the National Institutes of Health under the award number R01LM014079. The content is solely the responsibility of the authors and does not necessarily represent the official views of the National Institutes of Health or any other organization. The funder had no role in considering the study design or in the collection, analysis, interpretation of data, writing of the report, or decision to submit the article for publication. AWB is supported in part by the Arkansas Children’s Research Institute and the Arkansas Biosciences Institute. This work used Bridges-2 and Ocean at Pittsburgh Supercomputing Center (PSC) through allocation #CIS230380 from the Advanced Cyberinfrastructure Coordination Ecosystem: Services & Support (ACCESS) program36, which is supported by National Science Foundation, United States grants #2138259, #2138286, #2138307, #2137603, and #2138296.

Funding

This work was supported by the National Library of Medicine of the National Institutes of Health under the award number R01LM014079. The content is solely the responsibility of the authors and does not necessarily represent the official views of the National Institutes of Health or any other organization. The funder had no role in considering the study design or in the collection, analysis, interpretation of data, writing of the report, or decision to submit the article for publication. AWB is supported in part by the Arkansas Children’s Research Institute and the Arkansas Biosciences Institute.

Author information

Author notes

  1. These authors contributed equally: Lan Jiang, Xiangji Ying.

Authors and Affiliations

  1. School of Information Sciences, University of Illinois Urbana-Champaign, Champaign, IL, USA

    Lan Jiang, Mengfei Lan, Wenxuan Song, Joe D. Menke & Halil Kilicoglu

  2. Department of Epidemiology, University of North Carolina Gillings School of Global Public Health, Chapel Hill, NC, USA

    Xiangji Ying & Evan Mayo-Wilson

  3. Department of Biostatistics, University of Arkansas for Medical Sciences, Little Rock, AR, USA

    Andrew W. Brown

  4. Arkansas Children’s Research Institute, Little Rock, AR, USA

    Andrew W. Brown

  5. Department of Epidemiology and Biostatistics, Indiana University School of Public Health-Bloomington, Bloomington, IN, USA

    Colby J. Vorland

Authors

  1. Lan Jiang
  2. Xiangji Ying
  3. Andrew W. Brown
  4. Mengfei Lan
  5. Wenxuan Song
  6. Joe D. Menke
  7. Colby J. Vorland
  8. Evan Mayo-Wilson
  9. Halil Kilicoglu

Corresponding author

Correspondence to Halil Kilicoglu.

Ethics declarations

Competing interests

The authors declare no competing interests.

Additional information

Publisher’s note Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Open Access This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made. The images or other third party material in this article are included in the article’s Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article’s Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit http://creativecommons.org/licenses/by/4.0/.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Jiang, L., Ying, X., Brown, A.W. et al. SPIRIT-CONSORT-ELM: element-level annotated dataset and large language model approach for assessing randomized controlled trial reporting. npj Digit. Med. (2026). https://doi.org/10.1038/s41746-026-03318-6

Download citation

  • Received: 26 May 2026

  • Accepted: 16 September 2026

  • Published: 06 October 2026

  • DOI: https://doi.org/10.1038/s41746-026-03318-6

来源:Nature 健康护理主题 · nature.com