SPIRIT-CONSORT-ELM:基于要素级标注与大型语言模型的随机对照试验报告质量评估
SPIRIT-CONSORT-ELM: element-level annotated dataset and large language model approach for assessing randomized controlled trial reporting
研究人员扩展了临床试验数据集SPIRIT-CONSORT-ELM,为150篇文章添加了要素级标注,并将其转化为包含119个问题的机器阅读理解任务。开发了一种结合PubMedBERT证据检索与GPT-5问答的自动化评估管道,该管道在150篇文章测试集上取得了0.822的F1分数和0.796的Gwet’s AC1一致性,为评估随机对照试验报告质量建立了细粒度的基准。
Abstract
Randomized controlled trials (RCTs) are central to assessing the benefits and harms of interventions, but incomplete reporting undermines their verifiability and usefulness. Although SPIRIT and CONSORT reporting guidelines promote complete reporting of RCT protocols and results publications, many RCTs remain incompletely reported. Automated manuscript checking could help improve reporting completeness before publication. We previously developed SPIRIT-CONSORT-TM, a corpus of 200 articles (100 protocol-results publication pairs) annotated with 83 checklist items from SPIRIT 2013 and CONSORT 2010, and trained models for item-level assessment. However, checklist items may comprise multiple constituent elements, which prior work did not capture or evaluate. Here, we extend the corpus with element-level annotations (SPIRIT-CONSORT-ELM) and formulate assessment as a machine reading comprehension task operationalized through 119 questions targeting specific reporting elements. Two annotators independently assessed 50 articles (25 pairs), with discrepancies resolved through discussion; one annotator assessed the remaining 150 articles. We then developed an automated pipeline combining PubMedBERT-based evidence retrieval with GPT-5-based question answering. Inter-annotator agreement was high (Gwet’s AC1: 0.782), and the pipeline achieved high performance (F1: 0.822, Gwet’s AC1: 0.796). Component analyses demonstrated the importance of evidence retrieval quality and modest benefits from illustrative in-context examples. SPIRIT-CONSORT-ELM provides a benchmark for fine-grained assessment of RCT reporting completeness, while the automated pipeline establishes a robust baseline and shows potential for supporting authors, reviewers, and editors.
Similar content being viewed by others



Acknowledgements
This work was supported by the National Library of Medicine of the National Institutes of Health under the award number R01LM014079. The content is solely the responsibility of the authors and does not necessarily represent the official views of the National Institutes of Health or any other organization. The funder had no role in considering the study design or in the collection, analysis, interpretation of data, writing of the report, or decision to submit the article for publication. AWB is supported in part by the Arkansas Children’s Research Institute and the Arkansas Biosciences Institute. This work used Bridges-2 and Ocean at Pittsburgh Supercomputing Center (PSC) through allocation #CIS230380 from the Advanced Cyberinfrastructure Coordination Ecosystem: Services & Support (ACCESS) program36, which is supported by National Science Foundation, United States grants #2138259, #2138286, #2138307, #2137603, and #2138296.
Funding
This work was supported by the National Library of Medicine of the National Institutes of Health under the award number R01LM014079. The content is solely the responsibility of the authors and does not necessarily represent the official views of the National Institutes of Health or any other organization. The funder had no role in considering the study design or in the collection, analysis, interpretation of data, writing of the report, or decision to submit the article for publication. AWB is supported in part by the Arkansas Children’s Research Institute and the Arkansas Biosciences Institute.
Author information
Author notes
These authors contributed equally: Lan Jiang, Xiangji Ying.
Authors and Affiliations
School of Information Sciences, University of Illinois Urbana-Champaign, Champaign, IL, USA
Lan Jiang, Mengfei Lan, Wenxuan Song, Joe D. Menke & Halil Kilicoglu
Department of Epidemiology, University of North Carolina Gillings School of Global Public Health, Chapel Hill, NC, USA
Xiangji Ying & Evan Mayo-Wilson
Department of Biostatistics, University of Arkansas for Medical Sciences, Little Rock, AR, USA
Andrew W. Brown
Arkansas Children’s Research Institute, Little Rock, AR, USA
Andrew W. Brown
Department of Epidemiology and Biostatistics, Indiana University School of Public Health-Bloomington, Bloomington, IN, USA
Colby J. Vorland
Authors
- Lan Jiang
- Xiangji Ying
- Andrew W. Brown
- Mengfei Lan
- Wenxuan Song
- Joe D. Menke
- Colby J. Vorland
- Evan Mayo-Wilson
- Halil Kilicoglu
Corresponding author
Correspondence to Halil Kilicoglu.
Ethics declarations
Competing interests
The authors declare no competing interests.
Additional information
Publisher’s note Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
Rights and permissions
Open Access This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made. The images or other third party material in this article are included in the article’s Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article’s Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit http://creativecommons.org/licenses/by/4.0/.
About this article
Cite this article
Jiang, L., Ying, X., Brown, A.W. et al. SPIRIT-CONSORT-ELM: element-level annotated dataset and large language model approach for assessing randomized controlled trial reporting. npj Digit. Med. (2026). https://doi.org/10.1038/s41746-026-03318-6
Received: 26 May 2026
Accepted: 16 September 2026
Published: 06 October 2026
DOI: https://doi.org/10.1038/s41746-026-03318-6
来源:Nature 健康护理主题 · nature.com