Training-corpus overlap and evaluation protocol constrain measured gains in a fine-tuned clinical language model
Training-corpus overlap and evaluation protocol constrain measured gains in a fine-tuned clinical language model
一项重新评估显示,80亿参数的QLoRA微调模型在多个基准测试中,未经修改的原始模型表现甚至优于微调模型,特别是在MedQA测试中。细粒度分析表明,训练数据集与基准测试的重叠限制了模型性能提升的可衡量范围。
We’re sharing this article early to provide faster access to peer-reviewed, accepted research. It is citable and carries a permanent DOI. This version is subject to further edits and will be replaced automatically by the final Version of Record. All legal disclaimers apply.
Abstract
Fine-tuned medical language models are frequently assessed using public question-answering benchmarks, which may overlap with their training data. We re-evaluated an 8.2-billion-parameter QLoRA fine-tune of Qwen3-8B under a fixed generative chain-of-thought harness and compared eight benchmarks against its 97,041-item fine-tuning corpus using 13-gram, exact-stem and longest-common-subsequence matching. Exact-stem overlap exceeded 70% for PubMedQA, MedQA and MMLU Professional Medicine, making those scores unsuitable for capability claims, while MedMCQA, evaluated on a held-out split, was effectively unflagged. On several overlapping benchmarks the unmodified parent checkpoint matched or exceeded the fine-tune, significantly so on MedQA. On the 1,838 MedXpertQA items not flagged by exact-stem matching, the fine-tune scored 20.8% against 15.6% for its parent, a paired gain of 5.2 percentage points (McNemar exact P = 7.9 × 10⁻⁷), against an originally reported 7.1. A bundled ablation, in which reasoning traces, sampling weights, curriculum staging, token exposure and output-format supervision differed simultaneously, preserved approximately 70% of the observed improvement. The remaining 1.6-point difference was not statistically significant (P = 0.175) and cannot be attributed to any single component. The evaluation protocol and the overlap of the training corpus limit the scope of measurable outcomes in these benchmarks.
Acknowledgements
The authors thank the United Arab Emirates University for institutional support and for access to its high-performance computing infrastructure. We acknowledge the developers of the open-source software on which this work depends, in particular the vLLM inference engine and the Unsloth training framework, and the Hugging Face platform, through which the benchmark datasets and model weights used in this study are distributed.
Funding
Article processing charges were supported by United Arab Emirates University. Computational resources were provided in part by the Google Cloud Research Credits programme and in part by institutional high-performance computing infrastructure at United Arab Emirates University. No funding body had any role in the design of the study, the analysis or interpretation of data, or the decision to submit for publication.
Author information
Authors and Affiliations
Department of Internal Medicine, College of Medicine and Health Sciences, United Arab Emirates University, Al Ain, United Arab Emirates
Adnan Agha & Eram Anwar
Authors
- Adnan Agha
- Eram Anwar
Corresponding author
Correspondence to Adnan Agha.
Ethics declarations
Ethics approval and consent to participate
This study evaluates language models on public benchmarks and involves no human subjects or patient data.
Reproducibility statement
Three distinct things, stated separately because they are not equivalent. First, the accuracy results and the associated statistical comparisons can be reproduced from the released weights, the benchmark loaders, the per-item scoring outputs and the deposited evaluation code, under the software environment and request-stream protocol specified in “Evaluation harness” section. Second, the deposited overlap flags and exact-stem-unflagged item lists permit audit and reuse of every reported subset. Third, the overlap analysis itself cannot be independently regenerated by third parties, because doing so requires the training corpus against which matching was performed, and that corpus is not redistributable. We state this as an acknowledged limitation of reproducibility rather than a resolved one.
Competing interests
A.A. developed the training methodology evaluated in this manuscript. No patent or intellectual property application covers the 8-billion-parameter model reported here; a separate institutional invention disclosure relates to a different model within the same research programme. To mitigate this interest: the evaluated model weights are publicly released without access restriction and were verified byte-identical by SHA-256 across all four weight shards to the checkpoint reported here; all evaluation uses greedy decoding at temperature zero, so replication under the disclosed software environment and request-stream protocol is not subject to sampling variability; the evaluation and overlap-detection code is deposited under a DOI (10.5281/zenodo.21827589); and the training methodology is disclosed in full in “Model and training” section and the Supplementary Information. No funding was received from any organisation with a commercial interest in the models evaluated here; computational resources were provided partly under the Google Cloud Research Credits programme, whose provider had no role in the design, analysis or reporting of this study. E.A. declares no competing interests.
Generative-AI statement
Generative-AI tools were used to assist with code generation and review, analysis consistency checking, and language editing of the manuscript, including code deposited in the accompanying repository; all reported numerical results were checked by the authors against the underlying output files, and the authors take full responsibility for the accuracy, integrity and conclusions of the work.
Additional information
Publisher’s note
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
Rights and permissions
Open Access This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made. The images or other third party material in this article are included in the article’s Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article’s Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit http://creativecommons.org/licenses/by/4.0/.
About this article
Cite this article
Agha, A., Anwar, E. Training-corpus overlap and evaluation protocol constrain measured gains in a fine-tuned clinical language model. Sci Rep (2026). https://doi.org/10.1038/s41598-026-77367-y
Received: 23 May 2026
Accepted: 19 October 2026
Published: 02 November 2026
DOI: https://doi.org/10.1038/s41598-026-77367-y
Keywords
来源:Nature 健康护理主题 · nature.com