糖尿病检测机器学习模型的多维度预测安全性综合分析
A comprehensive study on multi-dimensional predictive safety analysis of machine learning models in diabetes detection
一项在Scientific Reports发表的研究通过基准测试评估了不同机器学习模型在糖尿病风险检测中的预测安全性。研究发现,即使模型区分性能(PR-AUC)相近,其校准质量、不确定性有效性和子群体可靠性仍存在显著差异。该工作表明仅依赖区分指标不足以保障临床决策可靠性,需纳入校准、公平性及保形预测等维度进行综合评估。
- Article
- Open access
- Published: 02 October 2026
- Nasirul Mumenin ORCID: orcid.org/0000-0002-8615-83481,
- Md. Appel Mahmud Pranto2,
- Mohammad Abu Yousuf3,
- Muhammad Mostafa Monowar4,
- Md Abdul Hamid4,
- Fahd S. Alotaibi5 &
- …
- Madini O. Alassafi4
Scientific Reports (2026) Cite this article
We’re sharing this article early to provide faster access to peer-reviewed, accepted research. It is citable and carries a permanent DOI. This version is subject to further edits and will be replaced automatically by the final Version of Record. All legal disclaimers apply.
Abstract
High discrimination performance is often interpreted as evidence of model reliability in clinical machine learning (ML). However, strong predictive accuracy does not guarantee safe decision behavior. In safety-critical domains such as healthcare, miscalibrated probabilities, ineffective uncertainty estimation, and overconfident errors can lead to harmful decisions despite high area-under-curve metrics. This study presents a comprehensive multi-dimensional predictive safety analysis demonstrating that models with comparable discrimination performance can exhibit substantially different reliability and risk characteristics. We conduct a rigorous benchmarking evaluation across two heterogeneous diabetes risk datasets, a large population-scale BRFSS cohort and a small-sample early-stage clinical dataset. A diverse set of classical, ensemble, and deep tabular models are evaluated under a strict leakage-free cross-validation protocol with dedicated calibration splits. Beyond conventional discrimination metrics (PR-AUC, ROC-AUC), we assess probabilistic calibration (ECE, Brier score, NLL), epistemic uncertainty using a unified bootstrap-based framework, uncertainty–error correlation, selective prediction safety, high-confidence error rates, subgroup fairness across age and sex, and conformal prediction with finite-sample marginal coverage guarantees under the standard exchangeability assumption. Our results reveal a consistent accuracy–safety gap that models achieving near-identical PR-AUC display markedly different calibration quality, uncertainty effectiveness, subgroup reliability, and overconfidence behavior. On the early-stage clinical dataset, near-ceiling discrimination performance is observed across models, but due to the limited sample size, these results should be interpreted cautiously, as variability across folds may affect the stability of comparative performance across models. The findings demonstrate that discrimination metrics alone are insufficient to characterize reliable clinical decision-making. A comprehensive evaluation incorporating calibration, uncertainty quality, subgroup fairness, and formal coverage guarantees is essential for the trustworthy pre-deployment assessment of machine learning in healthcare. Rather than introducing a new predictive algorithm, this work contributes a unified empirical audit of safety-relevant model behavior, addressing reliability dimensions that are commonly evaluated separately or omitted in conventional diabetes prediction benchmarks.
Acknowledgements
This Project was funded by the Deanship of Scientific Research (DSR) at King Abdulaziz University, Jeddah, under grant no.(IPP: 675-611-2025). The authors, therefore, acknowledge DSR’s technical and financial support with thanks.
Funding
Deanship of Scientific Research (DSR) at King Abdulaziz University, Jeddah, under grant no.(IPP: 675-611-2025).
Author information
Authors and Affiliations
Department of Computer Science and Engineering, Bangladesh University of Business and Technology, Dhaka, Bangladesh
Nasirul Mumenin
Department of Information and Communication Technology, Bangladesh University of Professionals, Dhaka, Bangladesh
Md. Appel Mahmud Pranto
Institute of Information Technology, Jahangirnagar University, Dhaka, Bangladesh
Mohammad Abu Yousuf
Department of Information Technology, Faculty of Computing and Information Technology, King Abdulaziz University, 21589, Jeddah, Kingdom of Saudi Arabia
Muhammad Mostafa Monowar, Md Abdul Hamid & Madini O. Alassafi
Department of Information Systems, Faculty of Computing and Information Technology, King Abdulaziz University, 21589, Jeddah, Kingdom of Saudi Arabia
Fahd S. Alotaibi
Authors
- Nasirul Mumenin
- Md. Appel Mahmud Pranto
- Mohammad Abu Yousuf
- Muhammad Mostafa Monowar
- Md Abdul Hamid
- Fahd S. Alotaibi
- Madini O. Alassafi
Corresponding author
Correspondence to Nasirul Mumenin.
Ethics declarations
Competing interests
The authors declare no competing interests.
Additional information
Publisher's note
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
Rights and permissions
Open Access This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made. The images or other third party material in this article are included in the article’s Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article’s Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit http://creativecommons.org/licenses/by/4.0/.
About this article
Cite this article
Mumenin, N., Pranto, M.A.M., Yousuf, M.A. et al. A comprehensive study on multi-dimensional predictive safety analysis of machine learning models in diabetes detection. Sci Rep (2026). https://doi.org/10.1038/s41598-026-73801-3
Received: 28 February 2026
Accepted: 25 September 2026
Published: 02 October 2026
DOI: https://doi.org/10.1038/s41598-026-73801-3
来源:Nature 健康护理主题 · nature.com