HC-MFT: 一种用于胸腔疾病诊断的层次跨模态融合变换器
HC-MFT: a hierarchical cross-modal fusion transformer for robust and explainable thoracic disease diagnosis
HC-MFT 是一种层次跨模态融合变换器,结合了视图变换器和 EfficientNet-B3-变换器,用于胸部 X 光和 CT 图像的诊断。该模型在 COVID-19、肺不张、病毒性肺炎和正常影像的 600 个测试配对中达到了 96.3% 的准确率和 0.989 的宏 AUC,展示了良好的诊断性能。
- Shah Imran1,
- Khalil Ullah2,
- Shah Khalid1,
- Ibrar Hussain3,
- Evgeny Karpulevich4,
- Yury Markin4 &
- …
- Sergei Romanov5,6
We’re sharing this article early to provide faster access to peer-reviewed, accepted research. It is citable and carries a permanent DOI. This version is subject to further edits and will be replaced automatically by the final Version of Record. All legal disclaimers apply.
Abstract
Chest radiography (CXR) and computed tomography (CT) provide complementary views of thoracic disease, yet most automated diagnostic models use a single modality. We present HC-MFT, a hierarchical cross-modal fusion transformer that encodes radiographs with a Vision Transformer and CT slices with an EfficientNet-B3–Transformer hybrid, exchanges information between the two token sequences through stacked bidirectional cross-attention, and combines them with a sample-adaptive, dimension-wise gate. Because no public dataset offers patient-matched CXR–CT examinations for the four classes studied, we evaluate the method on class-matched pseudo-pairs built from the COVID-19 Radiography Database and the SARS-CoV-2 CT-Scan Dataset; both image pools are split into training, validation and test partitions before pairing and audited for near-duplicates, and the CT branch is supervised only with the binary label it carries. With an identical training protocol for all models, HC-MFT reached 96.3% accuracy (95% CI [94.7, 97.7]), a macro AUC of 0.989 and a macro F1 of 0.963 on 600 test pairs covering COVID-19, lung opacity, viral pneumonia and normal radiographs, and a macro F1 of 0.957 ± 0.004 across 5 random splits. HC-MFT was the most accurate and most stable model, but its advantage over the strongest fusion baselines was not statistically significant. Pairing controls showed that the trained model relies almost entirely on the radiograph, and a quantitative saliency analysis revealed substantial attention outside the lung fields, which suggests source-related shortcuts. HC-MFT is therefore a competitive computational fusion framework; clinical use requires validation on genuinely paired, multi-centre cohorts
Acknowledgements
This work was supported by a grant, provided by the Ministry of Economic Development of the Russian Federation (agreement dated June 20, 2025 No. 139-15-2025-011, identifier 000000C313925P4G0002).
Author information
Authors and Affiliations
Department of Computer Science & IT, University of Malakand, Chakdara, Dir Lower, 18000, Pakistan
Shah Imran & Shah Khalid
Department of Software Engineering, University of Malakand, Chakdara, Dir Lower, 18000, Pakistan
Khalil Ullah
Quality Enhancement Cell, Shaheed Benazir Bhutto University, Sheringal, Dir Upper, 18050, Pakistan
Ibrar Hussain
Trusted AI Research Center, RAS, Moscow, Russia, 109004
Evgeny Karpulevich & Yury Markin
Intelligent Devices Institute, Saint Petersburg Electrotechnical University “LETI”, St. Petersburg, Russia, 197022
Sergei Romanov
Almazov National Medical Research Centre, St. Petersburg, Russia, 197341
Sergei Romanov
Authors
- Shah Imran
- Khalil Ullah
- Shah Khalid
- Ibrar Hussain
- Evgeny Karpulevich
- Yury Markin
- Sergei Romanov
Corresponding author
Correspondence to Sergei Romanov.
Ethics declarations
Competing interests
The authors declare no competing interests.
Additional information
Publisher's note
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
Rights and permissions
Open Access This article is licensed under a Creative Commons Attribution-NonCommercial-NoDerivatives 4.0 International License, which permits any non-commercial use, sharing, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if you modified the licensed material. You do not have permission under this licence to share adapted material derived from this article or parts of it. The images or other third party material in this article are included in the article’s Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article’s Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit http://creativecommons.org/licenses/by-nc-nd/4.0/.
About this article
Cite this article
Imran, S., Ullah, K., Khalid, S. et al. HC-MFT: a hierarchical cross-modal fusion transformer for robust and explainable thoracic disease diagnosis. Sci Rep (2026). https://doi.org/10.1038/s41598-026-81110-y
Received: 16 August 2026
Accepted: 12 November 2026
Published: 20 November 2026
DOI: https://doi.org/10.1038/s41598-026-81110-y
来源:Nature 健康护理主题 · nature.com