跳到正文
原文
Nature 健康护理主题·· 15 小时前AI 评分35

HC-MFT: 一种用于胸腔疾病诊断的层次跨模态融合变换器

HC-MFT: a hierarchical cross-modal fusion transformer for robust and explainable thoracic disease diagnosis

AI 导读

HC-MFT 是一种层次跨模态融合变换器,结合了视图变换器和 EfficientNet-B3-变换器,用于胸部 X 光和 CT 图像的诊断。该模型在 COVID-19、肺不张、病毒性肺炎和正常影像的 600 个测试配对中达到了 96.3% 的准确率和 0.989 的宏 AUC,展示了良好的诊断性能。

正文

We’re sharing this article early to provide faster access to peer-reviewed, accepted research. It is citable and carries a permanent DOI. This version is subject to further edits and will be replaced automatically by the final Version of Record. All legal disclaimers apply.

Abstract

Chest radiography (CXR) and computed tomography (CT) provide complementary views of thoracic disease, yet most automated diagnostic models use a single modality. We present HC-MFT, a hierarchical cross-modal fusion transformer that encodes radiographs with a Vision Transformer and CT slices with an EfficientNet-B3–Transformer hybrid, exchanges information between the two token sequences through stacked bidirectional cross-attention, and combines them with a sample-adaptive, dimension-wise gate. Because no public dataset offers patient-matched CXR–CT examinations for the four classes studied, we evaluate the method on class-matched pseudo-pairs built from the COVID-19 Radiography Database and the SARS-CoV-2 CT-Scan Dataset; both image pools are split into training, validation and test partitions before pairing and audited for near-duplicates, and the CT branch is supervised only with the binary label it carries. With an identical training protocol for all models, HC-MFT reached 96.3% accuracy (95% CI [94.7, 97.7]), a macro AUC of 0.989 and a macro F1 of 0.963 on 600 test pairs covering COVID-19, lung opacity, viral pneumonia and normal radiographs, and a macro F1 of 0.957 ± 0.004 across 5 random splits. HC-MFT was the most accurate and most stable model, but its advantage over the strongest fusion baselines was not statistically significant. Pairing controls showed that the trained model relies almost entirely on the radiograph, and a quantitative saliency analysis revealed substantial attention outside the lung fields, which suggests source-related shortcuts. HC-MFT is therefore a competitive computational fusion framework; clinical use requires validation on genuinely paired, multi-centre cohorts

Acknowledgements

This work was supported by a grant, provided by the Ministry of Economic Development of the Russian Federation (agreement dated June 20, 2025 No. 139-15-2025-011, identifier 000000C313925P4G0002).

Author information

Authors and Affiliations

  1. Department of Computer Science & IT, University of Malakand, Chakdara, Dir Lower, 18000, Pakistan

    Shah Imran & Shah Khalid

  2. Department of Software Engineering, University of Malakand, Chakdara, Dir Lower, 18000, Pakistan

    Khalil Ullah

  3. Quality Enhancement Cell, Shaheed Benazir Bhutto University, Sheringal, Dir Upper, 18050, Pakistan

    Ibrar Hussain

  4. Trusted AI Research Center, RAS, Moscow, Russia, 109004

    Evgeny Karpulevich & Yury Markin

  5. Intelligent Devices Institute, Saint Petersburg Electrotechnical University “LETI”, St. Petersburg, Russia, 197022

    Sergei Romanov

  6. Almazov National Medical Research Centre, St. Petersburg, Russia, 197341

    Sergei Romanov

Authors

  1. Shah Imran
  2. Khalil Ullah
  3. Shah Khalid
  4. Ibrar Hussain
  5. Evgeny Karpulevich
  6. Yury Markin
  7. Sergei Romanov

Corresponding author

Correspondence to Sergei Romanov.

Ethics declarations

Competing interests

The authors declare no competing interests.

Additional information

Publisher's note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Open Access This article is licensed under a Creative Commons Attribution-NonCommercial-NoDerivatives 4.0 International License, which permits any non-commercial use, sharing, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if you modified the licensed material. You do not have permission under this licence to share adapted material derived from this article or parts of it. The images or other third party material in this article are included in the article’s Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article’s Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit http://creativecommons.org/licenses/by-nc-nd/4.0/.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Imran, S., Ullah, K., Khalid, S. et al. HC-MFT: a hierarchical cross-modal fusion transformer for robust and explainable thoracic disease diagnosis. Sci Rep (2026). https://doi.org/10.1038/s41598-026-81110-y

Download citation

  • Received: 16 August 2026

  • Accepted: 12 November 2026

  • Published: 20 November 2026

  • DOI: https://doi.org/10.1038/s41598-026-81110-y

来源:Nature 健康护理主题 · nature.com