Context-Aware Cross-Modal Attention 在自然环境中的多模态药片图像识别
Multimodal pill image recognition in natural environments using context-aware cross-modal attention
该研究提出了一种名为 CACMA 的识别框架,用于自然环境下的药片图像识别。在 CURE 数据集上,CACMA 达到了 91.13% 的 Top-1 准确率和 95.50% 的 mAP,显著优于其他方法。该框架在准确性、效率和可解释性方面表现出色。
- Article
- Open access
- Published: 11 November 2026
Scientific Reports (2026) Cite this article
We’re sharing this article early to provide faster access to peer-reviewed, accepted research. It is citable and carries a permanent DOI. This version is subject to further edits and will be replaced automatically by the final Version of Record. All legal disclaimers apply.
Abstract
Medication errors remain a major source of preventable harm, and identifying a pill from a photograph taken in an uncontrolled setting is still difficult because the informative cues-color, shape, surface texture, and imprinted text-are unequally reliable from one image to the next. We present a recognition framework for natural environments in which a fusion module, context-aware cross-modal attention (CACMA), weighs these four modalities separately for every image using a query formed from their joint global context, an explicit confidence branch that estimates per-modality reliability, and a three-way integration of attended, confidence-weighted, and residual features, on top of \(\hbox {U}^2\)-Net segmentation (98.06% validation IoU) and connected-component localization. All results follow a unified end-to-end protocol, with segmentation, localization, OCR, and fusion executed at inference time, over three random seeds on two real-world datasets. On CURE (196 classes) CACMA attains 91.13±0.27% Top-1 accuracy and \(95.50\pm 0.16\%\) mAP, exceeding the published results of Ling et al. and Rádli et al. by 22.83 and 5.41 pp and summation fusion by 3.15 pp (\(p<0.001\)); the 0.76 pp margin over weighted summation is not statistically significant, which we report as parity rather than superiority, while CACMA attains the smallest across-seed variability of any method evaluated (\(\pm 0.27\) pp). On OGYEI-v2 (112 classes) it reaches \(83.23\pm 0.73\%\) Top-1 accuracy. What fixed-weight fusion cannot reproduce is directly measurable: the confidence assigned to the text modality rises from 0.362 to 0.602 when an imprint is legible (\(p = 1 \times 10^{-55}\)), with the released weight redistributed to the visual modalities. A fully fine-tuned ViT-B/16 is not significantly more accurate (\(92.07\pm 0.64\%\); McNemar \(p=0.32\)) while using 6.4\(\times \) the parameters and offering no modality-level explanation, placing CACMA at a favorable accuracy-efficiency-interpretability operating point. Paired significance testing applied to the ablations shows that no individual component accounts for the result on its own under this segmentation quality; together with calibration, saliency and failure analyses, these tests delimit where multimodal attention fusion helps and where it does not.
Acknowledgements
The authors would like to express their gratitude to Nguyen Tat Thanh University, Ho Chi Minh City, Vietnam; the Faculty of Information Technology, Van Lang School of Technology, Van Lang University, Vietnam; and the Faculty of Information Technology, Ho Chi Minh City Open University, Vietnam, for their support in terms of time and facilities for this study. This research was supported by Dr. Hoang Le Minh, Director of the Center for Applications of Technology and Artificial Intelligence, who provided the necessary computational resources.
Funding
This research received no specific grant or other financial support from any funding agency in the public, commercial, or not-for-profit sectors.
Author information
Authors and Affiliations
Faculty of Information Technology, Nguyen Tat Thanh University, 300A Nguyen Tat Thanh Street, Xom Chieu Ward, Ho Chi Minh City, 70000, Vietnam
Linh Nguyen Thi My & Tham Vo
Faculty of Information Technology, Van Lang School of Technology, Van Lang University, 69/68 Dang Thuy Tram Street, Binh Loi Trung Ward, Ho Chi Minh City, 70000, Vietnam
Linh Nguyen Thi My
Faculty of Information Technology, Ho Chi Minh City Open University, 35-37 Ho Hao Hon Street, Cau Ong Lanh Ward, Ho Chi Minh City, 70000, Vietnam
Vinh Truong Hoang
Authors
- Linh Nguyen Thi My
- Tham Vo
- Vinh Truong Hoang
Corresponding author
Correspondence to Vinh Truong Hoang.
Ethics declarations
Competing interests
The authors declare that they have no known competing financial interests or personal relationships that could have appeared to influence the work reported in this paper.
Additional information
Publisher's note
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
Rights and permissions
Open Access This article is licensed under a Creative Commons Attribution-NonCommercial-NoDerivatives 4.0 International License, which permits any non-commercial use, sharing, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if you modified the licensed material. You do not have permission under this licence to share adapted material derived from this article or parts of it. The images or other third party material in this article are included in the article’s Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article’s Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit http://creativecommons.org/licenses/by-nc-nd/4.0/.
About this article
Cite this article
Nguyen Thi My, L., Vo, T. & Truong Hoang, V. Multimodal pill image recognition in natural environments using context-aware cross-modal attention. Sci Rep (2026). https://doi.org/10.1038/s41598-026-70234-w
Received: 25 January 2026
Accepted: 01 September 2026
Published: 11 November 2026
DOI: https://doi.org/10.1038/s41598-026-70234-w
Keywords
来源:Nature 健康护理主题 · nature.com