跳到正文
原文
Nature 健康护理主题·· 22 天前精选AI 评分71

EVLF-FM 医学视觉语言基础模型基于约 130 万张影像训练,评测诊断、视觉定位与问答三类任务

A clinical multimodal vision language foundation model with fine-grain explainability

AI 导读

研究团队提出多模态医学视觉语言模型 EVLF-FM,基于约 130 万张、覆盖 10 种影像模态的临床图像训练,可完成疾病诊断、医学视觉定位与视觉问答并给出文字解释。

推荐理由

论文给出该模型在诊断、视觉定位与问答三类任务上的对比结果,并说明其解释性训练与评测方法。

正文

Main

The emergence of foundation models (FMs) has marked a paradigm shift in artificial intelligence (AI). Previously, AI in medicine generally referred to supervised deep learning models, which were limited to diagnosing a narrow set of diseases from prespecified labels1. By contrast, FMs are developed through large-scale unsupervised, semi-supervised or self-supervised pretraining and fine-tuning that confers more general abilities such as simultaneous detection of multiple diseases or accurate interpretation of images without specific training2,3,4,5. This versatility is compounded by a reduced requirement for labeled data to elicit useful performance relative to conventional deep learning. In addition, conversational AI systems extend FM capabilities by simulating clinician−patient interactions such as through structured history-taking and explanation of diagnosis and management6,7,8. These emerging applications illustrate the growing potential of FMs in medicine to support complex clinical reasoning and multimodal processing in diverse contexts, marking a major step toward general medical AI (GMAI)9.

However, FMs continue to exhibit important limitations. First, unimodal FMs, although highly performant within their own domains, are computationally expensive and impractical to scale across the breadth of clinical practice2,3. Multimodal models tend to underperform on tasks that require pixel-level precision10. These tasks include segmentation, where anatomical regions (for example, organs, tumors or vasculature) are annotated to facilitate measurement and comparison, as well as medical visual grounding, which involves defining relevant parts of an image in response to a query or report11,12. The tradeoff between the multimodal versatility with pixel-level performance and portability raises concerns about whether these models can be used within high-stakes healthcare settings. A second persistent challenge concerns the trustworthiness of model outputs, encompassing factual correctness, explainability and reasoning faithfulness—that is, ensuring that generated rationales genuinely reflect diagnostic evidence rather than plausible but post hoc narratives. Although explainable AI tools such as saliency maps and SHapley Additive exPlanations (SHAP) values have been explored in conventional deep learning systems, most are not directly applicable to emerging FMs13,14. Most FMs still function as black boxes, limiting transparency and reducing clinician confidence in their output.

To address these challenges, we introduce EVLF-FM, a state-of-the-art medical vision language model (VLM) designed to integrate the breadth of generalist models with the pixel-level precision and explainability of specialist systems (Fig. 1). EVLF-FM supports three core clinical tasks—disease diagnosis, visual grounding and medical visual question answering (VQA)—while providing textual explanations for its predictions. EVLF-FM was trained on one of the most diverse and comprehensive medical vision language training corpora assembled to date, including over 1.3 million curated clinical images spanning six clinical specialties and 10 imaging modalities, such as computed tomography (CT), magnetic resonance imaging (MRI), X-ray (XR), ultrasound (US), positron emission tomography (PET), histopathology images, dermatoscopy, color fundus photography (CFP), optical coherence tomography (OCT) and endoscopic images. It incorporates supervised fine-tuning (SFT) and visual reinforcement fine-tuning (RFT) within a generalist FM framework, aligning high-level diagnostic reasoning with fine-grained spatial localization. Unlike conventional models that provide only a final prediction, EVLF-FM generates assistive explanations that highlight clinically relevant features and localize image regions associated with its outputs, linking predictions to visual evidence. These explanations are post hoc, and their faithfulness to the underlying decision process is explicitly evaluated. EVLF-FM enables both versatile performance across modalities and enhanced model interpretability, representing a step toward the development of clinically trustworthy GMAI.

Fig. 1: Overview of the EVLF-FM framework, training strategy and clinical capabilities.

EVLF-FM was developed using approximately 1.3 million multimodal medical images spanning 10 imaging modalities. The model was trained through a two-stage pipeline consisting of SFT with disease-level and pixel-level encoders, followed by RFT using GRPO with task-specific reward functions. The resulting model supports three core clinical capabilities: disease diagnosis, VQA and medical visual grounding. ~, approximately; KL, Kullback−Leibler.

Results

To evaluate the performance of EVLF-FM, we benchmarked the model against state-of-the-art generalist VLMs as well as specialist models across core medical AI tasks, including image-based diagnosis, VQA and medical visual grounding, using a wide range of imaging modalities. Next, we assessed zero-shot and few-shot capabilities on external datasets not seen during training, comparing the performance of EVLF-FM with other VLMs. In addition, we conducted human evaluation of the model’s explanatory outputs, along with subgroup performance, ablation and data efficiency analysis, to assess the fairness and robustness of model outputs.

Internal validation across core medical AI tasks

Medical image diagnostics

We benchmarked the performance of EVLF-FM in image diagnostics with four other generalist VLM models (LLaVA version 1.5 13B, Qwen2.5-VL 7B, LLava-Med V1.5 7B and InternVL 8B); six specialist FMs (Virchow, RETFound-CFP, RETFound-OCT, Rad-DINO, USFM and BiomedCLIP); and eight specialist deep learning models (VGG 16, ResNet 50, DenseNet 121, EfficientNet B0, ViT, DINO, CLIP and SAM)2,3,5,15,16,17,18,19,20,21,22,23,24,25,26,27,28. More than 1.3 million training images were assembled from datasets spanning dermatoscopy (HAM10000, BCN20000), histopathology images (MIDOG25, LC25000), XR (RSNA Pneumonia, NIH ChestX-ray14), US (BUSI, Fetal_Planes), CFP (PAPILA, OAEFD-CFP), OCT (OAEFD-OCT) and CT (RAPIER), together with the 12 datasets from MedMNIST (PathMNIST, ChestMNIST, DermaMNIST, OCTMNIST, PneumoniaMNIST, RetinaMNIST, BreastMNIST, BloodMNIST, TissueMNIST, OrganAMNIST, OrganCMNIST and OrganSMNIST)29,30,31,32,33,34,35,36,37,38,39.

EVLF-FM demonstrated state-of-the-art diagnostic performance across multiple internal benchmarks, achieving the highest overall mean accuracy (0.895) among all evaluated models (Fig. 2a,b and Extended Data Table 1). Across the internal validation datasets, EVLF-FM consistently outperformed all generalist VLMs in both accuracy and F1 score. Compared to specialist models, it achieved the highest accuracy among specialist deep learning models on six of 11 datasets and among specialist FMs on four of 11 datasets. Specifically, EVLF-FM attained the highest accuracy among all models in key datasets, such as BCN20000 (0.949), BUSI (0.932), OAEFD-OCT (0.990) and MedMNIST (0.851), while also achieving the best F1 scores across multiple datasets, including BUSI (0.926), OAEFD-OCT (0.977) and MedMNIST (0.808). Although certain specialist models demonstrated competitive performance in isolated tasks—for example, ResNet50 in RSNA Pneumonia (accuracy 0.841) and LC25000 (accuracy 1.000)—these models exhibited reduced generalizability across datasets. By contrast, EVLF-FM maintained consistently strong performance across all tasks, highlighting its robustness as a generalist medical VLM.

Fig. 2: Internal validation results of EVLF-FM compared against generalist VLMs, specialist deep learning and specialist FMs.

a, Accuracy of EVLF-FM and comparator models across 11 diagnostic tasks, with separate radar plots comparing EVLF-FM against generalist VLMs, specialist deep learning models and specialist FMs. b, F1 score of EVLF-FM and comparator models across the same 11 diagnostic tasks, shown using the same three comparator groups. c, Closed-ended VQA accuracy of EVLF-FM and generalist VLM comparators across three benchmark datasets. d, Open-ended VQA recall of EVLF-FM and generalist VLM comparators across three benchmark datasets. e, Visual grounding performance of EVLF-FM compared to a generalist VLM across nine imaging modalities, evaluated using accuracy at IoU thresholds of 10%, 30% and 50% (Acc@0.1, Acc@0.3 and Acc@0.5), together with mIoU. DL, deep learning.

Source data

Within the MedMNIST benchmark (Extended Data Table 2), EVLF-FM similarly achieved the highest mean accuracy (0.851) and F1 score (0.808), outperforming all generalist and specialist models overall. In terms of accuracy, EVLF-FM exceeded generalist VLMs in 11 of 12 tasks and achieved the highest accuracy among specialist models in five of 12 tasks, including ties. Similarly, EVLF-FM achieved higher F1 scores than all generalist VLMs across all 12 tasks and specialist models in five of 12 tasks. Specialist models generally retained advantages on tasks closely aligned with their training domains, particularly Virchow on TissueMNIST, Rad-DINO on PneumoniaMNIST and RETFound CFP on RetinaMNIST. The comparatively lower performance observed in the ChestMNIST dataset may be attributable to its multi-label structure and substantial class imbalance. To contextualize chest XR performance across datasets and comparator models, we conducted a per-class evaluation on two internal (NIH ChestX-ray14 and VinDr-CXR) and one external (AIMx) chest XR cohorts, benchmarking EVLF-FM against InternVL3-8B and MedGemma 27B on four representative classes (consolidation/pneumonia, pulmonary mass, pneumothorax and no-finding; Extended Data Table 3). EVLF-FM consistently achieved the highest mean F1 score and competitive mean accuracy across all three cohorts (NIH ChestX-ray14: 0.751/0.545; VinDr-CXR: 0.928/0.629; AIMx: 0.840/0.668) and obtained the best per-class performance on the majority of settings. Although other models achieved higher accuracy for certain pathologies, such as InternVL for consolidation/pneumonia on NIH ChestX-ray14 (0.690 versus 0.634 for EVLF-FM), EVLF-FM achieved a substantially higher corresponding F1 score (0.571 versus 0.340). This discrepancy between accuracy and F1 score suggests that the apparently higher accuracy of comparator models may partly be due to class imbalance, whereas EVLF-FM demonstrated more balanced classification performance.

VQA

VQA tasks challenge models to generate accurate answers by integrating visual and textual information. We benchmarked the performance of EVLF-FM in VQA against five specialized models: RadFM, LLaVA-Med, Med-Flamingo, MedDr and InternVL 8B17,40,41,42,43. Three datasets were used for internal validation: VQA-RAD, SLAKE and Path-VQA44,45,46. Model performance was evaluated in terms of closed accuracy with multiple-choice questions as well as open recall with open-ended questions.

EVLF-FM demonstrated exceptional closed accuracy (Fig. 2c) and open recall (Fig. 2d). Closed accuracy was consistently high across all three datasets, with performance around 90% (Extended Data Table 4). Open recall, which assesses the ability of the model to generate contextually appropriate free-text responses, is inherently more challenging than closed-ended VQA. Nevertheless, EVLF-FM achieved the highest open recall of all tested models on both VQA-RAD and SLAKE, exceeding 80%, highlighting its strong reasoning capabilities. Moreover, EVLF-FM maintained consistently high performance across datasets and task formats. For example, although InternVL 8B achieved high open recall (77.5%) on the SLAKE dataset, its open recall dropped substantially to 52.9% on the VQA-RAD dataset, suggesting limited consistency across different formats and topics. By contrast, EVLF-FM consistently performed well on both closed-ended and open-ended VQA tasks.

Medical visual grounding

We assessed EVLF-FM’s medical visual grounding capabilities on BiomedParse and VinDr-CXR datasets across nine different imaging modalities (MRI, pathology slides, XR, CT, OCT, CFP, US, dermatoscopy and endoscopy) using conventional localization metrics—Acc@0.1, Acc@0.3 and Acc@0.5—and mean intersection over union (mIoU)47,48. In these evaluations, EVLF-FM was compared against a strong multimodal comparator (InternVL 8B)43. EVLF-FM consistently outperformed InternVL 8B across all metrics (Fig. 2e). Averaged over all modalities, EVLF-FM achieved Acc@0.1 = 0.946, Acc@0.3 = 0.902, Acc@0.5 = 0.842 and mIoU = 0.741, surpassing the baseline models’ corresponding scores of 0.865, 0.82, 0.755 and 0.656.

At the modality level, EVLF-FM consistently outperformed InternVL 3 8B across all nine imaging modalities and all four localization metrics (Extended Data Table 5). The largest gains in Acc@0.5 were observed for endoscopic images (0.742 versus 0.625), CFP (0.968 versus 0.86) and dermatoscopy (1 versus 0.905). Similarly, EVLF-FM demonstrated the largest improvements in localization overlap for CFP (mIoU: 0.781 versus 0.682) and endoscopic images (0.660 versus 0.564), indicating more accurate localization of clinically relevant regions. These consistent improvements across diverse imaging modalities highlight the strong generalizability of EVLF-FM for medical visual grounding.

External testing of zero-shot and few-shot performance

Comparison with VLMs for diagnostics

We compared the performance of EVLF-FM in image diagnostics against five other VLMs: Internvl3 8B, HuatuoGPT-Vision 7B, MedDr 32B MedGemma 4B and MedGemma 27B42,43,49,50. Twelve external datasets were used, spanning six different modalities (HRCTCov19 and SARS-CoV-2 for CT; CXR-Pneuomnia for XR; AMI-Br for pathology slices; BUS-UCLM and US3M for US; RetinalOCT and DRCR-OCT for OCT; and APTOS, Glaucoma fundus, SEED and DRCR-CFP for CFP)51,52,53,54,55,56,57,58,59,60. For EVLF-FM, we evaluated both its zero-shot performance and its few-shot performance, specifically eight-shot, 16-shot, 32-shot and 64-shot.

In the zero-shot setting, EVLF-FM achieved the highest F1 score and accuracy on eight benchmark datasets (Fig. 3a and Extended Data Table 6). Specifically, EVLF-FM attained the top performance on HRCT-Cov19 (0.972/0.969), CXR-Pneumonia (0.894/0.892), US3M (0.613/0.591) RetinalOCT (0.503/0.379), DRCR-OCT (0.861/0.861), DRCR-CFP (0.952/0.770), SEED (0.912/0.823) and Glaucoma fundus (0.746/0.743). By contrast, larger models demonstrated isolated strengths: MedDr-32B achieved superior accuracy on SARS-CoV-2 (0.781), and MedGemma-27B outperformed EVLF-FM on BUS-UCLM (0.679/0.519) in terms of both accuracy and F1 and APTOS in F1 (0.440). However, these gains were dataset specific and not consistently maintained across modalities. Notably, despite having a substantially smaller parameter size (8B), EVLF-FM remained competitive with or superior to larger models such as MedGemma 27B (27B) and MedDr 32B (32B), highlighting its efficiency and generalizability.

Fig. 3: External validation performance of EVLF-FM in zero-shot and few-shot settings compared to VLM comparators.

a, Zero-shot diagnostic performance, measured by accuracy, of EVLF-FM and comparator VLMs across 12 external datasets spanning diverse clinical imaging modalities. b, Zero-shot and few-shot diagnostic performance of EVLF-FM across the same 12 diagnostic tasks, demonstrating improved accuracy across most tasks as the number of in-context examples increases. c, Zero-shot and few-shot visual grounding performance of EVLF-FM and comparator models across eight imaging modalities, evaluated using accuracy at an IoU threshold of 50% (Acc@0.5).

Source data

As anticipated, performance across diagnostic tasks improved with the incorporation of few-shot examples (Fig. 3b and Extended Data Table 6). The largest improvement in accuracy was observed in RetinalOCT, from 0.503 in the zero-shot setting to 0.797 in the 64-shot setting, whereas the largest improvement in F1 score was seen in BUS-UCLM, from 0.200 to 0.780. Improvement was observed across most datasets, including HRCT-Cov19 (0.972/0.969 to 0.995/0.994), CXR-Pneumonia (0.894/0.892 to 0.966/0.964) and Glaucoma fundus (0.746/0.743 to 0.776/0.776). Interestingly, increased shot count did not uniformly translate to better performance. On the DRCR-CFP dataset, the zero-shot model achieved the highest accuracy (0.952), and the highest F1 score was achieved by the eight-shot model (0.771). These results suggest that, although few-shot tuning enhances diagnostic accuracy, optimal performance is not strictly dependent on the number of examples provided.

Comparison with VLMs for grounding

We benchmarked the performance of EVLF-FM in visual grounding against BiRD and MedGround on the Med-GRIT-270k dataset, which spans eight imaging modalities, including CT, MRI, XR, PET, endoscopy, dermatoscopy, CFP and US61,62. This dataset also serves as the training set for BiRD, providing a strong fully supervised baseline for comparison.

EVLF-FM achieved strong performance on the Med-GRIT-270k benchmark (Fig. 3c and Extended Data Table 6). In the zero-shot setting without Med-GRIT fine-tuning, EVLF-FM attained an average Acc@0.5 of 0.360, outperforming MedGround (0.317). Notably, MedGround achieved similar performance only after few-shot fine-tuning. With few-shot fine-tuning using a small subset of the training data, EVLF-FM further improved to an average Acc@0.5 of 0.545, surpassing BiRD, while requiring only a fraction of the training data. At the modality level, EVLF-FM achieved superior performance to BiRD on MRI (0.332 versus 0.293), dermatoscopy (0.882 versus 0.756), CFP (0.928 versus 0.842) and US (0.530 versus 0.460). Although BiRD retained higher performance on CT, XR, PET and endoscopy, the overall results demonstrate that EVLF-FM achieves competitive visual grounding performance across diverse imaging modalities with substantially reduced training data requirements.

Human grading of model thinking

To assess the quality of EVLF-FM’s generated thinking outputs, we sampled cases from OCT, CFP, chest XR, abdominal CT and pathology images. A total of 12 physicians, including four junior doctors and eight specialists across radiology, pathology and ophthalmology, evaluated the outputs using the S.C.O.R.E framework: safety (S), consensus and context (C), objectivity (O), reproducibility (R) and explainability (E). Each domain was scored from 1 (lowest) to 5 (highest) for each case63. Inter-grader agreement was further assessed using Gwet’s AC2 coefficient and the percentage of ratings within one point between grader pairs.

Overall, EVLF-FM demonstrated consistently strong performance across all S.C.O.R.E domains, achieving an overall mean score of 4.25 ± 0.94 (Fig. 4a and Extended Data Table 7). Among the five domains, objectivity received the highest score (4.36 ± 0.93), suggesting that generated thinking outputs were generally well supported by identifiable imaging findings. Across imaging modalities, OCT achieved the highest overall performance (4.73 ± 0.50). Further subgroup analysis between junior doctors and specialists demonstrated broadly consistent evaluations across most modalities, although junior doctors generally assigned higher scores (Fig. 4b). The difference between groups reached statistical significance only for CT, where specialist reviewers noted the inherent limitations of interpreting complex volumetric imaging studies from a single representative slice. Inter-grader agreement ranged from substantial to almost perfect (Gwet’s AC2: 0.5−0.93), with 74−91% of ratings falling within one point of each other for CT, CFP, OCT and chest XR.

Fig. 4: Evaluation of EVLF-FM thinking outputs using the S.C.O.R.E. framework across imaging modalities and representative examples.

a, Domain-level S.C.O.R.E. evaluation of EVLF-FM thinking outputs aggregated across all imaging modalities, including Safety, Consensus, Objectivity, Reproducibility and Explainability. Each gray dot represents the score for one case averaged across the 3−7 clinicians who evaluated it. Ratings from different clinicians for the same case were repeated observer assessments and were not treated as independent replicates. Center lines indicate medians, boxes indicate the 25th−75th percentiles and whiskers extend to the most extreme values within 1.5× the interquartile range. The minimum and maximum are the lowest and highest displayed points. Black points and bars indicate means and two-sided 95% confidence intervals. b, Comparison of S.C.O.R.E. evaluation scores assigned by junior doctors, senior specialists and the overall average across imaging modalities. Pathology was excluded because it was evaluated only by senior specialists. Ratings from clinicians assessing the same case were averaged at the case level and were not counted as independent replicates. Points and error bars indicate mean values and two-sided 95% confidence intervals across independent cases. c, Representative examples of EVLF-FM thinking outputs across five imaging modalities, showing the input image, predicted diagnosis, model-generated reasoning, average S.C.O.R.E. domain scores assigned by graders and summarized comments from senior specialists. HCC, hepatocellular carcinoma; RPE, retinal pigment epithelium.

Source data

Representative examples from the five evaluated imaging modalities are shown in Fig. 4c, including the input image, model-generated reasoning output, average S.C.O.R.E. domain scores assigned by graders and summarized comments from senior specialists. Overall, EVLF-FM generated structured, stepwise reasoning that explicitly linked visual observations to diagnostic conclusions and was generally well aligned with expert interpretation. Qualitative review from specialists highlighted occasional omission of clinically relevant but non-diagnosis-defining findings, including presence of vitreomacular traction or epiretinal membrane in OCT examples of diabetic macular edema or ancillary radiographic observations such as increased chest volume on chest XR. These findings suggest that, although EVLF-FM can reliably identify major pathological features and generate clinically meaningful reasoning outputs, further improvement in completeness and reporting granularity may enhance its alignment with specialist-level interpretation. Additional examples of thinking output for grounding tasks are included in Fig. 5.

Fig. 5: Representative examples of visual grounding performance of EVLF-FM across imaging modalities.

Representative examples from CT, MRI, dermoscopy, endoscopy and US are shown, including the model-generated grounding boxes, corresponding textual descriptions and comparison with ground truth annotations. Grounding performance is illustrated by the IoU between predicted and ground truth regions. FLAIR, fluid-attenuated inversion recovery; GT, ground truth; Pred, predicted.

Subgroup analysis based on age and sex

To evaluate performance consistency across demographic subgroups, we conducted stratified analyses by age and biological sex on four datasets with available metadata (HAM10000, RAPIER, PAPILA and AIMx). For AIMx, the analysis was restricted to the 381 samples with available age and sex annotations, benchmarking EVLF-FM against InternVL 3 8B and Qwen 3 VL 8B (Extended Data Table 8)29,37,43,64.

Across all datasets, EVLF-FM not only achieved the highest overall performance but also consistently outperformed comparator models within each subgroup (male, female, age 0−60 years and age >60 years). For sex-based stratification, EVLF-FM exhibited particularly small subgroup differences on HAM10000 (0.006) and AIMx (0.005) while maintaining the highest subgroup accuracies across all datasets. Although a larger sex-related difference was observed on PAPILA (0.166), EVLF-FM maintained higher accuracy in both male (0.684) and female (0.850) subgroups than InternVL 3 8B (0.632 and 0.717) and Qwen 3 VL 8B (0.605 and 0.717), respectively. The larger gap, therefore, reflected particularly strong performance in the female subgroup rather than reduced performance in the male subgroup.

A similar trend was observed across age groups. EVLF-FM demonstrated substantially reduced age-related performance gaps on HAM10000 (0.018) and RAPIER (0.018) compared to larger disparities for InternVL 3 8B (0.270 and 0.048) and Qwen 3 VL 7B (0.223 and 0.006), respectively. On HAM10000, the age-related performance gap was reduced by approximately 15-fold relative to InternVL 3 8B (0.018 versus 0.270). Similarly, although larger age-related differences were observed on PAPILA (0.067) and AIMx (0.062), EVLF-FM maintained the highest absolute accuracies in both age groups (PAPILA: 0.743 and 0.810; AIMx: 0.876 and 0.815), exceeding those of the comparator models.

Training strategy ablation analysis

To understand the contribution of each individual component to overall model performance, we performed an ablation study evaluating different components within the EVLF-FM training pipeline, including individual components introduced during first-stage SFT—chain-of-thought (CoT) reasoning data, pixel-level encoder and disease-level encoder—as well as the sentence-level clinical reasoning alignment reward introduced during second-stage group relative policy optimization (GRPO) (Extended Data Table 9).

During SFT training, incorporating curated CoT reasoning data improved model performance compared to the base model (accuracy/F1: 0.811/0.713 versus 0.766/0.648), and further addition of the pixel-level encoder maintained similar overall performance (0.812/0.712). A further improvement was observed after incorporating the disease-level encoder, increasing performance to 0.845/0.769, highlighting its contribution to disease-specific representation learning and diagnostic performance. Applying GRPO alone without SFT initialization resulted in substantially degraded performance (accuracy/F1: 0.676/0.531), confirming that reinforcement learning requires a well-initialized model. SFT alone achieved strong performance (0.845/0.769), and the addition of GRPO improved results to 0.858/0.797, with notable gains in F1 in more challenging datasets such as Breast (0.080) and Derma (0.021). Finally, incorporating the sentence-level clinical reasoning alignment reward yielded the full EVLF-FM model. Although the absolute gains were modest because performance was already near ceiling on several benchmarks, the consistent directional improvements support the effectiveness of the reasoning-aware reward design.

Data efficiency analysis

To assess how the amount of data used for second-stage RFT affects model performance, we conducted a data efficiency analysis. Specifically EVLF-FM was trained using 20% (EVLF-20), 40% (EVLF-40), 60% (EVLF-60) and 80% (EVLF-80) of the original RFT dataset and evaluated on internal diagnostic benchmarks. This analysis provided insight into the sensitivity of EVLF-FM’s performance to the amount of reinforcement data used, informing the scalability and efficiency of fine-tuning (Extended Data Table 9).

EVLF-FM demonstrated strong data efficiency, maintaining robust performance even with limited reinforcement data (Extended Data Table 9). EVLF-20 achieved a mean accuracy/F1 score of 0.823/0.766, compared to 0.851/0.808 for the fully trained model, with performance differences generally modest across datasets. Performance improved progressively with increased data, reaching near-saturation by EVLF-80 (0.849/0.806), suggesting diminishing returns beyond moderate data scaling. Notably, gains were more pronounced in challenging tasks such as ChestMNIST (F1: 0.227 to 0.313) and BloodMNIST (F1: 0.888 to 0.983), whereas high-performing tasks such as PneumoniaMNIST showed only marginal improvements, indicating early convergence. Collectively, these results demonstrate that EVLF-FM achieves strong performance with efficient use of training data while benefiting from a synergistic combination of supervised initialization and reinforcement learning with clinically aligned objectives.

Repeat-prompt consistency evaluation

To assess the reliability of EVLF-FM, we evaluated its response consistency under different prompt formulations by rephrasing each clinical diagnostic question using 10 distinct prompt templates. This experiment was conducted on the internal validation datasets for diagnostic tasks. For each evaluation sample, one of the 10 prompt variants was randomly selected; the complete set of prompt templates is provided in Extended Data Table 10.

EVLF-FM demonstrated highly consistent performance across all 12 datasets despite variation in prompt phrasing (Extended Data Table 10). Using randomly selected prompt variants resulted in only minimal changes in overall performance (mean accuracy/F1 score: 0.845/0.801 versus 0.851/0.808 for the base prompt), with performance remaining stable across all imaging modalities. These findings demonstrate that EVLF-FM is robust to natural language variation in user prompts—an essential characteristic for dependable deployment in real-world clinical practice.

Discussion

EVLF-FM is a multimodal medical VLM designed with additional emphasis on reasoning to enhance explainability in clinical settings. After training with one of the largest and most diverse clinical datasets assembled, EVLF-FM performs three key tasks: disease classification, medical VQA and visual grounding. It demonstrates state-of-the-art performance and consistency in both internal and external validation, outperforming existing generalist VLMs as well as specialist architectures. Through a unique training strategy combining SFT and RFT, EVLF-FM also exhibits robust reasoning capabilities, aligning outputs with visual evidence. These findings demonstrate EVLF-FM’s potential as a scalable, versatile and yet highly explainable clinical decision-support tool, supporting efforts to develop trustworthy, human-aligned AI for healthcare applications.

EVLF-FM introduces several key innovations and strengths. The three core tasks of EVLF-FM, comprising disease diagnosis, medical visual grounding and VQA, support a variety of downstream clinical applications. These tasks are designed to simulate real-life workflow: identifying anatomical structures on imaging, recognizing pathological findings and localizing them precisely with confidence. By incorporating explanation-driven outputs through both explicit stepwise reasoning and pixel-level grounding, EVLF-FM delivers operational interpretability and auditability far beyond traditional black-box models—an essential feature for clinical decision-making and trustworthiness65. Another major advance stems from EVLF-FM’s hybrid training strategy, which combines FM pretraining with SFT and a novel visual RFT phase implemented via GRPO. SFT helps the model adapt to domain-specific tasks and ensures strong baseline performance while requiring less task-specific data and computational overhead compared to training multiple specialist models42. Visual RFT addresses key limitations of supervised schemata by encouraging the model to generate structured, explainable reasoning rather than relying on memorized answers66. To promote reasoning faithfulness during training, we introduce a sentence-level clinical reasoning alignment reward that encourages both completeness and faithfulness of generated rationales against expert references67. This reasoning-aware reward design likely contributes to the superior performance of EVLF-FM across all three tasks relative to other VLMs. Reasoning quality is then independently assessed through clinician evaluation using the S.C.O.R.E. framework. By leveraging verifiable, task-specific rewards for reasoning coherence and grounding accuracy, this method promotes generalization to unfamiliar clinical settings and enhances interpretability, making EVLF-FM both efficient and trustworthy for real-world deployment, and is robust to differences in image format.

The design and capabilities of EVLF-FM lend themselves to further development of applications that can change clinical practice and education. First, versatility across a multitude of clinical imaging media can support healthcare delivery in a range of settings, across many specialties. As a generalizable backbone model, EVLF-FM can be adapted to numerous downstream applications through task-specific fine-tuning, enabling optimization for specific clinical tasks, patient populations or healthcare environments. Crucially, fine-tuning may be undertaken locally to preserve privacy if using local data. In addition, EVLF-FM’s ability to identify regions of abnormality and generate clinically relevant reasoning may provide additional support for image interpretation, workflow prioritization and decision-making. Such capabilities could help clinicians manage increasing imaging workloads by assisting with triage and prioritizing cases that require faster review68. Furthermore, its favorable evaluation by junior doctors in particular using the S.C.O.R.E framework suggests that EVLF-FM may have the potential to augment clinical capacity in resource-limited or high-volume settings, such as emergency departments where rapid preliminary assessment are especially valuable. This capability may also be beneficial in lower-income regions where access to specialist expertise and timely image interpretation remains limited69.

Beyond clinical care, EVLF-FM holds considerable value for medical education. It can enable both medical students and junior doctors to explore cases through interactive diagnosis, grounding and VQA, with visual justifications that support self-directed learning and reinforce clinical reasoning skills70,71. Additionally, similar capabilities could be adapted for patient-facing education, helping individuals better understand their health conditions and care plans. Integration with chatbot applications could further improve patient communication by explaining diagnoses, prognosis and treatment options10,72. A previous study conducted in China found that patients with diabetic retinopathy who received care from both primary care providers and large language models (LLMs) exhibited better self-management behaviors and higher adherence to follow-up treatment compared to those managed solely by primary care providers73. These improvements are likely attributable to the LLM’s capacity to deliver personalized, data-informed recommendations and to facilitate structured communication that supports shared decision-making between patients and clinicians.

Despite its strengths, EVLF-FM has several limitations that warrant further development work. Although the model demonstrated relatively stable performance across multiple datasets and tasks overall, its current accuracy does not yet reach the level typically required for direct clinical deployment (for example, consistently exceeding approximately 90% accuracy across datasets). In particular, performance on VQA tasks was generally lower than on diagnostic classification tasks, a pattern also observed in other medical FMs. This difference likely reflects the inherently greater complexity of VQA, which requires multimodal reasoning and language generation, as well as the relatively limited number of high-quality annotated VQA samples available for training compared to classification datasets. Similarly, further improvement in model reasoning remains an important future direction, as developing advanced reasoning capabilities requires large-scale datasets with high-quality medical images paired with expert annotations, reports and explanations that reflect clinical decision-making processes. Addressing this bottleneck will require closer collaboration between clinicians and engineers to generate richer multimodal training data, and emerging approaches, such as knowledge-enhanced pretraining, may further strengthen the reasoning capabilities of medical FMs74. Rather than representing a deployment-ready system, EVLF-FM should, therefore, be viewed as a generalizable multimodal backbone that can serve as a starting point for downstream applications. With additional task-specific fine-tuning, domain adaptation and integration with clinical workflows, the model has the potential to support a wide range of medical AI applications across specialties.

Another important limitation is that EVLF-FM currently operates on static, two-dimensional image inputs and does not explicitly process volumetric medical imaging data. Although recent FMs have begun to incorporate volumetric medical imaging such as CT and OCT, these models are typically developed for specific modalities and clinical domains with limited generalizability across diverse imaging types75,76. By contrast, EVLF-FM was designed as a broadly applicable multimodal FM across multiple specialties and imaging modalities, highlighting an important tradeoff between versatility and modality-specific optimization in current medical AI development. Future work may explore extending EVLF-FM toward volumetric and time-resolved imaging modalities, provided that sufficiently large-scale, high-quality datasets become available to support training while preserving its broad multimodal capabilities.

Beyond task-specific differences, concerns about model generalizability and bias also remain. Like other large-scale models, EVLF-FM is susceptible to inheriting biases from its training data, which may lead to reduced performance in underrepresented patient populations, clinical environments or imaging modalities77,78. Although our training dataset is large by current standards, it still captures a limited portion of real-world diversity. If the open-source data used are not representative, particularly where certain demographic groups, imaging devices or rare pathologies are underrepresented, EVLF-FM’s predictions in these contexts may be less reliable77,78. Addressing these limitations will require future efforts to curate more diverse, high-quality and representative datasets to ensure broader clinical applicability and fairness. Future work should prioritize curating more diverse and representative training datasets to mitigate this issue.

Additionally, the clinical integration of AI models raises important ethical and practical considerations. EVLF-FM is envisaged as the backbone of human-in-the-loop decision-support tools that augment, rather than replace, clinician judgment79,80. In this framework, although AI outputs can provide additional insights that may assist in pattern recognition, risk stratification or information synthesis, clinicians retain ultimate responsibility for interpretation and clinical decision-making81. Such a collaborative approach may help mitigate the risks associated with overreliance on automated systems while preserving clinical oversight. Rigorous validation in prospective clinical studies, transparency in model use and clear communication of confidence and uncertainty predictions will be necessary to build trust and minimize the risk of errors leading to harm.

In future work, several directions may further advance the development and clinical applicability of EVLF-FM. For instance, additional validation across real-world datasets and healthcare environments will be important to better assess the model’s robustness and generalizability. Collaboration with international clinical and research partners may enable access to larger datasets that are not publicly available82. Fine-tuning on such datasets may further improve model performance in specialized clinical tasks while maintaining the advantages of the general representations learned during FM pretraining. Other groups may undertake local fine-tuning where data sharing is not possible, with the aim of achieving optimal performance in a particular setting. For any intervention designed to change the clinical decision-making process, prospective evaluation in clinical settings will be essential to understand the real-world impact of EVLF-FM. These studies may include controlled prospective studies or randomized controlled trials designed to evaluate how integrating EVLF-FM into clinical workflows influences outcomes such as diagnostic accuracy, workflow efficiency, clinician workload and user trust83,84. Evaluating the model within a human-in-the-loop framework will also help identify potential limitations, workflow challenges and opportunities for refinement. Insights from these studies may guide further model improvement and support the safe and effective integration of multimodal FMs into clinical practice.

Methods

EVLF-FM aims to provide a single, unified solution for classification, detection and interpretative tasks in the medical domain. The framework comprises three main components. (1) Disease-level encoder: a vision encoder that captures global, high-level image features to support improved diagnostic reasoning. This encoder is fine-tuned on large-scale medical imaging data to embed expert-level disease context into its representations. (2) Pixel-level encoder: a vision encoder that retains dense, spatially detailed features for region-specific analysis. This component preserves fine-grained pixel-level information (for example, anatomical structures or lesion details) for precise medical grounding and detailed image understanding. (3) Training strategy: a multi-stage training regimen that first performs SFT (using low-rank adaptation (LoRA)) to align and specialize an LLM for medical tasks and then applies reinforcement learning via GRPO to directly optimize performance and reasoning88. This approach balances parameter efficiency, preservation of prior vision−language knowledge and domain-specific adaptation.

By unifying disease-level and pixel-level features within the model’s architecture, EVLF-FM can address a wide variety of tasks without sacrificing either global domain insight or localized detail.

Disease-level exemplar prompting

To capture clinically salient patterns at a high level, EVLF-FM employs a disease-level encoder denoted \({f}_{\mathrm{disease}}\left(\cdot \right)\), which is fine-tuned on large-scale medical imaging data. Given an input image I , this encoder produces a latent embedding vector e representing the image’s global disease-related features: \(e={f}_{\mathrm{disease}}\left(I\right)\).

A learnable vision−language connector \({\phi }_{\mathrm{disease}}\left(\cdot \right)\) then projects this disease-level embedding into the latent space of the language model, yielding a compatible representation \(\hat{e}={\phi }_{\mathrm{disease}}\left(e\right)\).

Through this mechanism, the model can incorporate high-level medical expertise relevant to disease diagnosis and classification tasks. In practice, \(\hat{e}\) serves as an exemplar prompt that informs the LLM of the image’s overall pathology context, ensuring that subsequent text generation attends to clinically relevant features.

Pixel-level exemplar prompting

To complement the global features, EVLF-FM integrates a pixel-level encoder \({f}_{\mathrm{pixel}}\left(\cdot \right)\) that preserves spatially detailed information for localized analysis. Given the same image I, the pixel-level encoder produces a high-resolution feature map \(p\in {R}^{H\times W\times C}\), where H × W is the spatial resolution and C is the number of channels in the feature map. Instead of spatially pooling these features into one global feature, we pass the full-resolution feature map through a second connector \({\phi }_{\mathrm{pixel}}\left(\cdot \right)\) to obtain a projected feature map \(\hat{p}\) in the LLM’s embedding space: \(\hat{p}={\phi }_{\mathrm{pixel}}\left(p\right)\).

By preserving the complete grid of image features, EVLF-FM can supply the language model with fine-grained visual information. This pixel-level prompt enables the framework to perform explicit region-level reasoning, making it well suited for tasks that require identifying specific anatomical locations or lesions (that is, medical visual grounding and detailed image explanation).

Unified exemplar prompting objective

EVLF-FM jointly leverages the disease-level and pixel-level representations during training through a unified learning objective. For each training sample consisting of an image I, an input instruction or question T and the corresponding ground truth response R, the model conditions on both the global feature \(\hat{e}\) and the fine-grained feature \(\hat{p}\) to generate the response. We train the model to maximize the likelihood of the correct response given these inputs. Formally, the training objective is to minimize the negative log-likelihood:

$$L=-\frac{1}{N}\sum _{i=1}^{N}\mathrm{log}\,{p}_{\theta }\left({R}_{i}|{I}_{i},{T}_{i},\widehat{{e}}_{i},\widehat{{p}}_{i}\right),$$

where N is the number of training examples and θ represents the trainable parameters of the model (including the LLM and the connector networks). In this formulation, the disease-level encoder \({f}_{\mathrm{disease}}\left(\cdot \right)\) and pixel-level encoder \({f}_{\mathrm{pixel}}\left(\cdot \right)\) provide crucial fixed representations (after their initial fine-tuning), and the connectors \({\phi }_{\mathrm{disease}}\left(\cdot \right)\) and \({\phi }_{\mathrm{pixel}}\left(\cdot \right)\) adapt those representations into the language model’s space. By optimizing L, EVLF-FM learns to generate the appropriate response R (such as a diagnosis, answer or description) given both the image and the associated prompts, effectively uniting high-level and low-level visual cues in one learning process.

SFT with LoRA (stage 1)

In the first training stage, we perform SFT to align the multimodal model with medical domain knowledge and instruction-following capabilities. To preserve the pretrained knowledge of the FM and ensure parameter efficiency, we freeze the core vision−language model weights, including the vision encoders and the original LLM parameters, and introduce LoRA layers into the LLM as well as into the disease-level and pixel-level encoders. We allow only the newly introduced LoRA parameters in the LLM and the connector weights (ϕdisease and ϕpixel) to be updated during fine-tuning. This way, the model learns to interpret the medical exemplar prompts and generate appropriate outputs for medical tasks, without overfitting or distorting the base model’s general language and vision−language capabilities.

Reinforcement learning with GRPO (stage 2)

After SFT, we further refine EVLF-FM using reinforcement learning to directly optimize task performance metrics and encourage correct reasoning behavior. We adopt GRPO, a recent variant of the proximal policy optimization algorithm that is particularly efficient for language model fine-tuning. In conventional proximal policy optimization (PPO), a learned value function (critic) is used to estimate a baseline for advantage calculation, which adds substantial complexity and memory overhead, and it can struggle to assign per-token values in language generation. GRPO abrogates the requirement for a separate critic network; instead, it computes advantages by comparing a group of sampled outputs for the same input and using their average reward as an implicit baseline. This group-based relative advantage estimation aligns well with the comparative nature of reward signals in our setting (because our reward functions, described below, compare each output against the correct answer). It also eliminates the variance introduced by an absolute value baseline and simplifies the reinforcement learning pipeline.

In each reinforcement learning update, we start with the policy model from the SFT stage (treated as the initial policy \({\pi }_{{\theta }_{\text{old}}}\)) and sample a set of G candidate outputs \({{\{o}_{i}\}}_{i=1}^{G}\) from it for a given input (for example, a particular question or case). We then compute a reward ri for each output oi based on task-specific criteria (described below). Let \(\bar{r}=\frac{1}{G}{\sum }_{i=1}^{G}{r}_{i}\) be the mean reward for that group of outputs. We define the advantage of each output i as \({A}_{i}={r}_{i}-\bar{r}\), which we further normalize by the group’s standard deviation to obtain \(\widehat{{A}}_{i}\) (for numerical stability). Intuitively, outputs that score higher than the group average receive a positive advantage, and those below the average get a negative advantage. The model is then updated with policy gradient steps that encourage higher probability for tokens in outputs with positive \(\widehat{{A}}_{i}\) and lower probability for tokens in outputs with negative \(\widehat{{A}}_{i}\). To ensure stable learning, we apply the PPO-style clipping mechanism on the policy update and include a Kullback−Leibler divergence penalty to keep the updated policy from drifting too far from the pretrained/SFT policy. Formally, the GRPO objective for the policy network is adapted from the clipped PPO loss as follows:

$${L}_{\text{GRPO}}\left(\theta \right)=-E\left[\min \left({r}_{t}\left(\theta \right)\,\widehat{{A}}_{t},\text{clip}\left({r}_{t}\left(\theta \right),1-\epsilon ,1+\epsilon \right)\widehat{{A}}_{t}\right)\right]+\beta {D}_{\text{KL}}\left({\pi }_{\theta }|{\pi }_{\text{ref}}\right).$$

Here, π denotes the model’s generation policy. \({r}_{t}\left(\theta \right)=\frac{{\pi }_{{\theta }}\left({a}_{t}|{h}_{t}\right)}{{\pi }_{\text{old}}\left({a}_{t}|{h}_{t}\right)}\) is the probability ratio at each token t between the current policy πθ and the the previous-iteration sampling policy πold, and \(\widehat{{A}}_{t}\) is the normalized advantage assigned to that token (in practice derived from the sequence-level Ai above). The hyperparameters \({{\epsilon }}\) and β control the clipping range and the strength of the Kullback−Leibler penalty, respectively. This objective encourages the policy to improve on outputs with above-average reward and avoid regressing on outputs with below-average reward while regularizing the policy update to prevent large deviations. Notably, GRPO achieves this without training a separate value function, substantially reducing the complexity and memory overhead of reinforcement learning training and making it feasible to fine-tune a large multimodal model with limited data and compute.

Reward design

We crafted verifiable reward functions for each task so that the model could be trained with feedback signals grounded in objective correctness, without requiring any human in the loop. In our reinforcement learning fine-tuning, we used only a very small fraction of the data, focusing on examples with reliable ground truth annotations to compute rewards. The reward for a given output was computed as follows for the key tasks:

Open-ended visual question answering

For general visual question-answering tasks requiring free-form explanations or reasoning, we did not apply reinforcement learning with direct numeric rewards as automatically grading open-ended answers goes well beyond simple exact-match criteria. Instead, we used F1 score (a standard evaluation metric in VQA that measures token-level overlap as the harmonic mean of precision and recall) to assess output quality—for example, treating the answer and ground truth as bags of tokens to compute F1 score.

Disease classification

The model’s output in this task is typically a diagnosis (sometimes accompanied by a short explanation). We assign a reward of +1 if the output contains the correct diagnosis in the expected format and zero otherwise. This encourages the model to not only be correct but also to present the answer in a clear, standardized way for clinical utility.

Pathology detection and localization

In this task, the model’s answer includes identification of a finding and an indicated location (for example, the model might output something like ‘Detected finding: lung nodule in the left upper lobe; Location: (x, y, width, height)’). The reward is the localization accuracy. Specifically, we compute the IoU between the model’s predicted region (for example, a bounding box or segmentation mask) and the ground truth region marked by experts; higher IoU yields a higher reward, and the reward is zero if IoU is below a low threshold (meaning the model pointed to the wrong location entirely).

Sentence-level clinical reasoning alignment

Beyond accuracy-based rewards, we introduced a reward that evaluates the quality of the model’s CoT reasoning against reference reasoning. Our method decomposes both the generated and reference reasoning into individual sentences and scores each generated−reference sentence pair by a directional match that aligns their individual clinical terms—crediting partially overlapping findings at the level of terms rather than whole sentences—and gates each match by clinical polarity, so that a statement contradicting the reference earns little or no credit even when its wording is similar (the matching procedure is detailed in Supplementary Note 2, Implementation details of EVLF). For each pair of generated sentence si and reference sentence rj, we obtain a directional match score. From these, we derive two clinically interpretable metrics: Completeness (recall), measuring whether the model’s reasoning addresses all key clinical findings identified by the expert, computed as the average, over reference sentences, of the maximum match score to any generated sentence; and Faithfulness (precision), measuring whether every claim in the model’s reasoning is grounded in the expert reference, computed as the average, over generated sentences, of the maximum match score to any reference sentence. Crucially, precision naturally penalizes verbose reasoning, as additional unsupported sentences directly reduce the precision score, eliminating the need for ad hoc length penalties. The final reasoning alignment reward is the harmonic mean (F1) of precision and recall, ensuring that neither metric can be trivially maximized at the expense of the other. This reward is combined additively with the task-specific accuracy reward: \({r}_{\mathrm{total}}={r}_{\mathrm{accuracy}}+\alpha {r}_{\mathrm{think}}\). Samples without reference reasoning receive rthink = 0, allowing graceful degradation to pure accuracy-based training. The sentence-level decomposition offers several advantages: it maps directly to established dimensions of clinical reasoning quality (Faithfulness and Completeness), provides natural verbosity control and is robust to stylistic variation in how clinical reasoning is expressed.

During GRPO training, the model generates multiple candidate answers per query and receives the above reward for each; the group-relative advantage formulation then guides the model to preferentially produce answers that maximize these rewards. Notably, this process does not require a human preference model or manual rubric; the reward for each output is computed automatically against the known correct answer or annotation, making the reinforcement learning procedure stable and reproducible. We ran the GRPO fine-tuning for only a small number of iterations on the selected subset of data (again emphasizing efficiency). Even with this limited exposure, the impact on EVLF-FM was notable: the model learned to avoid minor errors that it sometimes made after the SFT stage (for example, cases where it would output an almost-correct diagnosis but with slightly off terminology or omit a required detail), and it became more consistent in producing well-formatted, complete answers. This translates to improved predictive accuracy—for instance, we observed higher exact-match accuracy in diagnosis and better IoU scores in localization after reinforcement learning fine-tuning—as well as enhanced clinical interpretability of the model’s outputs. The interpretability gains are especially evident in the detection task, where the model (when asked) can now pinpoint the finding on the image with greater precision and appropriately calibrated confidence, essentially providing a visual explanation for its diagnosis. We also observed that the model’s open-ended explanations (when it provides reasoning or descriptions) became more aligned with clinical reasoning after reinforcement learning training, likely because the model learned to focus on the key image features that maximize the reward (which, by design, correspond to the ground truth findings). This outcome is in line with reports in other domains that policy optimization on a small subset of high-quality data can yield outsized improvements. By utilizing GRPO on the subset of our training data, we efficiently fine-tuned EVLF-FM to not only be more correct in its answers but also to present those answers in a manner that is trusted and useful to clinicians—a crucial aspect for real-world deployment.

In summary, our method blends architectural innovations with a comprehensive training regimen to produce a model with strong clinical reasoning abilities. The dual-encoder design provides EVLF-FM with a rich visual foundation: the disease-level encoder ensures broad diagnostic awareness of the image, and the pixel-level encoder provides granular visual evidence. The supervised training approach establishes the model’s medical knowledge and basic vision−language alignment, with LoRA enabling effective adaptation on top of a powerful FM without overfitting and GPT-augmented rationales seeding the model with improved explainability. Finally, GRPO-based reinforcement learning directly optimizes the model’s behavior toward correct and explainable outputs, honing the model’s focus on decision-critical details. Each step is deliberately designed to bridge the gap between raw medical images and the expert reasoning of a physician. The result is a multimodal FM—EVLF-FM—that achieves high performance on diagnostic tasks while also demonstrating interpretable reasoning and alignment with clinical norms. In practice, EVLF-FM has the potential to serve as a reliable AI assistant in medicine, capable of both answering ‘What is it?’ with accuracy and explaining ‘Why?’ in human-understandable terms, thereby maximizing its real-world utility for clinical decision support.

Reasoning data curation pipeline

To construct a high-quality CoT training dataset for medical visual reasoning, we designed a multi-stage data curation pipeline that systematically generates, validates and refines visual explanations for disease diagnosis. The pipeline comprises five stages: (1) CoT generation, (2) semantic uncertainty filtering, (3) GPT-based quality scoring, (4) CoT refinement with rescoring and (5) final quality filtering.

Dataset selection

We curated training data from 26 publicly available medical image classification datasets spanning seven imaging modalities: fundus photography (6), OCT (2), chest XR (5), CT (3), histopathology (4), dermoscopy (3) and US (3). These datasets are drawn from the training splits of established medical imaging benchmarks, including widely used public datasets such as AIROGS, EyePACS, HAM10000, RSNA Pneumonia and LC25000. This selection ensures broad coverage across anatomical regions, imaging protocols and disease categories while maintaining strict separation from the evaluation (test) splits.

Stage 1: CoT generation

For each training sample consisting of a medical image, a task-specific prompt and a ground truth label, we generated a CoT description that articulates the visual evidence supporting the diagnosis. We employed MedGemma-1.5 as the primary generator. Given an image and its ground truth label, the model was prompted to describe only the observable visual findings that support the answer, producing a structured response within <thinking> tags. For OCT images, where MedGemma’s performance is limited, we used GPT-5-mini via the OpenAI Batch API as an alternative generator.

Stage 2: Semantic uncertainty filtering

To filter out unreliable CoT descriptions where the model lacks genuine visual understanding, we adapted vision-amplified semantic entropy (VASE), a hallucination detection method based on semantic entropy85,86. For each sample, we performed M = 10 stochastic inference passes using MedGemma: five passes with weak visual transformations applied to the original image (random cropping, rotation and brightness/contrast adjustment) and five passes with the image additionally distorted by Gaussian and Poisson noise. The sampled responses were clustered into semantic equivalence classes using bidirectional entailment assessed by a DeBERTa-Large-MNLI model87. The probability of each semantic class was estimated by aggregating the generation probabilities of its constituent responses. After VASE85, we computed a contrastive semantic predictive distribution that amplifies the influence of visual input by contrasting the distributions derived from clean and distorted images85:

$$P_{\mathrm{con}}\left(c\mid x_v,x_q\right)=\sigma\left((1+\alpha)P\left(c\mid x_v,x_q\right)-\alpha P\left(c\mid x^\prime_v,x_q\right)\right)$$

where α is the vision-amplified ratio. The VASE score is then defined as the entropy of this contrastive distribution:

$$\mathrm{VASE}\left({x}_{v},{x}_{q}\right)=-\sum _{i=1}^{N}{P}_{\mathrm{con}}\left({c}_{i}|{x}_{v},{x}_{q}\right)\mathrm{log}\,{P}_{\mathrm{con}}\left({c}_{i}|{x}_{v},{x}_{q}\right)$$

A high VASE score indicates that the model’s response varies significantly with visual perturbations, suggesting hallucination or insufficient visual grounding. We retained only samples with VASE < 0.5.

Stage 3: GPT-based quality scoring

Samples passing the VASE filter were submitted to GPT-5-mini for multi-dimensional quality assessment. Each sample’s CoT was evaluated along four axes: (1) format correctness (proper tag usage), (2) grounding fidelity (descriptions limited to visible findings without hallucination), (3) relevance to the question and answer and (4) overall quality as a training sample. Each dimension was scored from 1 (very poor) to 5 (excellent). We retained samples with an overall score ≥3 for refinement.

Stage 4: CoT refinement and rescoring

For samples that passed the quality threshold, we performed a refinement step to further improve CoT quality. GPT-5-mini was prompted with the original CoT, its quality assessment and the identified issues and asked to produce a revised description that preserves accurate observations while correcting any deficiencies. The refined CoTs were then re-scored using the same multi-dimensional rubric to ensure that refinement did not degrade quality.

Stage 5: Final filtering

In the final stage, we applied a re-score threshold of ≥4 on the overall dimension to produce the curated training set.

Pipeline statistics

The pipeline processed 110,290 initial samples. After VASE filtering (<0.5), 78,049 samples remained (70.8% pass rate). After GPT quality scoring (≥3), 61,531 samples survived. After refinement and rescoring (≥4), the final curated dataset contained 60,455 high-quality CoT training samples spanning 26 datasets across seven imaging modalities, yielding an end-to-end retention rate of 55%. The VASE stage provided the most aggressive filtering, indicating that approximately 30% of initially generated CoTs contained hallucinated or insufficiently grounded content.

Reporting summary

Further information on research design is available in the Nature Portfolio Reporting Summary linked to this article.

Data availability

This study used a combination of publicly available, controlled-access and access-restricted clinical datasets for the development and evaluation of EVLF-FM. Publicly available datasets were obtained from their original repositories; controlled-access datasets were obtained through the corresponding data access mechanisms and data use agreements. A complete list of all datasets, including dataset description and accession information, is provided in the Reporting Summary and the Supplementary Information. Access-restricted clinical datasets, including RAPIER, AIMx and SEED, are not publicly available because they contain identifiable clinical data and are subject to institutional review board approvals, patient privacy regulations and institutional data governance policies. Deidentified data from these datasets may be made available for non-commercial academic research from the corresponding author (daniel.ting@duke-nus.edu.sg) upon reasonable request, subject to approval by the relevant institutional data management committees, applicable ethics requirements and execution of a data transfer or data use agreement. Access to the DRCR Retina Network imaging data is administered by the DRCR Retina Network and the Jaeb Center for Health Research and should be requested directly through their established data access procedures. Source data are provided with this paper.

Code availability

The code repository can be found at https://github.com/baiyang4/EVLF. The code is also archived on Zenodo at https://doi.org/10.5281/zenodo.21772695.

References

  1. LeCun, Y., Bengio, Y. & Hinton, G. Deep learning. Nature 521, 436–444 (2015).

    Article  CAS  PubMed  Google Scholar 

  2. Zhou, Y. et al. A foundation model for generalizable disease detection from retinal images. Nature 622, 156–163 (2023).

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  3. Chen, R. J. et al. Towards a general-purpose foundation model for computational pathology. Nat. Med. 30, 850–862 (2024).

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  4. Teo, Z. L. et al. Generative artificial intelligence in medicine. Nat. Med. 31, 3270–3282 (2025).

    Article  CAS  PubMed  Google Scholar 

  5. Zhang, S. et al. A multimodal biomedical foundation model trained from fifteen million image–text pairs. NEJM AI 2, AIoa2400640 (2025).

    Article  Google Scholar 

  6. Tu, T. et al. Towards conversational diagnostic artificial intelligence. Nature 642, 442−450 (2025).

  7. Thirunavukarasu, A. J. et al. Large language models in medicine. Nat. Med. 29, 1930–1940 (2023).

    Article  CAS  PubMed  Google Scholar 

  8. Singhal, K. et al. Toward expert-level medical question answering with large language models. Nat. Med. 31, 943–950 (2025).

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  9. Moor, M. et al. Foundation models for generalist medical artificial intelligence. Nature 616, 259–265 (2023).

    Article  CAS  PubMed  Google Scholar 

  10. Zhang, K. et al. A generalist vision–language foundation model for diverse biomedical tasks. Nat. Med. 30, 3129–3141 (2024).

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  11. Wang, R. et al. Medical image segmentation using deep learning: a survey. IET Image Process. 16, 1243–1267 (2022).

    Article  Google Scholar 

  12. Yang, H. et al. Multimodal self-supervised learning for lesion localization. Preprint at https://doi.org/10.48550/arXiv.2401.01524 (2024).

  13. Foo, L. L. et al. Deep learning system to predict the 5-year risk of high myopia using fundus imaging in children. NPJ Digit. Med. 6, 10 (2023).

    Article  PubMed  PubMed Central  Google Scholar 

  14. Lundberg, S. & Lee, S.-I. A unified approach to interpreting model predictions. Preprint at https://doi.org/10.48550/arXiv.1705.07874 (2017).

  15. Liu, H., Li, C., Li, Y. & Lee, Y. J. Improved baselines with visual instruction tuning. In 2024 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) 26286–26296 (IEEE, 2024).

  16. Bai, S. et al. Qwen2.5-VL Technical Report. Preprint at https://doi.org/10.48550/arXiv.2502.13923 (2025).

  17. Li, C. et al. LLaVA-Med: training a large language-and-vision assistant for biomedicine in one day. Preprint at https://doi.org/10.48550/arXiv.2306.00890 (2023).

  18. Simonyan, K. & Zisserman, A. Very deep convolutional networks for large-scale image recognition. Preprint at https://doi.org/10.48550/arXiv.1409.1556 (2014).

  19. Pérez-García, F. et al. Exploring scalable medical image encoders beyond text supervision. Nat. Mach. Intell. 7, 119–130 (2025).

    Article  Google Scholar 

  20. Jiao, J. et al. USFM: a universal ultrasound foundation model generalized to tasks and organs towards label efficient image analysis. Med. Image Anal. 96, 103202 (2024).

    Article  PubMed  Google Scholar 

  21. He, K., Zhang, X., Ren, S. & Sun, J. Deep residual learning for image recognition. In 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) https://doi.org/10.1109/CVPR.2016.90 (IEEE, 2016).

  22. Huang, G., Liu, Z., Maaten, L. V. D. & Weinberger, K. Q. Densely connected convolutional cetworks. In 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) 2261–2269 (IEEE, 2017).

  23. Tan, M. & Le, Q.V. EfficientNet: rethinking model scaling for convolutional neural networks. Preprint at https://doi.org/10.48550/arXiv.1905.11946 (2019).

  24. Dosovitskiy, A. et al. An image is worth 16x16 words: transformers for image recognition at scale. Preprint at https://doi.org/10.48550/arXiv.2010.11929 (2020).

  25. Radford, A. et al. Learning transferable visual models from natural language supervision. Preprint at https://doi.org/10.48550/arXiv.2103.00020 (2021).

  26. Caron, M., et al. Emerging properties in self-supervised vision transformers. In 2021 IEEE/CVF International Conference on Computer Vision (ICCV) 9630–9640 (IEEE Computer Society, 2021).

  27. Kirillov, A. et al. Segment anything. Preprint at https://doi.org/10.48550/arXiv.2304.02643 (2023).

  28. Vorontsov, E. et al. A foundation model for clinical-grade computational pathology and rare cancers detection. Nat. Med. 30, 2924–2935 (2024).

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  29. Tschandl, P., Rosendahl, C. & Kittler, H. The HAM10000 dataset, a large collection of multi-source dermatoscopic images of common pigmented skin lesions. Nat. Sci. Data 5, 180161 (2018).

    Article  Google Scholar 

  30. Hernández-Pérez, C. et al. BCN20000: dermoscopic lesions in the wild. Sci. Data 11, 641 (2024).

    Article  PubMed  PubMed Central  Google Scholar 

  31. Aubreville, M. et al. A comprehensive multi-domain dataset for mitotic figure detection. Sci. Data 10, 484 (2023).

    Article  PubMed  PubMed Central  Google Scholar 

  32. Borkowski, A. A. et al. Lung and colon cancer histopathological image dataset (LC25000). Preprint at https://doi.org/10.48550/arXiv.1912.12142 (2019).

  33. Stein, A. et al. RSNA Pneumonia Detection Challenge https://kaggle.com/competitions/rsna-pneumonia-detection-challenge (Kaggle, 2018).

  34. Wang, X. et al. ChestX-ray8: hospital-scale chest X-ray database and benchmarks on weakly-supervised classification and localization of common thorax diseases. Preprint at https://doi.org/10.1109/CVPR.2017.369 (2017).

  35. Al-Dhabyani, W., Gomaa, M., Khaled, H. & Fahmy, A. Dataset of breast ultrasound images. Data Brief 28, 104863 (2020).

    Article  PubMed  Google Scholar 

  36. Burgos-Artizzu, X. P. et al. Evaluation of deep convolutional neural networks for automatic classification of common maternal fetal ultrasound planes. Sci. Rep. 10, 10200 (2020).

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  37. Kovalyk, O. et al. PAPILA: dataset with fundus images and clinical data of both eyes of the same patient for glaucoma assessment. Sci. Data 9, 291 (2022).

    Article  PubMed  PubMed Central  Google Scholar 

  38. Ocular Disease Recognition https://www.kaggle.com/datasets/andrewmvd/ocular-disease-recognition-odir5k (Kaggle, 2020).

  39. Yang, J. et al. MedMNIST v2 - a large-scale lightweight benchmark for 2D and 3D biomedical image classification. Sci. Data 10, 41 (2023).

    Article  PubMed  PubMed Central  Google Scholar 

  40. Wu, C. et al. Towards generalist foundation model for radiology by leveraging web-scale 2D&3D medical data. Nat. Commun. 16, 7866 (2025).

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  41. Moor, M. et al. Med-Flamingo: a multimodal medical few-shot learner. Preprint at https://doi.org/10.48550/arXiv.2307.15189 (2023).

  42. He, S. et al. Towards generalizable AI in medicine via generalist–specialist collaboration. Nat. Biomed. Eng. https://doi.org/10.1038/s41551-026-01653-3 (2026).

  43. Chen, Z. et al. Intern VL: scaling up vision foundation models and aligning for generic visual-linguistic tasks. In 2024 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) 24185–24198 (IEEE, 2024).

  44. Lau, J. J., Gayen, S., Ben Abacha, A. & Demner-Fushman, D. A dataset of clinically generated visual questions and answers about radiology images. Sci. Data 5, 180251 (2018).

    Article  PubMed  PubMed Central  Google Scholar 

  45. Liu, B. et al. Slake: a semantically-labeled knowledge-enhanced dataset for medical visual question answering. In 2021 IEEE 18th International Symposium on Biomedical Imaging (ISBI) 1650–1654 (IEEE, 2021).

  46. He, X., Zhang, Y., Mou, L., Xing, E. & Xie, P. PathVQA: 30000+ questions for medical visual question answering. Preprint at https://doi.org/10.48550/arXiv.2003.10286 (2020).

  47. Zhao, T. et al. A foundation model for joint segmentation, detection and recognition of biomedical objects across nine modalities. Nat. Methods 22, 166−176 (2024).

  48. Nguyen, H. Q. et al. VinDr-CXR: an open dataset of chest X-rays with radiologist’s annotations. Sci. Data 9, 429 (2022).

    Article  PubMed  PubMed Central  Google Scholar 

  49. Chen, J. et al. HuatuoGPT-Vision, towards injecting medical visual knowledge into multimodal LLMs at scale. Preprint at https://doi.org/10.48550/arXiv.2406.19280 (2024).

  50. Sellergren, A. et al. An open vision-language model for diverse medical applications. Nat. Med. https://doi.org/10.1038/s41591-026-04626-w (2026).

  51. Abedi, I., Vali, M., Otroshi, B., Zamanian, M. & Bolhasani, H. HRCTCov19-a high-resolution chest CT scan image dataset for COVID-19 diagnosis and differentiation. BMC Res. Notes 17, 32 (2024).

    Article  PubMed  PubMed Central  Google Scholar 

  52. Soares, E., Angelov, P., Biaso, S., Froes, M. H. & Abe, D. K. SARS-CoV-2 CT-scan dataset: a large dataset of real patients CT scans for SARS-CoV-2 identification. Preprint at medRxiv https://doi.org/10.1101/2020.04.24.20078584 (2020).

  53. Kermany, D. S. et al. Identifying medical diagnoses and treatable diseases by image-based deep learning. Cell 172, 1122–1131 (2018).

    Article  CAS  PubMed  Google Scholar 

  54. Bertram, C. A. et al. Histologic dataset of normal and atypical mitotic figures on human breast cancer (AMi-Br). In Bildverarbeitung für die Medizin 2025 (eds Palm, C. et al.) 113–118 (Springer Fachmedien Wiesbaden, 2025).

  55. Vallez, N., Bueno, G., Deniz, O., Rienda, M. A. & Pastor, C. BUS-UCLM: breast ultrasound lesion segmentation dataset. Sci. Data 12, 242 (2025).

    Article  PubMed  PubMed Central  Google Scholar 

  56. Multimodal Breast Ultrasound Dataset US3M. https://www.kaggle.com/datasets/timesxy/multimodal-breast-ultrasound-dataset-us3m (Kaggle, 2026).

  57. Naren, O. S. Retinal OCT Image Classification - C8. https://doi.org/10.34740/KAGGLE/DSV/2736749 (Kaggle, 2021).

  58. Karthik, M. & Sohier, D. APTOS 2019 Blindness Detection. https://kaggle.com/competitions/aptos2019-blindness-detection (Kaggle, 2019).

  59. Ahn, J. M. et al. A deep learning model for the detection of both advanced and early glaucoma using fundus photography. PLoS ONE 13, e0207982 (2018).

    Article  PubMed  PubMed Central  Google Scholar 

  60. Majithia, S. et al. Cohort profile: the Singapore Epidemiology of Eye Diseases study (SEED). Int. J. Epidemiol. 50, 41–52 (2021).

    Article  PubMed  Google Scholar 

  61. Huang, X., et al. A refer-and-ground multimodal large language model for biomedicine. In Medical Image Computing and Computer Assisted Intervention – MICCAI 2024 (eds. Linguraru, M. G., et al.) 399–409 (Springer Nature Switzerland, 2024).

  62. Zou, K. et al. Uncertainty-aware medical diagnostic phrase identification and grounding. IEEE Trans. Pattern Anal. Mach. Intell. 47, 11315–11329 (2025).

    Article  PubMed  Google Scholar 

  63. Tan, T. F. et al. A S.C.O.R.E. framework for evaluating open-ended responses from large language models in healthcare. Cell Rep. Med. 7, 102883 (2026).

  64. Bai, S. et al. Qwen3-VL Technical Report. https://doi.org/10.48550/arXiv.2511.21631 (2025).

  65. Rosenbacke, R., Melhus, Å, McKee, M. & Stuckler, D. How explainable artificial intelligence can increase or decrease clinicians’ trust in AI applications in health care: systematic review. JMIR AI 3, e53207 (2024).

    Article  PubMed  PubMed Central  Google Scholar 

  66. Liu, Z. et al. Visual-RFT: visual reinforcement fine-tuning. Preprint at https://doi.org/10.48550/arXiv.2503.01785 (2025).

  67. Pan, J., et al. MedVLM-R1: incentivizing medical reasoning capability of vision-language models (VLMs) via reinforcement learning. In Medical Image Computing and Computer Assisted Intervention – MICCAI 2025 (eds. Gee, J.C., et al.) 337–347 (Springer Nature Switzerland, 2026).

  68. Annarumma, M. et al. Automated triaging of adult chest radiographs with deep artificial neural networks. Radiology 291, 196–202 (2019).

    Article  PubMed  PubMed Central  Google Scholar 

  69. Tan, T. F. et al. Artificial intelligence and digital health in global eye health: opportunities and challenges. Lancet Glob. Health 11, e1432–e1443 (2023).

    Article  CAS  PubMed  Google Scholar 

  70. Ng, F. Y. C. et al. Artificial intelligence education: an evidence-based medicine approach for consumers, translators, and developers. Cell Rep. Med. 4, 101230 (2023).

    Article  PubMed  PubMed Central  Google Scholar 

  71. Ning, Y. et al. How can artificial intelligence transform the training of medical students and physicians? Lancet Digit. Health 7, 100900 (2025).

    Article  CAS  PubMed  Google Scholar 

  72. Huo, B. et al. Large language models for chatbot health advice studies: a systematic review. JAMA Netw. Open 8, e2457879 (2025).

    Article  PubMed  PubMed Central  Google Scholar 

  73. Li, J. et al. Integrated image-based deep learning and language models for primary diabetes care. Nat. Med. 30, 2886−2896 (2024).

  74. Zhou, X. et al. Knowledge-enhanced pretraining for vision-language pathology foundation model on cancer diagnosis. Cancer Cell 44, 777–791 (2026).

    Article  CAS  PubMed  Google Scholar 

  75. Liu, Z. et al. A three-dimensional multi-modal foundation model for optical coherence tomography. Nat. Biomed. Eng. https://doi.org/10.1038/s41551-026-01662-2 (2026).

  76. Blankemeier, L. et al. Merlin: a computed tomography vision–language foundation model and dataset. Nature 652, 1318–1328 (2026).

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  77. Yu, A. C., Mohajer, B. & Eng, J. External validation of deep learning algorithms for radiologic diagnosis: a systematic review. Radiol. Artif. Intell. 4, e210064 (2022).

    Article  PubMed  PubMed Central  Google Scholar 

  78. Yang, Y. et al. Demographic bias of expert-level vision-language foundation models in medical imaging. Sci. Adv. 11, eadq0305 (2025).

    Article  PubMed  PubMed Central  Google Scholar 

  79. Liu, M. et al. A translational perspective towards clinical AI fairness. NPJ Digit. Med. 6, 172 (2023).

    Article  PubMed  PubMed Central  Google Scholar 

  80. Yu, F. et al. Heterogeneity and predictors of the effects of AI assistance on radiologists. Nat. Med. 30, 837–849 (2024).

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  81. Thirunavukarasu, A. J. Large language models will not replace healthcare professionals: curbing popular fears and hype. J. R. Soc. Med. 116, 181–182 (2023).

    Article  PubMed  PubMed Central  Google Scholar 

  82. Tham, Y. C. et al. Building the world’s first truly global medical foundation model. Nat. Med. 31, 3580−3585 (2025).

  83. Wu, Y. et al. An eyecare foundation model for clinical assistance: a randomized controlled trial. Nat. Med. 31, 3404–3413 (2025).

    Article  CAS  PubMed  Google Scholar 

  84. van Amsterdam, W. A. C. et al. Clinical trials for continuously monitored and updated AI systems. Nat. Med. 32, 1972−1974 (2026).

  85. Wang, C., Zhou, X., Fu, W. & Zhou, Y. Mitigating hallucinations in large vision-language models with internal fact-based contrastive decoding. Preprint at https://doi.org/10.48550/arXiv.2502.01056 (2025).

  86. Farquhar, S., Kossen, J., Kuhn, L. & Gal, Y. Detecting hallucinations in large language models using semantic entropy. Nature 630, 625–630 (2024).

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  87. He, P., Liu, X., Gao, J. & Chen, W. DeBERTa: decoding-enhanced BERT with disentangled attention. Preprint at https://doi.org/10.48550/arXiv.2006.03654 (2020).

  88. Hu, E. J. et al. LoRA: low-rank adaptation of large language models. Preprint at https://doi.org/10.48550/arXiv.2106.09685 (2021).

Download references

Acknowledgements

This study was approved by the SingHealth Centralised Institutional Review Board (CIRB; reference no. 2025-0836).

Funding

This study is supported by grants from the National Medical Research Council, Singapore (MOH-001689-00, MOH-000655-00 and MOH-001014-00); Duke-NUS Medical School (Duke-NUS/RSF/2021/0018, 05/FY2020/EX/15-A58 and 05/FY2022/EX/66-A128); the Agency for Science, Technology and Research, Singapore (A20H4g2141 and H20C6a0032); Research to Prevent Blindness; and National Institutes of Health P30 EY003790. A.J.T. is supported by National Institute for Health & Care Research (ACF-2025-20-001). Y.Z. is supported by the National Natural Science Foundation of China (62506270). The funders had no role in study design, data collection and analysis, decision to publish or preparation of the manuscript.

Author information

Author notes

  1. These authors contributed equally: Yang Bai, Haoran Cheng, Yang Zhou.

  2. These authors jointly supervised this work: Yong Liu, Daniel Shu Wei Ting.

Authors and Affiliations

  1. Institute of Advanced Intelligence and Computing (IAIC), Agency for Science, Technology and Research (A*STAR), Singapore, Singapore

    Yang Bai, Yang Zhou, Jun Zhou, Le Dinh Linh, Rick Siow Mong Goh & Yong Liu

  2. Singapore Eye Research Institute, Singapore National Eye Center, Singapore, Singapore

    Haoran Cheng, Jie Yao, Kanae Fukutsu, Chrystie Wan Ning Quek, Ashley Shuen Ying Hong, Laura Gutierrez, Alessandro Berni, Zhen Ling Teo, Darren Shu Jeng Ting, Zengxiang Li, Ching Yu Cheng, Yih Chung Tham, Gavin Siew Wei Tan, Leopold Schmetterer, Damon Wong, Marcus Ang, Rahat Hussain, Jodhbir S. Mehta, Tin Aung, Tien Yin Wong & Daniel Shu Wei Ting

  3. Duke-NUS Medical School, National University of Singapore, Singapore, Singapore

    Haoran Cheng, Yuhe Ke, Jie Yao, Zhen Ling Teo, Darren Shu Jeng Ting, Hiok Hong Chan, Gavin Siew Wei Tan, Leopold Schmetterer, Damon Wong, Marcus Ang, Rahat Hussain, Jodhbir S. Mehta, Tin Aung, Lionel Tim-Ee Cheng, Chee Leong Cheng, Iain Beehuat Tan, Soon Thye Lim & Daniel Shu Wei Ting

  4. School of Computer Science, Wuhan University, Wuhan, China

    Yang Zhou

  5. Nuffield Department of Clinical Neurosciences, Medical Sciences Division, University of Oxford, Oxford, UK

    Arun Thirunavukarasu

  6. International Centre for Eye Health, London School of Hygiene & Tropical Medicine, London, UK

    Arun Thirunavukarasu

  7. Department of Anesthesia and Perioperative Medicine, Singapore General Hospital, Singapore, Singapore

    Yuhe Ke

  8. Department of Ophthalmology, Faculty of Medicine and Graduate School of Medicine, Hokkaido University, Sapporo, Japan

    Kanae Fukutsu

  9. Department of Inflammation and Ageing, College of Medicine and Health, University of Birmingham, Birmingham, UK

    Darren Shu Jeng Ting

  10. Birmingham and Midland Eye Centre, Sandwell and West Birmingham NHS Trust, Birmingham, UK

    Darren Shu Jeng Ting & Daniel Shu Wei Ting

  11. Byers Eye Institute, Stanford University School of Medicine, Palo Alto, CA, USA

    Brian T. Soetikno & Darius M. Moshfeghi

  12. Cumming School of Medicine, University of Calgary, Calgary, Alberta, Canada

    Christopher S. Nielsen

  13. Massachusetts Eye and Ear, Department of Ophthalmology, Harvard Medical School, Boston, MA, USA

    Tobias Elze & Daniel Shu Wei Ting

  14. SingHealth AI Office, Singapore, Singapore

    Zengxiang Li

  15. Cleveland Clinic Cole Eye Institute, Cleveland, OH, USA

    Sumit Sharma

  16. Cleveland Clinic Lerner College of Medicine of Case Western Reserve University, Cleveland, OH, USA

    Sumit Sharma

  17. Heart, Vascular and Thoracic Institute, Cleveland Clinic, Cleveland, OH, USA

    Nicholas Ruthmann

  18. Department of Ophthalmology, National University Health System, Singapore, Singapore

    Victor Koh & Marcus Tan

  19. Centre for Innovation and Precision Eye Health, Department of Ophthalmology, Yong Loo Lin School of Medicine, National University of Singapore, Singapore, Singapore

    Victor Koh, Marcus Tan, Ching Yu Cheng & Yih Chung Tham

  20. Department of Ophthalmology, National Healthcare Group, Singapore, Singapore

    Kelvin Z. Li & Leonard Yip

  21. Lee Kong Chien School of Medicine, Nanyang Technology University, Singapore, Singapore

    Kelvin Z. Li & Leonard Yip

  22. SERI-NTU Advanced Ocular Engineering (STANCE) Program, Singapore, Singapore

    Leopold Schmetterer

  23. School of Chemistry, Chemical Engineering and Biotechnology, Nanyang Technological University, Singapore, Singapore

    Leopold Schmetterer

  24. Department of Clinical Pharmacology, Medical University of Vienna, Vienna, Austria

    Leopold Schmetterer

  25. Aier Hospital Group, Changsha, China

    Leopold Schmetterer

  26. Fondation Ophtalmologique Adolphe De Rothschild, Paris, France

    Leopold Schmetterer

  27. Center for Medical Physics and Biomedical Engineering, Medical University Vienna, Vienna, Austria

    Leopold Schmetterer

  28. Department of Ophthalmology and Visual Science, Seoul St. Mary’s Hospital, College of Medicine, The Catholic University of Korea, Seoul, Korea

    Rae-Young Kim

  29. Department of Cardiothoracic and Abdominal Radiology, Singapore General Hospital, Singapore, Singapore

    Lionel Tim-Ee Cheng & Nguyen Tuan Anh Tran

  30. SingHealth Community Hospitals, Singapore Health Services, Singapore, Singapore

    Lionel Tim-Ee Cheng

  31. Department of Anatomical Pathology, Singapore General Hospital, Singapore, Singapore

    Chee Leong Cheng & Tony Kiat Hon Lim

  32. School of Clinical Medicine, Beijing Tsinghua Changgung Hospital, Tsinghua Medicine, Tsinghua University, Beijing, China

    Tien Yin Wong

  33. Beijing Visual Science and Translational Eye Research Institute, Beijing Tsinghua Changgung Hospital Eye Center, Tsinghua Medicine, Tsinghua University, Beijing, China

    Tien Yin Wong

  34. School of Biomedical Engineering, Tsinghua Medicine, Tsinghua University, Beijing, China

    Tien Yin Wong

  35. Centre for Quantitative Medicine, Duke-NUS Medical School, Singapore, Singapore

    Nan Liu

  36. Programme in Health Services and Systems Research, Duke-NUS Medical School, Singapore, Singapore

    Nan Liu

  37. NUS AI Institute, National University of Singapore, Singapore, Singapore

    Nan Liu

  38. Division of Medical Oncology, National Cancer Centre Singapore, Singapore, Singapore

    Iain Beehuat Tan & Soon Thye Lim

  39. Genome Institute of Singapore, Agency for Science, Technology and Research (A*STAR), Singapore, Singapore

    Iain Beehuat Tan

  40. Generative AI Research Program, Icahn School of Medicine at Mount Sinai, New York, NY, USA

    Eyal Klang

Authors

  1. Yang Bai
  2. Haoran Cheng
  3. Yang Zhou
  4. Jun Zhou
  5. Arun Thirunavukarasu
  6. Yuhe Ke
  7. Jie Yao
  8. Kanae Fukutsu
  9. Chrystie Wan Ning Quek
  10. Ashley Shuen Ying Hong
  11. Laura Gutierrez
  12. Alessandro Berni
  13. Zhen Ling Teo
  14. Darren Shu Jeng Ting
  15. Brian T. Soetikno
  16. Christopher S. Nielsen
  17. Tobias Elze
  18. Zengxiang Li
  19. Sumit Sharma
  20. Nicholas Ruthmann
  21. Le Dinh Linh
  22. Hiok Hong Chan
  23. Victor Koh
  24. Marcus Tan
  25. Kelvin Z. Li
  26. Leonard Yip
  27. Ching Yu Cheng
  28. Yih Chung Tham
  29. Gavin Siew Wei Tan
  30. Leopold Schmetterer
  31. Rae-Young Kim
  32. Damon Wong
  33. Marcus Ang
  34. Rahat Hussain
  35. Jodhbir S. Mehta
  36. Tin Aung
  37. Lionel Tim-Ee Cheng
  38. Nguyen Tuan Anh Tran
  39. Chee Leong Cheng
  40. Tien Yin Wong
  41. Darius M. Moshfeghi
  42. Nan Liu
  43. Iain Beehuat Tan
  44. Soon Thye Lim
  45. Eyal Klang
  46. Tony Kiat Hon Lim
  47. Rick Siow Mong Goh
  48. Yong Liu
  49. Daniel Shu Wei Ting

Contributions

Y.B. and Y.Z. designed the methodology, algorithms and computational framework, curated the data and developed, trained and implemented the model. Y.B. and H.C. performed the formal analyses and validation and prepared the visualizations. Y.B. and H.C. wrote the original draft, with contributions from A.T., Y.K., L.G., Z.L.T., D.S.J.T., B.T.S. and C.S.N. A.T., J.Y., A.S.Y.H., C.W.N.Q., A.B., K.F., R.K., C.L.C., L.T.-E.C. and N.T.A.T. evaluated the model’s reasoning outputs. A.T., Y.K., Z.L.T., D.S.J.T., B.T.S., C.S.N., N.R., T.Y.W., D.M.M., Y.L. and D.S.W.T. reviewed and edited the manuscript. D.S.W.T. and Y.L. conceptualized the study, administered the project, supervised the research, provided resources and acquired funding. All authors reviewed and approved the final manuscript.

Corresponding author

Correspondence to Daniel Shu Wei Ting.

Ethics declarations

Competing interests

D.S.W.T. holds a patent on a deep learning system for the detection of retinal diseases (10201706186V) and a computer-implemented method for training an image classifier using weakly annotated training data (10201901083Y) and stocks at EyRIS, Singapore and aISIGHT. A.J.T. has received funding from HealthSense to support research concerning biomedical applications of large language models. All other authors declare no conflicts of interest.

Peer review

Peer review information

Nature Medicine thanks Jansen Seheult and the other, anonymous, reviewer(s) for their contribution to the peer review of this work. Primary Handling Editors: Mattia Andreoletti and Joao Monteiro, in collaboration with the Nature Medicine team.

Additional information

Publisher’s note Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Extended data

Extended Data Table 1 Detailed results on the performance EVLF-FM for medical image disease diagnostics in internal validation against generalist and specialist models in internal validation

Full size table

Extended Data Table 2 Detailed results on the performance EVLF-FM for medical image disease diagnostics in internal validation against generalist and specialist models in internal validation in the MedMNIST datasets

Full size table

Extended Data Table 3 Per-class comparison on chest X-ray classification across internal (NIH ChestX-ray14, VinDr-CXR) and external (AIMx) evaluation sets

Full size table

Extended Data Table 4 Detailed results on the performance of EVLF-FM for visual question-answering (VQA) tasks in internal validation against other VLMs

Full size table

Extended Data Table 5 Detailed results on the performance of EVLF-FM for medical visual grounding tasks in internal validation against other VLMs

Full size table

Extended Data Table 6 Detailed results on the performance of EVLF-FM for medical image disease diagnostics and medical visual grounding in zero and few shot external validation against generalist VLMs

Full size table

Extended Data Table 7 Detailed results of the human evaluation of EVLF-FM thinking outputs using the S.C.O.R.E. framework

Full size table

Extended Data Table 8 Detailed results on the performance of EVLF-FM for medical image disease diagnosis stratified by age and sex, compared with generalist VLMs

Full size table

Extended Data Table 9 Detailed results on ablation study of EVLF-FM performance on training strategy components and varying proportions of RFT data

Full size table

Extended Data Table 10 Detailed results on the performance of EVLF-FM for medical image disease diagnostics tasks using different prompts in internal validation (MedMNIST)

Full size table

Rights and permissions

Open Access This article is licensed under a Creative Commons Attribution-NonCommercial-NoDerivatives 4.0 International License, which permits any non-commercial use, sharing, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if you modified the licensed material. You do not have permission under this licence to share adapted material derived from this article or parts of it. The images or other third party material in this article are included in the article’s Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article’s Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit http://creativecommons.org/licenses/by-nc-nd/4.0/.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Bai, Y., Cheng, H., Zhou, Y. et al. A clinical multimodal vision language foundation model with fine-grain explainability. Nat Med (2026). https://doi.org/10.1038/s41591-026-04655-5

Download citation

  • Received: 29 September 2025

  • Accepted: 17 August 2026

  • Published: 29 October 2026

  • Version of record: 29 October 2026

  • DOI: https://doi.org/10.1038/s41591-026-04655-5

来源:Nature 健康护理主题 · nature.com