跳到正文
原文
Nature 健康护理主题·· 25 天前精选AI 评分71

大型语言模型在药物推荐中的产品名称偏好

Large language models show product name convergence in drug recommendations

AI 导读

研究评估了五种大型语言模型在177,120个临床案例中的药物推荐,发现产品名称高度集中,多数情况下仅推荐少数几种药物,这可能增加药物成本并影响市场公平竞争。

推荐理由

研究量化了大型语言模型在药物推荐中的产品名称偏好,揭示了潜在的经济和市场影响。

正文

Main

Large language models (LLMs) are rapidly transforming healthcare delivery, clinical workflows and public access to health information. In 2024, 31.5% of US nonfederal acute care hospitals were already using LLMs integrated into electronic health record systems, with adoption estimated to reach about half of hospitals by the end of 2025 (ref. 1). On the patient side, 21.5% of adults reported using LLMs for online health information searches2. LLMs’ capabilities across clinical tasks3,4,5,6,7,8,9,10 may also influence how individuals interpret health information and consider care options.

Drug recommendation is a promising application for LLMs11. Across diverse settings, ranging from emergency to outpatient care, selecting an appropriate drug requires integrating diagnoses, comorbidities, contraindications and patient preferences from unstructured clinical text. This complexity increases the risk of medication errors, which are estimated to cost US$42 billion globally each year12. LLMs are well suited for this challenge given their advanced ability to interpret and integrate high-dimensional clinical contexts, and recent studies have demonstrated their potential in drug recommendation13 and error prevention14,15.

However, as LLMs increasingly influence medication-related decision-making, concerns about algorithmic bias have intensified. Because LLMs are trained on vast amounts of human-generated text, they risk perpetuating societal stereotypes16. Prior work has largely focused on sociodemographic biases, such as disparities in triage17,18 or imaging recommendations19 based on race, gender or income status. Yet a critical but under-examined dimension of algorithmic bias concerns which specific product names are recommended. Unlike clinicians and health systems that operate under guidelines prioritizing cost-effective drugs, LLMs trained on general web data may exhibit a popularity bias, preferentially recommending widely publicized drugs over clinically equivalent alternatives20,21. Such convergence may have large-scale public health, operational and economic consequences, but the degree of convergence remains poorly quantified.

In this study, we evaluate product name recommendations from five LLMs across 177,120 clinical vignettes covering 123 common disease–drug pairs, asking each model to recommend US Food and Drug Administration (FDA)-approved products containing the active ingredient specified in each vignette. Our primary aim is to characterize the distribution of product name recommendations within each pair. We therefore measure the degree of convergence in selected products, their divergence from real-world prescribing patterns and projected acquisition cost implications. We also assess associations between patient sociodemographic factors and regulatory-nonconcordant outputs and identify product characteristics associated with recommendation frequency. These analyses establish a benchmark for evaluating regulatory concordance and commercial neutrality in LLM-generated product name recommendations.

Results

Overview of LLM product name recommendations

We evaluated product name recommendations from five LLMs using 177,120 synthetic clinical vignettes generated by combining 123 curated disease–drug pairs with 1,440 prespecified sociodemographic profiles (Fig. 1 and Supplementary Table 1). Each pair linked a common disease to a prespecified active ingredient, and the profiles represented prespecified combinations of sociodemographic factors (see Methods for details). The five LLMs were GPT-5.4, Claude Sonnet 4.6, Gemini 3.1 Flash, Llama 4 and MedGemma. Using a standardized prompt, each LLM was asked to recommend up to three FDA-approved drugs by their product names. The five models generated a total of 883,403 responses with at least one drug recommendation (range per model: 175,197–177,119; Supplementary Fig. 1 and Supplementary Table 2). Among these recommendations, FDA-approved drugs corresponding to the specified active ingredient were identified and aggregated by disease–drug pair. The most frequently recommended products per LLM for 123 disease–drug pairs are shown in Supplementary Table 3.

Fig. 1: Study design and analytical workflow.

A total of 177,120 synthetic clinical vignettes were generated by combining 123 disease–drug pairs with 1,440 sociodemographic profiles. Each vignette was submitted to five LLMs using a standardized prompt requesting up to three FDA-approved product names, listed in order of preference. Outputs were reviewed by four physicians to standardize product names and assess concordance with FDA approval status. The curated outputs were then evaluated for product name convergence across vignettes and models, divergence from real-world prescription patterns, regulatory nonconcordance across LLMs and sociodemographic descriptors, and product characteristics associated with recommendation frequency.

Product name convergence across vignettes and models

For each of the 123 disease–drug pairs, we aggregated first-choice recommendations across all patient vignettes and calculated the share accounted for by the most frequently recommended product. Across pairs and models, this product name convergence was pronounced, with the top product capturing a mean of 97.5% of all first-choice recommendations (Supplementary Table 4). The distribution of these shares is shown in Fig. 2a. This concentration is notable, given that multiple FDA-approved products were available for most disease–drug pairs (mean, 5.50; Supplementary Table 5), indicating that LLMs typically recommend only a narrow subset of clinically available options. When first-choice recommendations were pooled across all five LLMs, the same pattern persisted: the most frequently recommended product accounted for >90% of recommendations in 78.0% of pairs (96 of 123) (Fig. 2b). The identity of the dominant drug was also highly consistent across models (Fleiss’ κ = 0.895, 95% confidence interval (CI): 0.855–0.931; Supplementary Fig. 2), indicating almost perfect agreement22. Concordance with the pooled dominant drug was highest for Claude Sonnet 4.6 (96.6%) and lowest for MedGemma (90.6%) (Supplementary Fig. 3).

Fig. 2: Consistency of LLM product recommendations across vignettes and models.

a, Distribution of the proportion accounted for by the product most frequently recommended by each LLM across 123 disease–drug pairs. Data are visualized as violin plots (kernel density estimates, clipped at 100%) and box plots. Boxes indicate the interquartile range, centre lines denote medians, and whiskers extend to data points within 1.5 times the interquartile range from the quartiles. LLMs consistently recommended the same product regardless of patient characteristics, with that product representing a mean proportion of 97.5% of first-choice recommendations across disease–drug pairs and models. b, Aggregated proportion of the most frequently recommended product across all LLMs. Recommended drugs were largely consistent across models, with the most frequently recommended drug accounting for over 90% of recommendations in 96 of 123 disease–drug pairs.

Comparison of LLM and real-world product distributions

Next, we compared LLM-recommended product distributions with real-world prescription distributions derived from the Medical Expenditure Panel Survey (MEPS), a nationally representative US survey that includes data on prescribed medicine use23. Of the 123 disease–drug pairs, 100 had corresponding products represented in the 2019–2023 MEPS data and were included in subsequent analyses. To visualize these discrepancies, we compared the proportion of each product appearing in either LLM recommendations or MEPS prescriptions (Fig. 3a; shown separately for each disease–drug pair in Supplementary Fig. 4). Many drugs deviated substantially, being either over-represented or under-represented in LLM recommendations relative to MEPS prescriptions. We further examined the extent to which the most frequently prescribed drugs in MEPS aligned with the LLM recommendations (Fig. 3b). On average, the most frequently prescribed drug in MEPS matched the LLMs’ first-choice recommendation in only 29.7% of cases (s.d., 4.5%) and was among the LLMs’ top three recommendations in 58.8% of cases (s.d., 14.2%). When weighted by the number of prescriptions in MEPS, first-choice concordance decreased to 17.0% (s.d., 8.7%; Supplementary Fig. 5).

Fig. 3: Comparison of LLM-recommended drugs with real-world prescription distributions.

a, Comparison of product proportions in LLM recommendations versus real-world prescription data from MEPS. Each point represents a product; points above the dashed diagonal line indicate over-representation by LLMs, while points below indicate under-representation. b, Concordance with the most frequently prescribed drug in real-world practice. Stacked bars show the cumulative percentage of cases where the top-ranked drug in MEPS appeared within the LLM’s top one (dark), two (medium) or three (light) recommendations. c, TVD between the LLM and MEPS product distributions across disease–drug pairs. Higher TVD values indicate greater disparity between LLM-recommended product name distributions and MEPS prescription distributions.

We then quantified the overall distributional divergence using total variation distance (TVD; Fig. 3c), a measure of distributional difference ranging from 0 (identical distributions) to 1 (no overlap). Across all models, TVD values were concentrated near 1 for many disease–drug pairs, reflecting large differences between the two distributions. Median TVD values across the five models ranged from 0.93 to 0.98 (Supplementary Table 6), suggesting very limited overlap between the LLM-generated and MEPS-derived distributions.

Acquisition cost projections based on LLM recommendations

We next projected differences in drug acquisition costs between LLM-recommended and MEPS-derived product distributions using National Average Drug Acquisition Cost (NADAC) data, which reflect average acquisition costs based on invoice prices from retail community pharmacies24. Of the 101 unique active ingredients represented by the 123 disease–drug pairs, 18 were excluded because the corresponding MEPS prescription records were unavailable, and 32 were excluded because complete NADAC pricing was unavailable for all relevant products; the remaining 51 active ingredients were included in the cost analysis (Supplementary Table 7). We compared the mean cost per unit under LLM-recommended versus MEPS-derived product distributions and projected the total acquisition cost difference by multiplying these per-unit differences by the annual MEPS prescription volumes (Supplementary Table 8). Figure 4 shows the relationship between TVD, per-unit cost differences and prescription volume. Of the 51 active ingredients, 38 (74.5%) showed an increase in cost, indicating that LLM-recommended prescribing patterns are more expensive than those observed in real-world practice. When summed across the 51 active ingredients, these cost differences resulted in an estimated annual increase of US$55.6 million, equivalent to a 1,019% increase in drug acquisition costs.

Fig. 4: Economic impact of LLM-recommended drug distributions compared with real-world prescription distributions.

The bubble plot illustrates the relationship between distributional divergence, measured using TVD, and the estimated acquisition cost difference per unit between LLM recommendations and MEPS-derived real-world prescribing patterns. Each bubble represents an active ingredient, with its size proportional to the annual prescription volume in MEPS. Costs were calculated using NADAC data. Positive y values indicate that LLM recommendations favour more expensive drugs compared with current clinical practice. Active ingredient labels are shown for active ingredients with per-unit cost differences or MEPS prescription volumes above the 90th percentile.

Regulatory nonconcordance across models and sociodemographic descriptors

We further evaluated product-level regulatory nonconcordance in the generated drug names using the criteria described in Methods. After physician curation, these recommendations were categorized into five types (Supplementary Fig. 6). Across models, most regulatory-nonconcordant outputs referred to previously FDA-approved products that were no longer marketed (91.69–99.59%). Summed across these categories, Claude Sonnet 4.6 exhibited the highest regulatory nonconcordance rate (14.7%), whereas other models showed rates ranging from 12.4% (GPT-5.4) to 13.6% (MedGemma). Multivariable analysis using generalized estimating equations (GEEs) confirmed that both LLM type and patient sociodemographic factors were significant predictors of regulatory-nonconcordant output (Table 1). Relative to the reference model (Claude Sonnet 4.6), the odds were highest for MedGemma (odds ratio (OR) 0.64, 95% CI: 0.61–0.68), followed by Gemini 3.1 Flash (OR 0.63, 95% CI: 0.61–0.65). Several sociodemographic characteristics were also associated with increased odds of regulatory-nonconcordant recommendations, including being Black (OR 1.64, 95% CI: 1.58–1.69), Asian (OR 1.16, 95% CI: 1.12–1.19) or lesbian (OR 1.06, 95% CI: 1.02–1.11). Together, these results indicate that regulatory-nonconcordant outputs are driven not only by LLM characteristics but also by explicit sociodemographic factors in the vignettes, consistent with prior studies reporting sociodemographic group-based variation in LLM-generated clinical assessments and recommendations18,19.

Table 1 Multivariable analysis of LLMs and sociodemographic factors associated with regulatory-nonconcordant drug recommendations

Full size table

Drug characteristics associated with recommendation frequency

We further analysed which factors were associated with the LLMs’ preference for particular products. In the multivariable analysis (Table 2), originator (innovator) status was the strongest predictor of recommendation frequency (OR 20.5, 95% CI: 19.7–21.3), indicating a substantial preference for originator products over available generics. Other factors independently associated with higher recommendation frequency included a greater number of Google searches (OR 1.70, 95% CI: 1.67–1.73) and the presence of a dedicated Wikipedia page (OR 3.00, 95% CI: 2.79–3.22). In contrast, later FDA approval order (OR 0.76, 95% CI: 0.76–0.77), greater manufacturer promotional spending (OR 0.97, 95% CI: 0.97–0.97), higher real-world prescription volume in MEPS (OR 0.87, 95% CI: 0.86–0.87) and longer drug names (OR 0.88, 95% CI: 0.88–0.89) were negatively associated with being recommended.

Table 2 Multivariable analysis of factors associated with LLM drug output frequency

Full size table

Prompt sensitivity analysis

To evaluate the sensitivity of product name convergence to prompt instructions, we tested three alternative prompt settings: comprehensive product listing, lowest-cost preference and generic preference (see Methods for details). Product name recommendations remained highly concentrated within each prompt setting. The most frequently recommended product accounted for a mean of 93.1%, 94.9% and 97.6% of first-choice recommendations in the comprehensive product listing, lowest-cost preference and generic preference settings, respectively (Supplementary Fig. 7). However, the top-recommended product was not always the same as that under the original prompt setting. It changed in 45.2%, 62.6% and 77.9% of comparisons in the same three settings, respectively. Among comparisons in which the top-recommended product changed, shifts to a generic product were the most common pattern, occurring in 78.0%, 83.1% and 85.4% of cases in the same three settings, respectively (Supplementary Fig. 8). These findings indicate that alternative prompt instructions can shift LLM product name recommendations towards the instructed preference.

As an exploratory analysis, we also evaluated the disease-only setting, in which the active ingredient was omitted from the vignette (see Methods for details). We examined the product names recommended in response to these prompts and mapped each recommended product to its corresponding active ingredient, allowing us to evaluate convergence separately at both the active ingredient and product name levels. Compared with the original setting, convergence was attenuated but remained present at both levels, with the most frequently recommended active ingredient and product name accounting for means of 83.8% and 79.6% of outputs, respectively (Supplementary Fig. 9a,b). When first-choice recommendations were pooled across all five LLMs, the most frequently recommended active ingredient and product name accounted for >90% of recommendations in 26.9% (7 of 26) and 11.5% (3 of 26) of diseases, respectively (Supplementary Fig. 9c,d). Together, these findings suggest that recommendations remained concentrated within individual models but varied across models for the same disease.

Discussion

In this large-scale, multi-model evaluation of drug recommendations, we analysed approximately 883,000 LLM outputs from systematically constructed clinical vignettes across 123 disease–drug pairs. Across pairs and models, LLMs showed pronounced product name convergence, with a single product accounting for a mean of 97.5% of first-choice recommendations and near-perfect intermodel agreement (Fleiss’ κ = 0.895, 95% CI: 0.855–0.931). These recommendation distributions diverged substantially from MEPS-derived prescribing patterns (model-specific median TVD, 0.93–0.98) and—among 51 active ingredients with complete NADAC pricing—were associated with higher projected acquisition costs in 74.5% (38 of 51), corresponding to an estimated 1,019% annual cost increase. We further quantified regulatory-nonconcordant outputs, revealing notable variations in nonconcordance rates across models and patient demographics, and identified the underlying factors driving these recommendation patterns. Overall, this study systematically quantifies product name convergence in LLM-generated medication recommendations.

Our analysis revealed striking uniformity in LLM drug recommendations that persisted across diverse patient profiles and model architectures. This algorithmic homogeneity stands in sharp contrast to the heterogeneity observed in real-world clinical practice. Notably, despite a mean of 5.50 FDA-approved products per disease–drug pair, the most frequently recommended product accounted for >90% of first-choice recommendations in 78.0% of pairs (96 of 123). These associations are consistent with a popularity-related bias, with models more frequently recommending products that have higher public visibility, originator (innovator) status and earlier FDA approval. The implications of such algorithmic determinism extend beyond model-intrinsic issues; widespread deployment without calibration could narrow the range of product name options presented in medication-related workflows. By entrenching the dominance of commercially visible drugs, LLMs risk creating a feedback loop that marginalizes alternative FDA-approved products, thereby reducing their visibility in medication-related workflows.

LLM-guided prescribing could impose a substantial economic burden on healthcare systems and patients. In our NADAC-based analysis of 51 active ingredients with complete pricing data, 74.5% (38 of 51) showed higher costs under the LLM-recommended distributions, resulting in an estimated 1,019% annual cost increase. We found that models systematically favour originator products, and this preference increases estimated acquisition costs. This tendency contrasts with real-world guidelines that prioritize generic drugs to control costs25. Because LLMs may inherently overlook insurance constraints or cost signals, their adoption risks undermining cost-control policies and burdening patients with limited insurance coverage. In addition, repeatedly recommending the same drugs may reduce the visibility of other appropriate options, which could indirectly affect fair competition in the drug market26. Ultimately, these economic and market distortions could threaten the long-term sustainability of healthcare delivery.

Our prompt sensitivity analyses suggest that prompt instructions can influence product name convergence. Although convergence persisted, generic products often became the most frequently recommended options when prompts prioritized generic or lowest-cost items. These findings suggest that prompt design may guide LLM-generated medication recommendations towards product names that better reflect prescribing considerations such as generic use and cost. Therefore, prompt design may complement regulatory validation and human oversight in clinical workflows.

Furthermore, despite the near uniformity of first-choice drug recommendations, the regulatory-nonconcordant recommendations were found to be associated with patient descriptors. Consistent with prior evidence that sociodemographic factors can influence LLM outputs in medical question answering27 and triage18, we observed that vignettes specifying Black patients were associated with increased odds of regulatory-nonconcordant recommendations. Although such outputs were not predominant and do not necessarily imply clinical inappropriateness, certain categories—such as fabricated product names or ingredient mismatches—could complicate verification and warrant further evaluation. Accordingly, high intermodel agreement in product name recommendations should not be interpreted as evidence of regulatory concordance. Deployment evaluations should report regulatory concordance metrics stratified by sociodemographic descriptors and should separately assess clinical safety and patient outcomes.

As LLMs are increasingly deployed to streamline clinical workflows and support medical decision-making, preventing the amplification of medication-related biases is essential. Our findings underscore the need for continuous evaluation strategies that address both product name convergence—which was broadly observed across models and patient attributes—and variation in regulatory-nonconcordant outputs associated with model type and patient sociodemographic factors. These safeguards are important because deployment risks vary by context: tools integrated into electronic health records may widen disparities if clinicians fall into automation bias28, while direct patient use lacks the expert oversight needed to catch errors. Mitigation strategies, such as fine-tuning on reliable medical datasets29 and incorporating human-in-the-loop verification30, are therefore indispensable. Ensuring transparency regarding training data and modelling methodologies will be critical to building trust and safety in clinical artificial intelligence (AI).

Our study has several limitations. First, although we used systematically constructed vignettes to limit potential confounders and establish a baseline assessment of product name convergence, these synthetic cases may not fully capture the complexity of real-world prescribing and should not be interpreted as direct estimates of real-world prescribing behaviour31. Our analysis evaluated common disease–drug pairs encountered in routine clinical practice but did not examine rare diseases or more complex clinical scenarios. Second, although we evaluated the three alternative prompts, we did not assess their task-specific performance. Such analyses warrant dedicated study, and a high concentration under these prompts may reflect successful instruction following rather than persistent bias. Third, MEPS-derived distributions are subject to sampling variability and reflect clinical, payer, pharmacy and patient-level factors affecting dispensed product use. Differences from MEPS, therefore, do not necessarily indicate errors in the LLM outputs. Fourth, the cost analysis was limited to active ingredients with complete MEPS prescription records and NADAC pricing data for all relevant products. Because differences between included and excluded active ingredients could not be fully assessed, this restriction may introduce selection bias and limit generalizability. The cost estimates should be interpreted as simulated acquisition cost projections based on prespecified normalization assumptions for each active ingredient. In addition, our analysis evaluated LLM outputs at the recommendation stage and did not model downstream prescribing, pharmacy substitution, dispensing or purchasing processes that may affect the product ultimately received by the patient. Future research should explore more detailed clinical scenarios, systematically evaluate how different prompts affect model outputs and whether they achieve their intended goals, and investigate human–AI interactions. Examining biases and conducting rigorous evaluations against clinical guidelines will be essential for fully understanding the specific harms of algorithmic bias.

In conclusion, while LLMs hold promise for supporting medication-related decisions, this study identifies pronounced product name convergence, which has potential implications for drug costs and pharmaceutical markets. Continued evaluation of commercial neutrality, regulatory concordance and sociodemographic variation will help guide the responsible clinical integration of LLMs.

Methods

Study design

This model evaluation study quantitatively assessed product name convergence in drug recommendations generated by LLMs across a broad range of clinical conditions. We systematically constructed 177,120 synthetic clinical vignettes based on 123 curated disease–drug pairs representing common diseases encountered in routine clinical practice (Supplementary Table 1). Each vignette describes a concise patient scenario that includes sociodemographic factors and a brief description of the target disease. The recommended product name was the primary unit of analysis, allowing for comparisons among FDA-approved products containing the prespecified active ingredient.

Each LLM’s output was evaluated based on four outcomes: (1) intramodel and intermodel variation in first-choice drug recommendations across vignette scenarios; (2) distributional comparison between LLM-recommended product distributions and MEPS-derived real-world prescription distributions; (3) proportion and type of regulatory-nonconcordant responses across model characteristics and sociodemographic factors; and (4) factors that may explain why certain drugs are more frequently recommended by LLMs. The LLM evaluation used synthetic clinical vignettes and did not involve human participants. Ethical considerations regarding the experimental design and conduct were discussed, and this study did not require institutional review board approval. This study followed the TRIPOD-LLM32 checklist.

Vignette development

To systematically evaluate LLM behaviour under standardized and controlled conditions, we generated synthetic clinical vignettes based on 123 curated disease–drug pairs representing common conditions and commonly prescribed active ingredients. The pairs were selected through a four-step process: (1) identification of prevalent diseases by two physicians using the CDC National Health Interview Survey and MEPS23; (2) selection of guideline-recommended and commonly prescribed active ingredients; (3) verification against the FDA Orange Book33 and Purple Book34 to ensure regulatory status; and (4) clinical validation by two additional physicians, who refined an initial list of 198 pairs down to the final 123 through consensus. This final list is shown in Supplementary Table 1.

Each vignette described a patient using five sociodemographic factors: age, sex, race or ethnicity, LGBTQ+ identity and income status (Supplementary Methods 1). To capture both diversity and neutrality, each factor included an ‘unspecified’ option (that is, the attribute was omitted from the prompt). Specifically, the five sociodemographic factors were defined as follows:

  • Age: unspecified, young, middle-aged or elderly

  • Sex: unspecified, male or female

  • Race or ethnicity: unspecified, Asian, Black, Hispanic or white

  • LGBTQ+ identity: unspecified, lesbian, gay, bisexual, transgender, queer, nonbinary or heterosexual

  • Income status: unspecified, low-income or high-income

These combinations yielded 1,440 unique sociodemographic profiles (4 age groups × 3 sex groups × 5 race or ethnicity groups × 8 LGBTQ+ identity groups × 3 income status groups) for each disease–drug pair. Based on these diseases and their associated sociodemographic combinations, a total of 177,120 vignettes (123 disease–drug pairs × 1,440 sociodemographic factors) were generated.

LLM specifications

Five LLMs (three closed-source and two open-source) were used. The closed-source models were GPT-5.4 (gpt-5.4-2026-03-05) from OpenAI, Claude Sonnet 4.6 (claude-sonnet-4-6) from Anthropic, and Gemini 3.1 Flash (gemini-3.1-flash-lite-preview) from Google. The open-source models were Llama 4 (Llama-4-Maverick-17B-128E-Instruct-GGUF, a 3.5-bit quantized model) and MedGemma (medgemma-27b-it). For the temperature parameter, all models were set to 0. All other hyperparameters were set to their default values. Implementation details (API libraries, execution timeframe and hardware setup) are described in Supplementary Methods 2.

Inference settings

Each vignette was input into the LLM along with a system prompt (Supplementary Methods 3). To identify a prompt configuration that most reliably satisfied the task instructions and output format requirements, we performed a five-version stepwise prompt-ablation study using MedGemma (Supplementary Methods 4). Each model was instructed to list up to three FDA-approved drugs by their product names. The recommended names were then evaluated against entries in the FDA Orange Book (https://www.accessdata.fda.gov/scripts/cder/ob/index.cfm) and Purple Book (https://purplebooksearch.fda.gov/), which together cover approved drug products, including both originator (brand-name) products and generics. This design enabled us to quantify the concentration of product name recommendations across FDA-approved products and to perform consistent product-level mapping and cost attribution. In cases where no drug was recommended, vignettes were rerun with identical settings up to ten times before being excluded.

LLM output curation

To address inconsistencies in drug names across LLM outputs—such as variations in capitalization, spelling or expression—manual curation was performed by four physicians. Each physician examined whether the generated drug names corresponded to any FDA-approved drugs listed in the FDA Orange Book33 and Purple Book34 databases (accessed 12 July 2025). Regulatory concordance was assessed with reference to the specified active ingredient and its FDA-approved indications. Outputs that were not regulatory-concordant or could not be verified were categorized as follows:

  1. 1.

    The drug does not exist.

  2. 2.

    The drug exists but is not labelled for an approved indication of the specified active ingredient.

  3. 3.

    The drug contains an active ingredient different from the one specified in the prompt.

  4. 4.

    The drug has the same active ingredient but is not listed in the databases (for example, a brand-name over-the-counter product or a drug approved outside the USA).

  5. 5.

    The drug is either discontinued or predates FDA regulations.

This classification reflects regulatory appropriateness at the product level and does not imply clinical inappropriateness, as off-label use may be clinically appropriate. In cases where the classification was uncertain, the final decision was made by consensus among the physicians.

Drug-related metadata collection

To compare LLM-recommended drugs with real-world prescription distributions, we used data from MEPS23. MEPS is a nationally representative survey of the US civilian noninstitutionalized population administered by the Agency for Healthcare Research and Quality, with a sample drawn from the previous year’s National Health Interview Survey. The MEPS Household Component collects the number of dispensed prescriptions from an annual sample of 9,000–15,000 households (22,000–35,000 individuals)23. To construct a robust distribution of prescribed medications and mitigate the impact of annual fluctuations, we aggregated prescription data over a 5-year period from 2019 to 2023.

To estimate the economic impact of LLM-recommended drugs, pricing information was obtained from the NADAC database24, which is administered by the Centers for Medicare & Medicaid Services. The NADAC represents the average acquisition cost based on a nationwide survey of retail pharmacies. NADAC data as of the end of 2024 were applied to the drugs mentioned in the LLM outputs and the MEPS dataset.

To investigate potential factors associated with the frequency of drug recommendations across five LLMs, drug-level metadata were collected from publicly available sources. For each drug, the following information was obtained: (1) whether the drug is an originator (innovator) drug or a generic (bioequivalent) drug; (2) its FDA approval order (with earlier approval corresponding to a lower value); (3) the number of times the drug name was searched using Google; (4) the presence or absence of a dedicated Wikipedia page; (5) the number of prescriptions in the MEPS datasets; (6) the total promotional spending by the drug’s manufacturer, as reported in the Open Payments system (https://openpaymentsdata.cms.gov/); and (7) the character length of the drug name. Google search counts and Wikipedia page information were collected through their respective APIs on 10 December 2025. Promotional spending data were averaged over the 7 years from 2018 to 2024.

Prompt sensitivity analysis

To evaluate how prompt instructions affected LLM outputs, we performed additional prompt sensitivity analyses using multiple alternative prompt settings and compared the resulting product name recommendations with those generated under the original prompt setting. The detailed prompts are provided in Supplementary Methods 5. These settings include comprehensive product listing, lowest-cost preference and generic preference prompts. The comprehensive product listing prompt requested all equivalent FDA-approved generic and brand-name products for the specified active ingredient. The lowest-cost preference and generic preference prompts retained the ranked-list format but added instructions to prioritize the names of the lowest-cost products or generic names, respectively, when therapeutically equivalent FDA-approved products were available. Each alternative prompt setting was applied to the same 177,120 synthetic clinical vignettes used in the primary analysis, and the product name outputs were evaluated using the same curation procedures.

We also performed a disease-only prompt analysis to evaluate LLM recommendations when the active ingredient was not provided. For this analysis, we removed the active ingredient sentence from the original vignette format, leaving only the disease diagnosis and sociodemographic descriptors. The 123 disease–drug pairs included in the primary analysis represented 26 unique diseases. Disease-only vignettes were generated by combining each of these 26 diseases with the same 1,440 sociodemographic profiles, yielding 37,440 input vignettes. The same five LLMs and inference settings were used for this analysis.

Statistical analysis

First-choice drug recommendations were aggregated across all vignettes for each disease–drug pair, and the proportion accounted for by the most frequently recommended product was calculated. The same proportion was also calculated after pooling first-choice recommendations across the five LLMs. To quantify intermodel agreement on the identity of this dominant product, we computed Fleiss’ κ statistic35.

For each disease–drug pair, the proportion of LLM recommendations that matched the most frequently prescribed drug in MEPS was calculated. To further quantify distributional differences, the TVD was calculated36,37.

Economic impact was quantified as the difference in the mean unit cost between the LLM-recommended and real-world MEPS drug distributions. For each active ingredient, the mean unit cost was calculated by weighting NADAC prices according to the drug proportions in MEPS and LLM recommendations. This per-unit cost differential was then multiplied by the annual prescription volume (number of prescriptions × median units per prescription) recorded in MEPS to estimate the annual cost difference. Because products containing the same active ingredient may differ in the route of administration, dose unit and duration of action per unit, we standardized the per-unit cost for each active ingredient according to predefined rules. Specifically, for each active ingredient, we defined the most commonly used route of administration as the reference route and used it when a product had multiple possible routes of administration. Next, when multiple NADAC prices corresponding to different dosing units were available for the same active ingredient, we used the price corresponding to the smallest comparable dose unit to avoid overestimating costs. Finally, we defined a reference duration of action. When a product’s duration of action per unit differed from this reference, we adjusted the selected NADAC price by multiplying it by the ratio of the reference duration to the product’s duration of action per unit. This adjustment reflected the number of units needed to cover the reference duration. The cost analysis was conducted at the active ingredient level and included only active ingredients with corresponding MEPS prescription records and NADAC pricing for all products identified in either the LLM-generated or MEPS-derived distribution.

GEEs were used to assess the associations of model characteristics and sociodemographic factors with regulatory nonconcordance in LLM-recommended drugs, accounting for repeated responses across vignettes. GEEs were also used to evaluate associations between LLM drug output frequency and drug-related metadata. Google search counts, the number of prescriptions and manufacturers’ promotional spending were log-transformed before analysis to improve normality and linearity.

Statistical analyses were performed using Python (v3.9.12) and the statsmodels library (v0.13.2) for GEE modelling, and data were visualized using matplotlib (v3.9.4). Statistical significance was defined as P < 0.05.

Reporting summary

Further information on research design is available in the Nature Research Reporting Summary linked to this article.

Data availability

All synthetic vignettes are publicly available on the Hugging Face Hub (https://huggingface.co/datasets/Medical-AI-Lab/llm-drug-bias). The open-source model weights are publicly available in the following repositories on the Hugging Face Hub: MedGemma 27B (https://huggingface.co/google/medgemma-27b-it) and Llama 4-Maverick (https://huggingface.co/unsloth/Llama-4-Maverick-17B-128E-Instruct-GGUF/tree/main/UD-Q3_K_XL).

References

  1. Everson, J., Nong, P. & Richwine, C. Uptake of generative AI integrated with electronic health records in US hospitals. JAMA Netw. Open 8, e2549463 (2025).

    Article  PubMed  PubMed Central  Google Scholar 

  2. Ayo-Ajibola, O., Davis, R. J., Lin, M. E., Riddell, J. & Kravitz, R. L. Characterizing the adoption and experiences of users of artificial intelligence-generated health information in the United States: cross-sectional questionnaire study. J. Med. Internet Res. 26, e55138 (2024).

    Article  PubMed  PubMed Central  Google Scholar 

  3. Singhal, K. et al. Large language models encode clinical knowledge. Nature 620, 172–180 (2023).

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  4. Zakka, C. et al. Almanac—retrieval-augmented language models for clinical medicine. NEJM AI 1, AIoa2300068 (2024).

    Article  Google Scholar 

  5. Van Veen, D. et al. Adapted large language models can outperform medical experts in clinical text summarization. Nat. Med. 30, 1134–1142 (2024).

    Article  PubMed  PubMed Central  Google Scholar 

  6. Kumar, A., Wang, H., Muir, K. W., Mishra, V. & Engelhard, M. A cross-sectional study of GPT-4-based plain language translation of clinical notes to improve patient comprehension of disease course and management. NEJM AI 2, AIoa2400402 (2025).

    Article  Google Scholar 

  7. McDuff, D. et al. Towards accurate differential diagnosis with large language models. Nature 642, 451–457 (2025).

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  8. McCoy, L. G. et al. Assessment of large language models in clinical reasoning: a novel benchmarking study. NEJM AI 2, AIdbp2500120 (2025).

    Article  Google Scholar 

  9. Levine, D. M. et al. The diagnostic and triage accuracy of the GPT-3 artificial intelligence model: an observational study. Lancet Digit. Health 6, e555–e561 (2024).

    Article  CAS  PubMed  Google Scholar 

  10. Gaber, F. et al. Evaluating large language model workflows in clinical decision support for triage and referral and diagnosis. NPJ Digit. Med. 8, 263 (2025).

    Article  PubMed  PubMed Central  Google Scholar 

  11. Ong, J. C. L. et al. A scoping review on generative AI and large language models in mitigating medication related harm. NPJ Digit. Med. 8, 182 (2025).

    Article  PubMed  PubMed Central  Google Scholar 

  12. World Health Organization. WHO launches global effort to halve medication-related errors in 5 years. WHO https://www.who.int/news/item/29-03-2017-who-launches-global-effort-to-halve-medication-related-errors-in-5-years (2017).

  13. Lammert, J. et al. Expert-guided large language models for clinical decision support in precision oncology. JCO Precis. Oncol. 8, e2400478 (2024).

    Article  PubMed  Google Scholar 

  14. Pais, C. et al. Large language models for preventing medication direction errors in online pharmacies. Nat. Med. 30, 1574–1582 (2024).

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  15. Ong, J. C. L. et al. Large language model as clinical decision support system augments medication safety in 16 clinical specialties. Cell Rep. Med. 6, 102323 (2025).

    Article  PubMed  PubMed Central  Google Scholar 

  16. Celi, L. A. et al. Sources of bias in artificial intelligence that perpetuate healthcare disparities—a global review. PLOS Digit. Health 1, e0000022 (2022).

    Article  PubMed  PubMed Central  Google Scholar 

  17. Guerra-Adames, A., Avalos-Fernandez, M., Doremus, O., Gil-Jardiné, C. & Lagarde, E. Uncovering judgment biases in emergency triage: a public health approach based on large language models. In Proc. 4th Machine Learning for Health Symposium (eds Hegselmann, S. et al.) Vol. 259, 420–439 (PMLR, 2025).

  18. Omar, M. et al. Sociodemographic biases in medical decision making by large language models. Nat. Med. 31, 1873–1881 (2025).

    Article  CAS  PubMed  Google Scholar 

  19. Zack, T. et al. Assessing the potential of GPT-4 to perpetuate racial and gender biases in health care: a model evaluation study. Lancet Digit. Health 6, e12–e22 (2024).

    Article  CAS  PubMed  Google Scholar 

  20. Bender, E. M., Gebru, T., McMillan-Major, A. & Shmitchell, S. On the dangers of stochastic parrots: can language models be too big? In Proc. 2021 ACM Conference on Fairness, Accountability, and Transparency (eds Elish, M. C., Isaac, W. & Zemel, R. S.) https://doi.org/10.1145/3442188.3445922 (ACM, 2021).

  21. Kandpal, N., Deng, H., Roberts, A., Wallace, E. & Raffel, C. Large language models struggle to learn long-tail knowledge. In Proc. 40th International Conference on Machine Learning (eds Krause, A. et al.) Vol. 202, 15696–15707 (PMLR, 2023).

  22. Landis, J. R. & Koch, G. G. The measurement of observer agreement for categorical data. Biometrics 33, 159–174 (1977).

    Article  CAS  PubMed  Google Scholar 

  23. Cohen, J. W., Cohen, S. B. & Banthin, J. S. The Medical Expenditure Panel Survey: a national information resource to support healthcare cost research and inform policy and practice. Med. Care 47, S44–S50 (2009).

    Article  PubMed  Google Scholar 

  24. National Average Drug Acquisition Cost. Medicaid.gov https://www.medicaid.gov/medicaid/nadac (2026).

  25. ASPE Staff. Expanding the use of generic drugs. ASPE http://aspe.hhs.gov/reports/expanding-use-generic-drugs-0 (2010).

  26. Fletcher, A., Ormosi, P. L. & Savani, R. Recommender systems and supplier competition on platforms. J. Compet. Law Econ. 19, 397–426 (2023).

    Article  Google Scholar 

  27. Pfohl, S. R. et al. A toolbox for surfacing health equity harms and biases in large language models. Nat. Med. 30, 3590–3600 (2024).

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  28. Goddard, K., Roudsari, A. & Wyatt, J. C. Automation bias: a systematic review of frequency, effect mediators, and mitigators. J. Am. Med. Inform. Assoc. 19, 121–127 (2012).

    Article  PubMed  Google Scholar 

  29. Lu, H. et al. Toward fair medical advice: addressing and mitigating bias in large language model-based healthcare applications. Artif. Intell. Med. 168, 103216 (2025).

    Article  PubMed  Google Scholar 

  30. Strong, J., Men, Q. & Noble, J. A. Trustworthy and practical AI for healthcare: a guided deferral system with large language models. In Proc. 39th AAAI Conference on Artificial Intelligence (eds Walsh, T., Shah, J. & Kolter, Z.) Vol. 39, 28413–28421 (AAAI Press, 2025).

  31. Evans, S. C. et al. Vignette methodologies for studying clinicians’ decision-making: validity, utility, and application in ICD-11 field studies. Int. J. Clin. Health Psychol. 15, 160–170 (2015).

    Article  PubMed  PubMed Central  Google Scholar 

  32. Gallifant, J. et al. The TRIPOD-LLM reporting guideline for studies using large language models. Nat. Med. 31, 60–69 (2025).

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  33. Center for Drug Evaluation and Research. Approved drug products with therapeutic equivalence evaluations. US Food and Drug Administration https://www.fda.gov/drugs/drug-approvals-and-databases/approved-drug-products-therapeutic-equivalence-evaluations-orange-book (2026).

  34. Purple Book: Database of Licensed Biological Products https://purplebooksearch.fda.gov/ (US Food and Drug Administration, 12 July 2025).

  35. Fleiss, J. L. Measuring nominal scale agreement among many raters. Psychol. Bull. 76, 378–382 (1971).

    Article  Google Scholar 

  36. Törnqvist, M. et al. A text-to-tabular approach to generate synthetic patient data using LLMs. In 2025 IEEE 13th International Conference on Healthcare Informatics (ICHI) 18–29 (IEEE, 2025).

  37. Levin, D. A., Peres, Y. & Wilmer, E. L. Markov Chains and Mixing Times (American Mathematical Society, 2009).

Download references

Funding

This study was supported by JST PRESTO (JPMJPR2521).

Author information

Authors and Affiliations

  1. Department of Imaging Physics, The University of Texas MD Anderson Cancer Center, Houston, TX, USA

    Junya Sato & Jia Wu

  2. Department of Artificial Intelligence in Diagnostic Radiology, The University of Osaka Graduate School of Medicine, Osaka, Japan

    Junya Sato

  3. Department of Artificial Intelligence, Graduate School of Medicine, Osaka Metropolitan University, Osaka, Japan

    Shannon Walston, Hirotaka Takita, Yasuhito Mitsuyama, Kenichi Saito & Daiju Ueda

Authors

  1. Junya Sato
  2. Shannon Walston
  3. Hirotaka Takita
  4. Yasuhito Mitsuyama
  5. Kenichi Saito
  6. Jia Wu
  7. Daiju Ueda

Contributions

J.S., K.S. and D.U. designed the study. J.S. and D.U. conducted the statistical analyses. J.S. prepared the draft. J.S., H.T., Y.M. and D.U. verified the drug names. S.W. provided English proofreading. K.S., H.T., Y.M., S.W. and J.W. revised the manuscript. All authors reviewed and approved the final paper.

Corresponding author

Correspondence to Daiju Ueda.

Ethics declarations

Competing interests

The authors declare no competing interests.

Peer review

Peer review information

Nature Health thanks Robert Califf, Pierre Dubois and the other, anonymous, reviewer(s) for their contribution to the peer review of this work. Primary Handling Editor: Lorenzo Righetto, in collaboration with the Nature Health team.

Additional information

Publisher’s note Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Open Access This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made. The images or other third party material in this article are included in the article’s Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article’s Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit http://creativecommons.org/licenses/by/4.0/.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Sato, J., Walston, S., Takita, H. et al. Large language models show product name convergence in drug recommendations. Nat. Health (2026). https://doi.org/10.1038/s44360-026-00205-y

Download citation

  • Received: 10 April 2026

  • Accepted: 10 September 2026

  • Published: 26 October 2026

  • Version of record: 26 October 2026

  • DOI: https://doi.org/10.1038/s44360-026-00205-y

来源:Nature 健康护理主题 · nature.com