少数族裔患者亚组中预测模型性能指标的悖论性膨胀:上游偏差的作用
Paradoxical inflation of prediction model performance measures in minority patient subgroups: the role of upstream bias
研究表明,上游偏差可能导致预测模型在受影响的少数族裔亚组中产生悖论性的高性能指标。在评估用于预测术后持续性阿片类药物使用(POU)的模型时,Black 患者的 AUC 为 0.93,显著高于 White 患者的 0.86。研究团队通过子群匹配分析发现,这种性能差异源于基线特征中的系统性偏差,而非模型本身对少数族裔更具优势。
Introduction
Several national and international AI guidelines emphasize the need for predictive health AI models to be fair1,2,3,4, and many major AI vendors provide tools to assess fairness5,6,7,8. The need to consider fairness is especially critical when these models are used to allocate healthcare resources because biased allocation of resources can worsen inequities9. A model is considered fair when it performs comparably across subgroups and equitably allocates resources based on patients’ risk10,11,12,13. As a consequence, efforts to evaluate and mitigate bias often focus on first measuring performance across subgroups, followed by assessing how resource allocations would vary by group8. This approach has been successful in uncovering specific types of unfairness, including models used to prioritize patients for chronic care management or pain treatment9,14.
Recently, there has been growing recognition of the role of upstream bias in producing unfair models. Upstream biases are those introduced into data before AI models are trained15. These can result from differences in care, differences in technology, underrepresentation, and inaccurate or inconsistent measurement within subgroups of people16,17. Correcting these deficiencies can improve model performance within subgroups18, highlighting the importance of recognizing and correcting upstream biases.
Upstream biases are generally expected to produce worse performance among affected subgroups, particularly when those groups are underrepresented in training data or when systemic disparities distort the data generating process19,20. Given the typically limited sample sizes of Black and other minority patients in many healthcare datasets, this often leads to reduced model performance for these groups20. It was therefore surprising when a recent opioid prediction model developed by our team using the Michigan Genomics Initiative (MGI) perioperative cohort, hereafter the MGI model, produced better performance in Black patients by a substantial margin21. In this prior work, the area under the receiver operating characteristic curve (AUC) for a model predicting persistent opioid use (POU) after surgery was observed to be 0.93 (95% CI 0.88–0.97) in Black patients and 0.86 (95% CI 0.85–0.88) in White patients (p = 0.004). While the higher AUC in Black patients would typically be reassuring, this result was discordant with prior evidence, which has shown that Black patients are less likely than White patients to receive optimal treatment, including worse access to opioids22,23.
In this study, we sought to understand whether upstream bias could counterintuitively produce better performance in the affected subgroup. We did this by first evaluating the robustness of our prior work’s findings by replicating them using another published opioid prediction model, hereafter the refit claims model24, which predicts new POU after surgery among opioid-naive patients. Upon finding a similar result (higher AUC in Black patients), we used two types of subgroup matching to explore the relationship between baseline characteristics and observed model performance. This approach provides a systematic way to evaluate the presence and meaning of observed performance differences across subgroups. We hypothesized that observed performance differences were linked to differences in patient characteristics among subgroups in our study cohort, an indicator of potential upstream bias.
Results
This section first describes the study cohorts used to develop each model, then evaluates whether the previously reported higher AUC in Black patients replicates in an independently developed model. We then compare baseline characteristics between Black and White patients to identify factors that may explain this performance difference, and use subgroup matching to test whether these characteristics account for the observed gap.
Study cohort
For the MGI model, the cohort included 24,040 patients, with a median age of 55, with 53% female, 48% opioid-naive prior to surgery, and predominantly non-Hispanic White (89%). (Table 1A). 20% of patients experienced postoperative POU. Patients without POU were more likely to be opioid-naive at surgery, report lower non-surgical site pain, and have lower chronic pain (Table 1B). We observed key differences between the MGI model’s training and test datasets: the training data had a higher proportion of females (53% vs 51%, p = 0.006) and more patients who experienced POU (21% vs 19%, p < 0.001) (Supplement 2). While we observed statistically significant differences between the training and test data for age, gender, Elixhauser Comorbidity Index, average pain, PROMIS physical score, chronic pain, hydrocodone/acetaminophen scripts, and tramadol scripts, these differences were small and not clinically meaningful.
For the refit claims model, the cohort comprised 11,508 opioid-naive patients. This cohort was predominantly non-Hispanic White (89%) and largely nonsmokers (58%) (Table 2A). The 12% of patients who experienced new POU had higher Charlson Comorbidity Index scores, greater preoperative benzodiazepine use, and comorbidities such as fibromyalgia and osteoarthritis (Table 2B). Compared to the test set, training set patients had lower Charlson Comorbidity Index scores and were less likely to have osteoarthritis (Supplement 3).
Robustness of performance differences in racial subgroups
MGI model
As previously reported21, the MGI model had an AUC of 0.86 (0.85–0.88) in predicting POU overall, with an AUC of 0.86 (0.85–0.88) in White patients and 0.93 (0.88–0.97) in Black patients (Fig. 1). The MGI model performed better in Black patients with test AUC 0.066 higher than in the White population (p = 0.004).
Error bars capture the 95% CI of the AUC. p-values are for difference in AUC between White and Black patients based on the DeLong test
Refit claims model
In the opioid-naive subpopulation, the refit claims model had an AUC of 0.59 (0.56–0.63) overall, 0.59 (0.55–0.62) in White patients, and 0.71 (0.59–0.82) in Black patients (Fig. 1). While the refit claims model had a higher point estimate in Black patients with a testing AUC 0.121 higher than in the White population, this finding was not statistically significant (p = 0.075), potentially attributable to the reduced sample size of opioid-naive patients (testing data N = 2749).
Baseline characteristics in Black and White patients
To identify factors that may be contributing to the performance differences observed above, we compared baseline characteristics between Black and White patients. Figure 2 displays covariate distributions for the original White cohort, Black patients, and the matched White cohorts (1B-1W and 1B-2W; the matching procedure is described below). In the overall MGI cohort, Black patients were younger than the original White cohort (Fig. 2A). On the measure of average pain, Black patients had a wider distribution of reported pain compared to that for White patients (Fig. 2B). On the Elixhauser Comorbidity Index, White patients had a higher skew, compared to the relatively smoother distribution for Black patients (Fig. 2C). NaNDA (National Neighborhood Data Archive)-defined neighborhood disadvantage levels were more homogenous among White patients, and the overall neighborhood disadvantage score was lower in White patients as compared to Black patients (Fig. 2D, Supplement 4).
(A) Age in years. (B) Average pain in week not at surgical site (0 to 10 scale). (C) Combined Elixhauser Comorbidity Index. (D) Census tract-level neighborhood disadvantage from the National Neighborhood Data Archive (NaNDA), scored from 0 to 1. Each violin shows the distribution of the variable in one group, with width proportional to the density of patients at each value. White Original refers to unmatched White patients; Black refers to Black patients; White (1B-1W) and White (1B-2W) refer to White patients matched to Black patients at 1 Black:1 White and 1 Black:2 White, respectively.
Subgroup matching
If subgroup model performance differences stemmed from observable baseline characteristics, matching patients across subgroups could potentially attenuate these differences, suggestive of upstream bias. We used subgroup matching to characterize the role of patient characteristics in contributing to model performance differences. Each Black patient was matched to their nearest one or two White patients without duplication (1B-1W and 1B-2W matching) based on the consolidated set of variables used in the MGI and refit claims variables, excluding the outcome variables (Supplement 5). Covariate balance between White and Black patients improved after matching in the 1B-1W cohort, with matched White patients more closely resembling Black patients than unmatched White patients did (Fig. 3 and Supplement 6, 7). Improved covariate balance after matching was similarly observed in the 1B-2W cohort (Supplement 8).
A Depicts the absolute standardized mean differences and the KS statistics for select covariates in the MGI model between Black and White patients in the data before (red circles) and after matching (blue triangles). B Depicts the absolute standardized mean differences and the KS statistics for select covariates in the Refit Claims model between Black and White patients in the data before (red circles) and after matching (blue triangles).
Test-only matching: train on full training data, evaluate on matched testing data
The MGI model performance on matched White patients increased from an original AUC of 0.86 to a matched AUC of 0.88 (0.83–0.94) and 0.89 (0.86–0.93) in the 1B-1W and 1B-2W matched ratio test cohorts, respectively (Fig. 4). The matched AUC in Black patients was similar at 0.926 (0.88–0.97). The differences between Black and White patients were no longer statistically significant for both the (1B-1W) and (1B-2W) cohorts. A similar finding was observed for the refit claims model.
Error bars capture the 95% CI of the AUC. p-values are for differences in AUC between White and Black patients based on the DeLong test.
For the MGI model, the density distribution of predicted probabilities of the outcome is more similar between Black patients and the matched cohorts compared to that of the original White patient cohort in the MGI model (Fig. 5A). A similar convergence was observed for the refit claims model (Fig. 5B). Given that the outcome was not included in the matching step, this suggests a matching success not just in terms of the patient observables, but also their behavior patterns of interest.
(A) Michigan Genomics Initiative (MGI) model: predicted probability of persistent opioid use (POU) after surgery. (B) Refi t claims model: predicted probability of new POU after surgery among opioid-naive patients. In both panels, each curve shows the density of predicted probabilities for one group: White Original refers to unmatched White patients (black dashed line); Black refers to Black patients (black solid line); White (1B-1W) refers to White patients matched to Black patients at 1 Black:1 White (red dashed line); and White (1B-2W) refers to White patients matched at 1 Black:2 White (blue dashed line).
Full matching with training: trained on matched training data, evaluate on matched testing data
Reduced performance differences between the White and Black patients in the matched cohorts were also seen when models were re-trained using matched training data (Supplement 9).
Other performance metrics
To characterize model fairness more broadly than AUC alone allows, we examined a broader set of performance metrics across subgroups. Supplement 10 presents performance metrics for the MGI model across White patients, matched White patients, and Black patients, including sensitivity, specificity, positive predictive value (PPV), negative predictive value (NPV), Matthews Correlation Coefficient (MCC), and proportion of positives. Five of the six additional metrics showed a pattern consistent with the AUC findings, with more favorable performance in Black patients than unmatched White patients that attenuated after matching. Specificity was the exception, being lower in Black patients, possibly reflecting their higher outcome prevalence (26% vs 20%), which shifts predicted scores upward and produces more false positives at a fixed threshold. After subgroup matching, metrics in the matched White cohorts were more closely aligned with those in the Black cohort, supporting the role of observed characteristics in contributing to model performance differences.
Discussion
In this retrospective study using the longitudinal MGI cohort, we found that two opioid prediction models exhibited a higher point estimate for AUC among Black patients as compared to White patients, a finding that was statistically significant for one of the two models. A comparison of baseline patient characteristics revealed greater heterogeneity among Black patients in terms of predicted risk, comorbidities, and socioeconomic disadvantage measures. Matched subgroups of White patients in the testing data had a higher AUC as compared to the full cohort of White patients, with an AUC similar to that of Black patients. The matched White cohort had a more similar distribution of predicted outcome probabilities to Black patients as compared to the full cohort of White patients. This pattern held whether models were trained on the full training data or the matched training data. Attenuation of differences between Black patients and White patients following matching was also observed among the other performance metrics.
Our findings suggest that differences in baseline patient characteristics between Black and White patients are sufficient to explain much of the observed AUC disparity, though we cannot fully disentangle the contributions of the data distribution from those of the model itself. Because AUC is a measure of discriminative performance, the presence of extremely high- and low-risk patients in a dataset can inflate the AUC, as was seen in Black patients in our study. Conversely, in groups with more uniformly low- or average-risk patients (i.e., a more homogenous risk population), the model cannot readily differentiate events from non-events, resulting in a lower AUC. This phenomenon has been recognized in the criminal justice system25, where higher AUCs have been observed in female subgroups due to greater variation in types of crimes and sentencing. In our study, Black patients were more heterogeneous with respect to both their baseline characteristics and distribution of model probabilities.
Heterogeneity is not inherently problematic, as long as the heterogeneous characteristics for a population are not also important predictors of risk. In our study, heterogeneous baseline characteristics were linked to the patients’ risk. White patients matched on baseline characteristics exhibited more similar risk profiles and observed AUCs to Black patients. Thus, belonging to a uniformly lower-risk group (lower AUC and lower persistent opioid use) in our study would be preferable to being in a varying-risk group (higher AUC). When AUCs for two subgroups become similar after matching, upstream biases related to the representativeness of patient subgroups should be considered. On the other hand, if a model’s performance does not change in response to matching, this suggests the gap may not be fully explained by the observed covariates alone. For example, a recent study comparing sex differences in acute kidney injury models found that matching did not reduce subgroup differences26. Unmeasured confounders, such as income or insurance type, may reflect unequal access to care that matching on available variables cannot fully capture. This could be evaluated by comparing patients who received care to the broader eligible population to identify factors linked to differential access. Predictors may also serve as proxies for race in ways that do not generalize across groups, or outcome labels may be measured differentially due to variation in clinical documentation, scrutiny, or missingness. Where the source of a persistent gap is better understood, interventions such as targeted matching on relevant access variables, fairness constraints, feature auditing, or subgroup-specific models may be appropriate, though each involves tradeoffs with overall performance.
Our findings have major implications for the evaluation of fairness in predictive AI models. In May of 2024, the US Department of Health and Human Services Office for Civil Rights published a final rule—one that remains in effect—requiring that organizations using AI “must make reasonable efforts to mitigate the risk of discrimination.”27 This has been followed by state legislation aiming to promote fairness in AI28. Measuring subgroup performance is a common aspect of such efforts. The “AUC gap,” or the largest difference between subgroups for AUC, has been proposed as a way of checking if a model is fair29, with the implication that lower AUCs are worse. On the contrary, we demonstrate that lower AUCs may be driven in part by homogeneity of population risk profiles and may even be preferable. If AUC is to be used to compare population subgroups, we recommend that any stratified reporting of AUC across subgroups should be accompanied by a matched analysis.
Our study has limitations. First, the matching approach may fail if there are insufficient patients in the majority group to match with the minority group, and because training a model generally requires a larger sample size than validation, 1:1 matching on the training dataset may result in a sample too small for model training30. Second, while our matching framework provides evidence that covariate differences contribute to observed AUC disparities, it cannot rule out alternative explanations such as unmeasured confounding, label noise, or differential measurement error; our findings are therefore best interpreted as consistent with upstream influences rather than as definitive proof of upstream bias. Third, our primary indicator of structural disadvantage, neighborhood-level socioeconomic disadvantage (NaNDA), is a census tract-level proxy; individual-level measures of care access, diagnostic coding quality, and label reliability were not available, and the coarse spatial resolution of these measures limits the specificity of upstream inference. Fourth, this retrospective study was conducted using data from a large academic medical center, potentially limiting its generalizability. Finally, AUC differences within subgroups represent a small portion of a holistic evaluation of model fairness.
Despite these limitations, our study has important implications for assessing model fairness and bias. Our findings challenge the notion that a high AUC in a demographic minority is inherently reassuring regarding model performance. Instead, a higher AUC may reflect greater upstream bias and increased risk for adverse outcomes among minorities, an issue that can be investigated using our matching framework. Our study underscores the importance of measuring and accounting for demographic characteristics to ensure fairness across subgroups.
Methods
Study cohort
We used data from the Michigan Genomics Initiative (MGI), a longitudinal biorepository at the University of Michigan started in 201221,31,32. The repository enrolls perioperative patients, integrates electronic health records (EHRs) with self-reported questionnaire data, and links to Michigan’s Prescription Drug Monitoring Program (PDMP) via a data-sharing agreement, enabling high-quality tracking of opioid fills before and after surgery33. Our cohort included pre-surgical patients enrolled in the MGI cohort who underwent index surgery within 90 days of enrollment between January 1, 2015, and July 3, 2018. We excluded patients living outside Michigan as PDMP data were limited to the state of Michigan. This study was approved by the Institutional Review Board at Michigan Medicine (Ann Arbor, MI; HUM00149878), and the need for consent was waived by the same board. All methods were carried out in accordance with relevant guidelines and regulations governing human subjects research.
Data source
Our primary data source for this study is MGI data33. It is supplemented with census tract and zip code-level education variables (e.g., proportion with less than a high school diploma) from the National Neighborhood Data Archive (NaNDA), a publicly available resource comprising physical, economic, demographic, and social environmental variables at various spatial scales34.
Statistical analyses
We evaluated two published persistent opioid use prediction models using this data: one originally trained using MGI data21 by Singh et al., and another that was originally trained on claims data24 by Hur et al.
MGI model
Since the Singh et al. model was originally developed using the MGI cohort, we refer to it as the MGI model and used the original published coefficients.
Refit claims model
The original model in Hur et al. was developed with opioid-naive adults aged 18–64 using Optum’s Clinformatics Data Mart, which contains insurance claims data. Our EHR-based MGI data lacked some variables that were present in the Hur et al. prediction model. Specifically, the latter used individual patient education levels which were extensively missing and inconsistent in the EHR data. We replaced this with the NaNDA education attainment distribution from the patient’s census tract (eg: proportions with or without a high school diploma, and with a Bachelor’s degree or higher)34. Although the Hur et al. model was originally trained using claims data, we refit it using MGI data for the purposes of this study due to differences in the original claims data and our MGI EHR-based data that prevented our using it in its original form. We refer to the Hur et al. model, trained with these variables as the refit claims model.
Additionally, since the Hur et al. model applies only to opioid-naive patients, we restricted training and evaluation of the refit claims model to an opioid-naive cohort, defined as patients who had not filled an opioid prescription within 365 to 31 days prior to their index surgery. Results from our MGI and refit claims models are not directly comparable, but test the robustness of our findings. Because the refit claims model is applied to a population that differs substantially from the original Hur et al. development cohort in demographics, comorbidity profiles, and outcome definition, some attenuation of absolute performance is expected, as is common when prediction models are transported across populations35.
Outcomes
The two models we evaluated were trained to predict different outcomes. The MGI model predicts persistent opioid use (POU) after surgery, whereas the refit claims model predicts new POU after surgery among opioid-naive patients. POU after surgery was defined similarly to Howard et al.36, identified if a patient filled: (1) an opioid prescription twice during the perioperative period (30 days before surgical admission to 3 days after discharge) and/or (2) an opioid prescription during post-discharge days 4–90, and (3) an opioid prescription during post-discharge days 91–180. New POU was defined as having filled an opioid prescription during post-discharge days 91–180.
MGI model
The logistic regression model to predict POU included key demographic, clinical, and patient-reported predictors and is detailed in Supplement 1A. This model was originally developed using the same data source as in this study.
Refit claims model
The XGBoost model predicting new POU included demographic and clinical predictors, detailed in Supplement 1B.
Model training and testing cohorts
Data were split by index surgery date, with surgeries before July 5th, 2017 assigned to the training cohort and those after to the testing cohort. The MGI model used the same coefficients as the original manuscript21. The refit claims model was refit (i.e., trained) using the training cohort, tuned using 5-fold cross-validation with a maximum of 5000 iterations, and then evaluated in the testing cohort. The final model parameters were: eta = 0.3, max depth = 1, lambda = 5. We conducted subgroup analyses for all models to evaluate model performance separately in Black and White subgroups.
Differences between subgroups
We compared the following observed characteristics between Black and White patient cohorts: age, average pain in week not at surgical site, the Elixhauser Comorbidity Index, and socioeconomic status (SES), and NaNDA’s neighborhood disadvantage variable (ranging from 0 to 1). The disadvantage variable averages four census indicators: proportion of female-headed families with children, proportion of households receiving public assistance income or food stamps; proportion of families below the federal poverty level; proportion of population aged 16+ unemployed. The MGI model does not include SES-related predictors, while the refit claims model uses education level as a predictor.
Subgroup matching
We applied two matching-based approaches to characterize the role of patient characteristics in contributing to model performance differences:
-
1)
Test-only Matching (primary analysis): applying the original models (trained on the full, unmatched training data) to the matched testing cohorts, and
-
2)
Full Matching with Training (secondary analysis): re-training models on matched training data and evaluating them on the matched testing cohorts.
The first approach evaluates upstream bias within the testing set alone, whereas the second approach evaluates upstream bias within the training and testing sets. While the second approach allows for a more comprehensive evaluation of upstream bias, it is limited by the need to re-train models and a reduction in sample size in the training data due to matching.
In the first approach, models trained on the full unmatched training data were evaluated on a subset of the testing data in which White patients were matched to Black patients based on observed characteristics. The MatchIt37 package was used to apply a case-control propensity score matching approach with a logistic regression generalized linear model. Each Black patient was matched to the nearest White patient by propensity score, without replacement, using Black:White ratios of 1:1 and 1:2. Matching included all predictor variables from both the MGI and refit claims models, along with gender. For education, we used only the percentage of the census tract population with less than a high school degree to avoid collinearity among the education variables. Match quality was assessed using “love plots” to compare matched and unmatched cohort characteristics38. In the second approach, the models were retrained on matched training data and similarly evaluated on matched testing cohorts. AUC by subgroup was compared using the DeLong test39.
Other performance metrics
In addition to subgroup AUC comparisons, we evaluated a broader set of performance metrics across subgroups: Sensitivity, Specificity, PPV, NPV, the MCC, and proportion of positives. We calculated these metrics for Black patients, unmatched White patients, and matched White patients. This evaluation allowed us to characterize model fairness more broadly.
Missing data
For socioeconomic level variables such as education, we imputed missing values using the mean of other patients within the same three-digit zip code. Remaining patient-level missing values were imputed using bootstrapped aggregated (bagged) trees. For missing data on lamotrigine fill quantities, however, we imputed values for patients with a prescription but unknown quantities using the median of patients with known non-zero lamotrigine quantities.
Data availability
We used data from the Michigan Genomics Initiative (MGI), a longitudinal biorepository at the University of Michigan started in 2012. Patient confidentiality prevents the sharing of these data publicly. However, the data underlying the study’s results are available from the Michigan Genomics Initiative (https://sites.lsa.umich.edu/aidhi-data-details/genetic-data-2/) for researchers who meet the criteria for confidential data access. Researchers may request access via the form on the MGI website (https://aidhi.umich.edu/mgi-community/#mgi-data), subject to eligibility and approval.
Code availability
The R code used to conduct the analyses in this study is available from the corresponding author upon reasonable request.
References
Tabassi, E. Artificial Intelligence Risk Management Framework (AI RMF 1.0). https://www.nist.gov/publications/artificial-intelligence-risk-management-framework-ai-rmf-10?trk=public_post_comment-text (2023) https://doi.org/10.6028/nist.ai.100-1
Blueprint For Trustworthy AI Implementation Guidance And Assurance For Healthcare. (2023).
Collins, G. S. et al. TRIPOD+AI statement: updated guidance for reporting clinical prediction models that use regression or machine learning methods. BMJ 385, e078378 (2024).
Moons, K. G. M. et al. PROBAST+AI: an updated quality, risk of bias, and applicability assessment tool for prediction models using regression or artificial intelligence methods. BMJ 388, e082505 (2025).
Lutz, R. Fairlearn: assessing and improving fairness of AI systems. J. Mach. Learn. Res. abs/2303.16626, 1–8 (2023).
Wexler, J. et al. The what-if tool: interactive probing of machine learning models. IEEE Trans. Vis. Comput. Graph. 26, 56–65 (2020).
User Guide. https://epic-open-source.github.io/seismometer/user_guide/index.html#fairness-audit.
Obermeyer, Z. et al. Algorithmic bias playbook. Center for Applied AI at Chicago Booth (2021).
Obermeyer, Z., Powers, B., Vogeli, C. & Mullainathan, S. Dissecting racial bias in an algorithm used to manage the health of populations. Science 366, 447–453 (2019).
Barocas, S., Hardt, M. & Narayanan, A. Fairness and Machine Learning: Limitations and Opportunities (MIT Press, 2023).
Paulus, J. K. & Kent, D. M. Predictably unequal: understanding and addressing concerns that algorithmic clinical prediction may increase health disparities. NPJ Digit. Med. 3, 99 (2020).
Mitchell, S., Potash, E., Barocas, S., D’Amour, A. & Lum, K. Algorithmic fairness: choices, assumptions, and definitions. Annu. Rev. Stat. Appl. 8, 141–163 (2020).
Courtland, R. Bias detectives: the researchers striving to make algorithms fair. Nature 558, 357–360 (2018).
Pierson, E., Cutler, D. M., Leskovec, J., Mullainathan, S. & Obermeyer, Z. An algorithmic approach to reducing unexplained pain disparities in underserved populations. Nat. Med. 27, 136–140 (2021).
Karumbaiah, S. C. The Upstream Sources of Bias: Investigating Theory, Design, and Methods Shaping Adaptive Learning Systems (University of Pennsylvania, 2022).
Suresh, H. & Guttag, J. A Framework for understanding sources of harm throughout the machine learning life cycle. in Equity and Access in Algorithms, Mechanisms, and Optimization 1–9 (Association for Computing Machinery, 2021).
Gichoya, J. W. et al. AI recognition of patient race in medical imaging: a modelling study. Lancet Digit Health 4, e406–e414 (2022).
Zink, A., Luan, H. & Chen, I. Y. Access to care affects electronic health record reliability and AI-driven disease prediction. Nat. Health 1, 316-325 (2026).
Chen, I. Y., Joshi, S. & Ghassemi, M. Treating health disparities with artificial intelligence. Nat. Med. 26, 16–17 (2020).
Seyyed-Kalantari, L., Zhang, H., McDermott, M. B. A., Chen, I. Y. & Ghassemi, M. Underdiagnosis bias of artificial intelligence algorithms applied to chest radiographs in under-served patient populations. Nat. Med. 27, 2176–2182 (2021).
Singh, K. et al. Predicting persistent opioid use after surgery using electronic health record and patient-reported data. Surgery 172, 241–248 (2022).
Morden, N. E., Chyn, D., Wood, A. & Meara, E. Racial inequality in prescription opioid receipt - role of individual health systems. N. Engl. J. Med. 385, 342–351 (2021).
Lee, P. et al. Racial and ethnic disparities in the management of acute pain in US emergency departments: Meta-analysis and systematic review. Am. J. Emerg. Med. 37, 1770–1777 (2019).
Hur, J. et al. Predicting postoperative opioid use with machine learning and insurance claims in opioid-naïve patients. Am. J. Surg. 222, 659–665 (2021).
Howard, P. D. The effect of sample heterogeneity and risk categorization on area under the curve predictive validity metrics. Crim. Justice Behav. 44, 103–120 (2017).
Cao, J. et al. Generalizability of an acute kidney injury prediction model across health systems. Nat. Mach. Intell. 4, 1121–1129 (2022).
Mello, M. M. & Roberts, J. L. Antidiscrimination law meets artificial intelligence-new requirements for health care organizations and insurers. JAMA Health Forum 5, e243397 (2024).
California State Senate Approves the California Artificial Intelligence Bill of Rights. California State Senator Steve Padilla https://sd18.senate.ca.gov/news/california-state-senate-approves-california-artificial-intelligence-bill-rights.
Lee, J., Brooks, C., Yu, R. & Kizilcec, R. Fairness Hub Technical Briefs: AUC Gap. (2023).
Riley, R. D. et al. Calculating the sample size required for developing a clinical prediction model. BMJ 368, m441 (2020).
Spector-Bagdady, K. et al. Respecting autonomy and enabling diversity: the effect of eligibility and enrollment on research data demographics. Health Aff. 40, 1892–1899 (2021).
Zawistowski, M. et al. The Michigan Genomics Initiative: a biobank linking genotypes and electronic clinical records in Michigan Medicine patients. Cell Genom. 3, 100257 (2023).
Fernandez, A. C. et al. Identifying persistent opioid use after surgery: the reliability of pharmacy dispensation databases. Ann. Surg. https://doi.org/10.1097/SLA.0000000000005529 (2022).
National Neighborhood Data Archive (NaNDA). https://www.icpsr.umich.edu/web/ICPSR/series/1920.
Steyerberg, E. W. Clinical Prediction Models (Springer Nature, 2020).
Howard, R. et al. New persistent opioid use after inguinal hernia repair. Ann. Surg. https://doi.org/10.1097/sla.0000000000004560 (2020).
Ho, D., Imai, K., King, G. & Stuart, E. A. MatchIt: Nonparametric Preprocessing for Parametric Causal Inference. J. Stat. Softw. 42, 1–28 (2011).
Greifer, N. Covariate balance tables and plots: A guide to the cobalt package. https://cran.microsoft.com/snapshot/2017-11-08/web/packages/cobalt/vignettes/cobalt_basic_use.html (2017).
DeLong, E. R., DeLong, D. M. & Clarke-Pearson, D. L. Comparing the areas under two or more correlated receiver operating characteristic curves: a nonparametric approach. Biometrics 44, 837–845 (1988).
Acknowledgements
This study was supported by a Propelling Original Data Science (PODS) Grant from the Michigan Institute for Data and AI in Society (MIDAS) at the University of Michigan. The funder had no role in the design of the study; the collection, analysis, or interpretation of the data; the writing of the manuscript; or the decision to submit the manuscript for publication.
Author information
Author notes
These authors contributed equally: Karandeep Singh, Anne C. Fernandez.
Authors and Affiliations
Department of Health Management & Policy, School of Public Health, University of Michigan, Ann Arbor, MI, USA
Rahul Ladhania
Department of Biostatistics, School of Public Health, University of Michigan, Ann Arbor, MI, USA
Rahul Ladhania
Institute for Healthcare Policy and Innovation, University of Michigan, Ann Arbor, MI, USA
Rahul Ladhania & Chad M. Brummett
Department of Bioinformatics, University of Michigan Medical School, Ann Arbor, MI, USA
Allister Z. Ho
School of Information, University of Michigan, Ann Arbor, MI, USA
Paramveer Dhillon
Division of Pain Medicine, Department of Anesthesiology, University of Michigan Medical School, Ann Arbor, MI, USA
Chad M. Brummett
Opioid Research Institute, University of Michigan, Ann Arbor, MI, USA
Chad M. Brummett
Joan and Irwin Jacobs Center for Health Innovation, University of California, San Diego, CA, USA
Karandeep Singh
Division of Biomedical Informatics, Department of Medicine, University of California, San Diego, CA, USA
Karandeep Singh
Department of Psychiatry, University of Michigan Medical School, Ann Arbor, MI, USA
Anne C. Fernandez
Authors
- Rahul Ladhania
- Allister Z. Ho
- Paramveer Dhillon
- Chad M. Brummett
- Karandeep Singh
- Anne C. Fernandez
Contributions
R.L. co-wrote the main manuscript text, co-led the empirical analysis, was responsible for the interpretation of data for the work and providing revisions to the work for important intellectual content; and gives final approval of the version to be published; and agrees to be accountable for all aspects of the work in ensuring that questions related to the accuracy or integrity of any part of the work are appropriately investigated and resolved. A.Z.H. was responsible for executing the statistical analyses, the interpretation of data for the work and providing revisions to the work for important intellectual content; and gives final approval of the version to be published; and agrees to be accountable for all aspects of the work in ensuring that questions related to the accuracy or integrity of any part of the work are appropriately investigated and resolved. P.D. guided the empirical analysis, was responsible for the interpretation of data for the work and providing revisions to the work for important intellectual content; and gives final approval of the version to be published; and agrees to be accountable for all aspects of the work in ensuring that questions related to the accuracy or integrity of any part of the work are appropriately investigated and resolved. C.M.B. provided domain expertise and guidance, was responsible for the interpretation of data for the work and providing revisions to the work for important intellectual content; and gives final approval of the version to be published; and agrees to be accountable for all aspects of the work in ensuring that questions related to the accuracy or integrity of any part of the work are appropriately investigated and resolved. K.S. co-wrote the main manuscript text, guided the empirical analysis, was responsible for the interpretation of data for the work and providing revisions to the work for important intellectual content; and gives final approval of the version to be published; and agrees to be accountable for all aspects of the work in ensuring that questions related to the accuracy or integrity of any part of the work are appropriately investigated and resolved. A.C.F. co-wrote the main manuscript text, was responsible for the interpretation of data for the work and providing revisions to the work for important intellectual content; and gives final approval of the version to be published; and agrees to be accountable for all aspects of the work in ensuring that questions related to the accuracy or integrity of any part of the work are appropriately investigated and resolved.
Corresponding authors
Correspondence to Rahul Ladhania or Karandeep Singh.
Ethics declarations
Competing interests
C.M.B. serves as a consultant for Heron Therapeutics, a biotech company that produces a nonopioid analgesic for acute pain, and he served as a one-time consultant for Vertex Pharmaceuticals, Alosa Health, and the Benter Foundation. In addition, C.M.B. provides expert medicolegal testimony unrelated to this research. K.S. reports consulting fees from Google for participation on Google’s Consumer Health Advisory Panel. The other authors do not have a competing interest.
Additional information
Publisher’s note Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
Rights and permissions
Open Access This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made. The images or other third party material in this article are included in the article's Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article's Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit http://creativecommons.org/licenses/by/4.0/.
About this article
Cite this article
Ladhania, R., Ho, A.Z., Dhillon, P. et al. Paradoxical inflation of prediction model performance measures in minority patient subgroups: the role of upstream bias. npj Health Syst. 3, 99 (2026). https://doi.org/10.1038/s44401-026-00158-0
Received: 25 September 2025
Accepted: 10 September 2026
Published: 21 October 2026
Version of record: 21 October 2026
DOI: https://doi.org/10.1038/s44401-026-00158-0
来源:Nature 健康护理主题 · nature.com