跳到正文
原文
Nature 健康护理主题·· 2026-10-02AI 评分45

OpenEvidence 在多学科炎症病临床委员会中的可行性与专家评估

Feasibility and expert evaluation of a medical large language model in an interdisciplinary clinical board

AI 导读

一项前瞻性观察研究评估了医疗大型语言模型 OpenEvidence 在 100 例多学科炎症病病例讨论中的可行性。来自 7 个学科的 21 名专家通过 9 项量表评估了其临床相关性、诊断支持及时间效率等维度。

正文

Introduction

Inflammatory diseases span a broad range of medical disciplines and often present diagnostic or therapeutic challenges that extend beyond the boundaries of individual disciplines. Interdisciplinary clinical boards (ICB) on chronic inflammatory diseases are platforms for the discussion of particularly complicated or multidisciplinary cases among experts in their field. Still, even highly specialized experts need to consult additional resources when encountering rare diseases or unusual clinical constellations. This need is further amplified by the rapid and constant expansion of medical knowledge, with more than 40 million references to journal articles listed in PubMed/MEDLINE to date1. Conventional literature search is time-consuming and typically performed before or after the discussion. Large language models (LLMs) offer promising opportunities to augment expert decision-making with real-time access to and synthesis of relevant literature in clinical settings, providing evidence-based responses to clinical questions2. This allows the integration of literature research directly into ICB discussions.

While many LLM-models have been evaluated for their performance on medical exams, only a few studies have assessed their use with real patient data3. OpenEvidence (OE) is an LLM specifically designed for clinical decision support and is widely used among healthcare professionals, although independent data quantifying its distribution are not publicly available. It delivers evidence from trusted peer‑reviewed sources with traceable source references enabling clinicians to verify the underlying literature and reasoning of its outputs. While OE has shown to reinforce clinical decisions in primary care and in highly circumscribed clinical questions, it has not yet been evaluated in complex or multidisciplinary clinical cases4. This study explores the use and integration of OE into an ICB comprising specialists in gastroenterology, dermatology, rheumatology, neurology, pediatrics, nephrology, and hemato-oncology by evaluating OE through a structured questionnaire. Experts assessed OE across multiple dimensions, including the clinical relevance of the answer, delivering new therapeutic insights, diagnostic support, time efficiency, alignment with clinical guidelines, ease of integration into the board process, enhancement of decision confidence, and recommendation for future use. By examining the model’s impact across diverse clinical domains, this study aims to provide first-hand insights into the feasibility, value, and limitations of using LLMs like OE in interdisciplinary clinical practice.

Methods

Study design and case selection

This prospective, single-center, observational study was conducted using 100 consecutive real-world patient cases, discussed in our ICB between July and November 2025. The study was approved by the Ethik-Kommission der Friedrich-Alexander-Universität Erlangen-Nürnberg under reference number 25-151-B. All methods were performed in accordance with the relevant guidelines and regulations. Each participant gave written informed consent prior to participation. Cases reflected the complex spectrum of inflammatory diseases typically encountered in multidisciplinary settings, and included aspects from gastroenterology, dermatology, rheumatology, neurology, pediatrics, nephrology, and hemato-oncology. OE was tasked with providing evidence-based diagnostic and therapeutic suggestions tailored to each case. The prompt for OE was prepared prior to the ICB session, aiming to cover all relevant aspects of the case (MR, IS, JL and LS). No personal patient data was entered in OE. Each case was presented and discussed during regular ICB sessions. The case was first discussed by the board independently of OE and a preliminary recommendation was pre-noted in the institutional documentation system; the prepared question was then submitted to OE and the answer was displayed to the board. The ICB had the opportunity to add or amend aspects that had not been raised during the preceding expert discussion; and the recommendation was subsequently confirmed as final and documented.

After the final recommendation had been confirmed, the answer of OE for the same case was rated by the participants within the same session. A total of 21 specialists from seven disciplines participated in the evaluations. Each participating clinician independently evaluated the answer of OE using a standardized questionnaire comprising nine items. Eight items were assessed using a 5-point Likert scale (1 = strongly disagree, 5 = strongly agree), one item allowed for free-text comments. The specific items assessed are listed in Table S1. Each specialist participating in the board provided a separate evaluation for every case. Responses were collected pseudonymously to encourage candid feedback. Quantitative data from the Likert scale questions were analyzed descriptively, while qualitative data from the free-text responses were reviewed to identify recurring themes and suggestions for improvement.

Queries to OE were submitted between 14 July 2025 and 10 November 2025 in round 1 and between 28 February 2026 and 6 March 2026 in round 2. OE is a proprietary and continuously updated commercial platform; no model or product version identifier was disclosed by the provider at either time point. All queries were submitted using the default platform settings, with one generation per case and without regeneration of or selection among multiple outputs. As the platform does not provide deterministic output, identical answers cannot be reproduced.

Data and statistics

Data was recorded using Microsoft Excel (version 2108). Descriptive statistics (mean, standard deviation, minimum, and maximum) were calculated for all quantitative items. Reliability analyses were performed in R (version 4.5.3; R Foundation for Statistical Computing, Vienna, Austria). Internal consistency of the eight-item Likert scale was assessed using Cronbach’s α based on polychoric correlations, with 95% confidence intervals obtained by bootstrapping.

Retrospective rating procedure

Answers of OE were retrospectively evaluated for their agreement with the final decision of the ICB, which was formally documented and served as the clinical reference standard. Cases in which a reliable assessment was not possible were excluded from the primary reliability analysis. This was the case, if e.g. the board referred the case to external specialists. Two raters (WS and JL) independently assessed the degree of agreement between the OE response and the corresponding ICB recommendation for all remaining cases (n = 97). Both raters are authors of this manuscript, members of the ICB and JL was involved in the preparation of 10 case prompts. Rating was performed in a blinded, non-collaborative manner, with raters having no access to the other rater’s assessments. Blinding to the evaluation round was not feasible, since the round could be identified from the structure of the OE answer. Agreement was classified on a three-point ordinal scale: 0 = no agreement, 1 = partial agreement, 2 = complete agreement. Rating was performed in two rounds. In the first round, original answers of OE were assessed. In the second round, all cases were re-entered into OE using a structured, optimized prompt (shown below) and reassessed by the same two physicians. The same case vignette and questions were used and inserted after the text block below.

You represent the collective expertise of the members of an interdisciplinary inflammation board. Analyze the following case in three steps:

  1. 1.

    Information assessment: Is the available information sufficient to make an informed recommendation?

  2. 2.

    If information is missing: Ask specific follow-up questions, prioritized by clinical relevance.

  3. 3.

    Provide a structured recommendation with justification based on the available information.

Case:[case vignette].

Question:

[question]

With the optimized prompt, OE returned follow-up questions in all cases. These follow-up questions were not answered and no additional information was supplied to OE before the recommendation was generated.

Inter-rater reliability

Inter-rater reliability was assessed for both evaluation rounds independently. Percent agreement, linearly weighted Cohen’s kappa (κ), and squared weighted Cohen’s kappa were calculated using the irr package (version 0.84.1) in R. Kappa values were interpreted according to the benchmarks proposed by Landis and Koch: values below 0.20 were classified as slight agreement, 0.21–0.40 as fair, 0.41–0.60 as moderate, 0.61–0.80 as substantial, and values above 0.80 as almost perfect agreement5.

Comparison of evaluation rounds

To test whether prompt optimization resulted in an improvement in OE response quality, average ratings per case, calculated as the arithmetic mean of both raters’ scores, were compared between round 1 and round 2. Given the ordinal scale and the paired structure of the data (identical cases evaluated in both rounds), a Wilcoxon signed-rank test was applied. Zero differences were excluded, so that the effective sample size corresponds to the number of non-zero differences; as the absolute differences were heavily tied, the normal approximation with correction for ties and continuity was used (wilcox.test, exact = FALSE, correct = TRUE). Effect size is reported as the matched-pairs rank-biserial correlation r with a bootstrap 95% confidence interval (rcompanion 2.5.4, R 4.5.3). A two-sided significance level of p < 0.05 was considered statistically significant. As no single ground-truth rating was available for cases with inter-rater disagreement, performance of OE is reported both per rater separately as well as consensus cases, defined as cases in which both raters assigned identical scores.

Temporal stability and prompt effects

To assess temporal stability, all cases were resubmitted with both prompt versions on 7 September 2026 (pipeline configuration non_us_tier1_hcp, model profile “Osler”), with both versions run within minutes to eliminate any between-version time effect. Output was then compared to the original answers with automated text-analysis via Claude, Opus 5, Anthropic on 9 September 2026.

Results

Case characteristics

A total of 100 patient cases were evaluated, resulting in 759 rater-case evaluations. On average, 2.3 different specialties were represented per case, and all specialists participating in the ICB rated OE’s response for each case, regardless of whether the case fell within their own specialty. In 333 evaluations, raters assessed cases in which their own specialty was directly involved. Of the 100 cases, dermatological aspects were present in 82%; gastroenterological aspects in 52%, rheumatologic aspects in 36%, hemato-oncological aspects in 26%, neurological aspects in 9%, pediatric aspects in 7%, nephrological aspects in 6% and aspects of other specialties in 8%.

Prompt characteristics

Prompts for OE contained main diagnosis in 99%, additional diagnoses in 77%, current or previous therapies in 77%, laboratory results in 25%, clinical symptoms in 22%, imaging results in 15%, and histological findings in 8%. Questions to the ICB were primarily of therapeutic concern (89%) rather than diagnostic (11%). Of those, 12 were addressing potential adverse effects of current therapy, 14 addressed oncologic risk under current or planned therapy, 11 addressed infectious risk under current or planned therapy, 6 addressed potential drug-disease interactions under current or planned therapy, 24 addressed options for multi-drug refractory or complex cases.

Likert ratings by experts

Among the eight dimensions assessed on a 5‑point Likert scale, OE achieved the highest mean rating for “providing clinically relevant information”, followed by “smooth integration into the board discussion” and “recommendation for future use”. Ratings for “Increased confidence in the final decision”, “alignment with clinical guidelines”, and “providing new therapeutic insights” were slightly lower but still near the upper end of the scale. “Improved diagnostic clarity” was rated similarly, while perceived “time saved in decision-making” received the lowest rating across all dimensions. All items achieved a median score of 4, and no item fell below a mean of 3.5, indicating an overall positive assessment across all evaluated dimensions. Notably, the comparatively higher standard deviation for “time saved in decision-making” suggests greater variability in the perceived time efficiency of OE across raters and cases. Detailed results are shown in Fig. 1. Items of the questionnaire are shown in Table S1.

Fig. 1

Expert evaluation of OpenEvidence across eight dimensions. Diverging stacked bar chart showing the distribution of Likert-scale ratings (1 = strongly disagree to 5 = strongly agree) for each questionnaire item (n = 759 evaluations from 100 cases). The vertical line represents the neutral midpoint. All items achieved a median score of 4. “Clinical Relevance” received the highest mean rating (4.06 ± 0.91), while “Saved Time” received the lowest (3.50 ± 1.20).

Ratings varied across specialties, with hemato-oncologists providing the highest overall average scores (range across dimensions 3.67–4.59), followed by gastroenterologists and neurologists (both 4.05). Dermatologists rated OE similarly positive overall (3.95), with particularly high scores for “recommendation for future use” (4.22) and “smooth integration” (4.17), but notably lower ratings for “time saved in decision-making” (3.17). Rheumatologists provided more conservative ratings across most dimensions (overall average 3.59), with the lowest scores among non-nephrology specialists for “smooth integration” (3.49) and “alignment with clinical guidelines” (3.54). Nephrologists consistently provided the most conservative ratings across all dimensions (overall average 2.85), with particularly low scores for “time saved in decision-making” (1.90), “improved diagnostic clarity” (2.30), and “providing new therapeutic insights” (2.57). Pediatricians rated “time saved in decision-making” highest among all specialties (4.09), suggesting a perceived efficiency benefit in this subgroup. These results should be interpreted with caution given the limited number of raters per specialty (n = 2–6) and the unequal distribution of cases across disciplines, which preclude formal statistical comparisons between specialties. Ratings stratified by specialty are shown in Figure S1.

The performed reliability analyses based on polychoric correlations showed very high internal consistency of our 8-item scale (α = 0.97; 95% CI: 0.93–0.99; Figure S2). Corrected item-total correlations were very high for all items (r = 0.82–0.94). Cronbach’s α remained stable (α = 0.97) when any item was removed, indicating that all items contribute meaningfully to the scale. Moreover, the exploratory factor analysis supports the unidimensionality and homogeneity of the scale measuring perceived trust in OE.

Free text provided by experts

Free-text comments were provided in 7 of 759 evaluations (0.92%). Limitations of OE, identified in the free text item, were the missing differentiation and weighting of main and secondary diagnoses. Another point was the strict adherence to guidelines, even if the clinical situation required deviation. Some cases seemed too complex for OE. The critical discussion of the level of evidence was identified as a strength.

Agreement of OpenEvidence with final board decision

Answers of OE were evaluated for their agreement (none, partial, complete) to the final decision of the ICB independently by two physicians, blinded to each other’s ratings (Round 1, Fig. 2A). In the primary per-rater analysis of all evaluable cases (n = 97), complete agreement with the final ICB decision was rated in 40.2% of cases by rater 1 and 50.5% by rater 2, partial agreement in 38.1% and 24.7%, and no agreement in 21.6% and 24.7%, respectively (Table 1). In the secondary analysis restricted to consensus cases, i.e. cases in which both raters assigned identical agreement scores (n = 62 of 97 evaluable cases), OE answers showed complete agreement with the final ICB decision in 51.6%, partial agreement in 22.6%, and no agreement in 25.8%.

Fig. 2

Distribution of agreement categories between OpenEvidence answers and the final board decision, stratified by rater in n = 97 evaluable cases. (A) Round 1 (original prompt). (B) Round 2 (optimized prompt). Agreement was classified on a three-point ordinal scale: none (0), partial (1), or complete (2).

Table 1 Agreement between OpenEvidence responses and the final decision of the interdisciplinary clinical board.

Full size table

The inter-rater reliability analysis showed 63.9% raw agreement between the two raters (62 of 97 evaluable cases; Fig. 3A). Linear (κ = 0.555) and quadratic weighted (κ = 0.669) Cohen’s kappa are reported with 95% confidence intervals in Table S2.

Fig. 3

Inter-rater agreement between two independent physicians assessing the concordance of OpenEvidence answers with the final board decision. Confusion matrices showing the cross-tabulation of ratings by Rater 1 and Rater 2 on a three-point ordinal scale (none, partial, complete). (A) Round 1 linearly weighted Cohen’s κ = 0.555, indicating moderate agreement. (B) Round 2 linearly weighted Cohen’s κ = 0.591, indicating moderate agreement.

Prompt optimization

After prompt optimization, all cases were processed again in OE and answers were rated by the same two physicians (Round 2, Fig. 2B). In the primary per-rater analysis complete agreement was rated in 50.5% of cases by rater 1 and 51.5% by rater 2, partial agreement in 38.1% and 29.9% and no agreement in 11.3% and 18.6%, respectively (Table 1). In consensus cases (n = 69 of 97 evaluable cases), complete agreement was rated in 56.5%, partial agreement in 29.0% and no agreement in 14.5%. In the comparison between the two rounds, answers improved in 29 cases, were unchanged in 48 and worsened in 20 cases (Fig. 4). Mean rating changed from 1.222 in round 1 to 1.361 in round 2 (Wilcoxon signed-rank test on the 49 of 97 pairs with non-zero differences, V = 792.5, p = 0.0642; matched-pairs rank-biserial correlation r = 0.294, 95% CI - 0.008 to 0.565). In the inter-rater analysis, raw agreement was 71.1% (69 of 97 evaluable cases) and weighted Kappa was κ = 0.591 (linear) and κ = 0.670 (quadratic), indicating moderate agreement (Fig. 3B).

Fig. 4

Per-case change in agreement ratings between Round 1 (original prompt) and Round 2 (optimized prompt). Each dot represents one case, plotted by the average rating of both raters in Round 1 (x-axis) and Round 2 (y-axis). The dashed diagonal line indicates no change. Cases above the line improved with prompt optimization (green, n = 29), cases on the line remained unchanged (grey, n = 48), and cases below the line worsened (red, n = 20). Mean rating improved from 1.222 to 1.361 (Wilcoxon signed-rank test, p = 0.0642; n = 49 of 97 matched cases; V = 792.5, r = 0.294 (95% CI - 0.008 - 0.565)).

Temporal stability and prompt effects

The same prompts (original and optimized) of all cases were re-run 10–14 months (original prompt) and 6 months (optimized prompt) after the initial request. Outputs of the same prompt were not reproducible: for the original prompt, mean response length increased by 91.1% (+ 218 words, 240 to 458) and the mean number of cited references by 93.0% (+ 5.7 references, 6.2 to 11.9), with only 20.4% of originally cited references retained. For the optimized prompt the corresponding increases were 33.5% (+ 137 words, 410 to 547) and 22.1% (+ 2.2 references, 10.0 to 12.2), with 23.6% of references retained and 37.1% published in 2025 or later.

When run on the same day within minutes, the different prompts changed the output of the same case to a lesser degree: mean response length increased by 19.6% (+ 90 words) and the mean number of cited references by only 3.1% (+ 0.4 references), and the core clinical recommendation was identical in 87 of 100 runs (87.0%).

Discussion

The integration of specialized LLMs represents a promising opportunity for clinical decision support, as LLMs can enhance access to medical knowledge, support diagnostic reasoning, and facilitate evidence-based recommendations across diverse specialties6. Medically specialized LLMs seem to be most suitable, while even non-specialized LLMs like ChatGPT have shown to be able to provide the correct diagnosis in a number of complex clinical cases7. However OE, as specialized LLM, has shown to outperform ChatGPT in certain tasks8.

In this study, OE was assessed by experts from each specialty using a multi-dimensional questionnaire, aligning with emerging best practices for LLM evaluation, which emphasize multidimensional expert assessment beyond simple accuracy metrics. Ratings were positive (mean > 3) in all dimensions, with very high internal consistency. Experts rated answers of OE as clinically relevant and helpful in delivering data on therapeutic options and possible diagnostics. However, the identification of relevant data beforehand was key. In the case of a question regarding diagnostic or therapeutic options, missing information is routinely demanded by the relevant expert in the ICB discussion. OE provides a more general answer in these cases, leading to less accurate and sometimes misleading answers. Complete agreement between the answers of OE and the final decision of the ICB was reached in a minority to a bare majority of cases in round 1, depending on whether all evaluable cases or only consensus cases are considered, and tended to be higher after prompt optimization. In either analysis, a relevant proportion of answers did not fully agree with the board decision, indicating that human oversight is always needed. LLMs should therefore not be used, if their answer cannot be judged by the user. In studies comparing board decisions with LLMs, a comparable accuracy was found. However, these studies assessed decisions in oncological settings, which typically provide a narrower range of diseases and a more standardized question to the LLM9,10.

Concordance with the ICB decision tended to be higher in the second round, in which a structured, optimized prompt was used. This observation must be interpreted as exploratory. The two rounds were not run concurrently but three to eight months apart, while OE is a continuously updated commercial platform. Changes in the underlying model or retrieval corpus are therefore inseparable from the effect of the prompt. Therefore, we tested the temporal stability which showed that outputs were not reproducible over time. The between-version drift over time exceeded the effect of prompt optimization itself. In the confounded original design response length and number of references increased profoundly, whereas the de-confounded, same-session prompt effect accounted for only a fraction of it. Reference turnover was the least stable output characteristic, with roughly four out of five sources replaced within a year despite an unchanged prompt.

In addition, blinding of the evaluation rounds was not feasible due to structure of the responses, so that expectation bias cannot be excluded. The approach of prompt optimization should nevertheless be refined systematically and evaluated prospectively with elimination of time gaps. In future studies, the information provided in the prompt should be clearly defined in advance to ensure that it is precise, complete, reproducible, and independent of individual user interpretation. Standardized prompt templates or guidelines could help ensure consistent and reliable prompt formulation across different users and clinical settings.

With the optimized prompt, OE returned follow-up questions in all cases. While these questions were not answered and no additional information was supplied, the follow-up questions were perceived as useful for verifying whether all clinically relevant aspects of a case had been considered by the board, and this function may be of value independently of the recommendation itself. Whether answering the follow-up questions improves the quality of the recommendation was not tested and should be addressed prospectively.

The degree of automation by AI applications in medicine is not binary, but rather spans a continuous spectrum of human–machine interaction. In his review, Topol compared AI applications in medicine with self-driving cars, labeling the degree of autonomy with level 0 (no automation) to level 5 (full automation without potential for human backup). The integration of OE in the ICB comprises a level 1 intervention (assistance). Importantly, he further suggested not to surpass level 3 (conditional automation) in medicine11.

A major limitation in the use of LLMs in medicine identified is the human-computer-interface, when data is entered in an oversimplified or incomplete manner. Prompt quality therefore confounds the performance of LLMs. It must be emphasized that human expertise is and remains critical in the use of LLMs in complex cases, as the gathering and identification of relevant information is solely up to the user, as well as the interpretation of the LLM’s answer. Particularly physicians with limited expertise in a given field need to be cautious and should avoid relying solely on answers by any LLM. Even if the question entered is answered correctly, missing information in the prompt or misinterpretation of the answer can lead to errors, as missing information in the prompt is not requested by OE. Development and improvement of medical LLMs should therefore not only focus on improving answer accuracy, but also on mechanisms to verify if all relevant data has been provided. Additionally, so-called hallucinations, i.e. the generation of plausible yet factually incorrect outputs, can further compromise answer quality12. Access to primary data such as laboratory and imaging data could mitigate this weakness, but it could also make auditing and monitoring such systems more difficult. In addition, access to primary patient data may raise concerns regarding data privacy and the protection of patients’ personal rights.

Several limitations of this study should be acknowledged. First, this is a single-center study, and the case mix reflects the specific patient population and referral patterns of our institution, which may limit the generalizability of the findings to other centers or ICB settings. Second, the distribution of cases across specialties was unequal, with dermatological and gastroenterological aspects being overrepresented, while nephrological and neurological aspects were less frequently encountered. This precluded formal statistical comparisons between specialties, and specialty-specific findings should therefore be interpreted with caution. Third, the number of ICB members per specialty was small (n = 2–6), which further limits the robustness of specialty-level conclusions. Further, prompt quality was not formally standardized or controlled. Although prompts were prepared by a small group of physicians aiming to cover all relevant aspects, variability in the information included across cases represents an uncontrolled confounder that may have influenced OE’s performance independently of the model’s actual capabilities. The agreement between OE’s answers and the ICB’s final decision was evaluated by two physicians using a predefined but necessarily subjective definition of agreement, reflecting the inherent difficulty of operationalizing clinical concordance. The remaining discordance between the two raters underscores the complexity of this assessment. Rater independence is a further limitation. Both raters are authors, members of the ICB and one was involved in the preparation of 10 case prompts. Blinding of the evaluation round was not feasible, as the structure of the answers generated with the optimized prompt revealed the round. Fully independent external assessment would strengthen the study and should be included in future work. A further limitation is that the perceived benefit in terms of time was assessed only by a Likert item, which was the lowest-rated and most variable item, while no objective timing, such as session duration or time per case with and without OE, was recorded. Statements on time efficiency therefore refer to perception and not to measured time savings. Likewise, the frequency with which the OE answer led to a supplementation or modification of the pre-noted board recommendation was not recorded systematically, so that the actual contribution of OE to the final decision cannot be quantified. Finally, this study evaluated a single LLM within a specific time frame. Given the rapid development of LLM technology, the findings may not be representative of future model versions or alternative platforms. Outputs were not reproducible over time and newer models may bring bigger improvements than prompt optimization. Of note, OE has since discontinued its services in the European Union, underscoring the volatility of the current AI landscape and the need to evaluate platform-independent principles of LLM integration rather than individual products.

This study suggests that the integration of OE into an ICB of experts is feasible, provided that both, entered data and answers are critically reviewed. The majority of board members reported increased confidence in decision-making and recommended continued use of OE. These findings do not permit conclusions on patient safety, since patient outcomes, clinical harm, the severity of errors, the consequences of discordant recommendations, hallucinations and unsupported citations were not assessed in this study. Crucially, final clinical responsibility remains with the treating physician, and any LLM‑generated information must be regarded as decision‑support rather than a substitute for professional judgment.

Data availability

The data that support the findings of this study are available from the corresponding author upon reasonable request.

References

  1. National Library of Medicine [Internet]. PubMed. https://pubmed.ncbi.nlm.nih.gov (2026).

  2. Thirunavukarasu, A. J. et al. Large language models in medicine. Nat. Med. 29 (8), 1930–1940 (2023).

    Article  PubMed  Google Scholar 

  3. Bedi, S. et al. Testing and Evaluation of Health Care Applications of Large Language Models: A Systematic Review. JAMA 333 (4), 319–328 (2025).

    Article  PubMed  PubMed Central  Google Scholar 

  4. Hurt, R. T. et al. The Use of an Artificial Intelligence Platform OpenEvidence to Augment Clinical Decision-Making for Primary Care Physicians. J. Prim. Care Community Health (2025).

  5. Landis, J. R. & Koch, G. G. The measurement of observer agreement for categorical data. Biometrics 33 (1), 159–174 (1977).

    Article  PubMed  Google Scholar 

  6. Shool, S. et al. A systematic review of large language model (LLM) evaluations in clinical medicine. BMC Med. Inf. Decis. Mak. 25 (1), 117 (2025).

    Article  Google Scholar 

  7. Ríos-Hoyo, A. et al. Evaluation of large language models as a diagnostic aid for complex medical cases. Front. Med. 11, 1380148 (2024).

    Article  Google Scholar 

  8. Nihal, A., Yang, K. & Pavlidakey, P. G. OpenEvidence Is Superior to GPT in Skin Cancer Guideline Queries. Int. J. Dermatol. 17893. (2025).

  9. Ammo, T. et al. Evaluating ChatGPT-4o as a decision support tool in multidisciplinary sarcoma tumor boards: heterogeneous performance across various specialties. Front. Oncol. 14, 1526288 (2025).

    Article  PubMed  PubMed Central  Google Scholar 

  10. Yılmaz, M. et al. Comparison of artificial intelligence and multidisciplinary team recommendations in the management of colorectal cancer liver metastases. Sci. Rep. 16 (1), 7278 (2026).

    Article  ADS  PubMed  PubMed Central  Google Scholar 

  11. Topol, E. J. High-performance medicine: the convergence of human and artificial intelligence. Nat. Med. 25 (1), 44–56 (2019).

    Article  PubMed  Google Scholar 

  12. Roustan, D. & Bastardot, F. The Clinicians’ Guide to Large Language Models: A General Perspective With a Focus on Hallucinations. Interact. J. Med. Res. 14, e59823 (2025).

    Article  PubMed  PubMed Central  Google Scholar 

Download references

Acknowledgements

The present work was performed in (partial) fulfillment of the requirements for obtaining the degree ‘Dr. rer. biol. hum.’ for MR at the Friedrich-Alexander-Universität Erlangen-Nürnberg (FAU).

Funding

Open Access funding enabled and organized by Projekt DEAL. MR and LS were funded by the Deutsche Forschungsgemeinschaft (DFG, German Research Foundation) – 493624887 (Clinician Scientist Program NOTICE).

Author information

Authors and Affiliations

  1. Department of Dermatology, Uniklinikum Erlangen and Friedrich-Alexander-Universität (FAU) Erlangen-Nürnberg, Erlangen, Germany

    Moritz Ronicke, Lukas Sollfrank, Ioannis Sagonas, Isabel Kowalski, Jennifer Lermer, Wiebke Sondermann & Michel Sticherling

  2. Deutsches Zentrum Immuntherapie (DZI), Uniklinikum Erlangen and Friedrich-Alexander-Universität (FAU) Erlangen-Nürnberg, Erlangen, Germany

    Moritz Ronicke, Lukas Sollfrank, Ioannis Sagonas, Raja Atreya, Ricardo Grieshaber-Bouyer, André Hoerning, André Jefremow, Isabel Kowalski, Tobias Krickau, Jennifer Lermer, Norbert Meidenbauer, Jürgen Rech, Veit Rothhammer, Gunnar Schley, Wiebke Sondermann & Michel Sticherling

  3. Department of Gastroenterology, Uniklinikum Erlangen and Friedrich-Alexander-Universität (FAU) Erlangen-Nürnberg, Medicine 1, Erlangen, Germany

    Raja Atreya & André Jefremow

  4. Department of Rheumatology, Uniklinikum Erlangen and Friedrich-Alexander University (FAU) Erlangen-Nürnberg, Medicine 3, Erlangen, Germany

    Ricardo Grieshaber-Bouyer & Jürgen Rech

  5. Department of Pediatrics, Uniklinikum Erlangen and Friedrich-Alexander-Universität (FAU) Erlangen-Nürnberg, Erlangen, Germany

    André Hoerning & Tobias Krickau

  6. Department of Hematology and Oncology, Uniklinikum Erlangen and Friedrich-Alexander-Universität (FAU) Erlangen-Nürnberg, Medicine 5, Erlangen, Germany

    Norbert Meidenbauer

  7. Department of Neurology, Uniklinikum Erlangen and Friedrich-Alexander-Universität (FAU) Erlangen-Nürnberg, Erlangen, Germany

    Veit Rothhammer

  8. Department of Nephrology and Hypertension, Uniklinikum Erlangen and Friedrich-Alexander-Universität (FAU) Erlangen-Nürnberg, Medicine 4, Erlangen, Germany

    Gunnar Schley

  9. Department of Dermatology, Venereology and Allergology, University Hospital Essen, University Duisburg-Essen, Essen, Germany

    Wiebke Sondermann

Authors

  1. Moritz Ronicke
  2. Lukas Sollfrank
  3. Ioannis Sagonas
  4. Raja Atreya
  5. Ricardo Grieshaber-Bouyer
  6. André Hoerning
  7. André Jefremow
  8. Isabel Kowalski
  9. Tobias Krickau
  10. Jennifer Lermer
  11. Norbert Meidenbauer
  12. Jürgen Rech
  13. Veit Rothhammer
  14. Gunnar Schley
  15. Wiebke Sondermann
  16. Michel Sticherling

Contributions

M.R.: Conceptualization, Methodology, Investigation, Writing – Original Draft, Writing – Review & Editing, Visualization. L.S.: Investigation, Writing – Review & Editing. I.S.: Methodology, Data Curation, Software, Writing – Review & Editing. R.A.: Writing – Review & Editing. R.G.-B.: Writing – Review & Editing. A.H.: Writing – Review & Editing. A.J.: Writing – Review & Editing. I.K.: Writing – Review & Editing. T.K.: Writing – Review & Editing. J.L.: Writing – Review & Editing. N.M.: Writing – Review & Editing. J.R.: Investigation, Writing – Review & Editing. V.R.: Writing – Review & Editing. G.S.: Writing – Review & Editing. W.S.: Investigation, Writing – Review & Editing. M.S.: Writing – Review & Editing.

Corresponding author

Correspondence to Moritz Ronicke.

Ethics declarations

Competing interests

The authors declare no competing interests.

Ethics approval and consent to participate

This study was approved by the Ethik-Kommission der Friedrich-Alexander-Universität Erlangen-Nürnberg (reference number 25-151-B). All participating physicians provided informed consent prior to participation in the study. As no personal patient data was entered into the LLM and only anonymized case vignettes were used, individual patient consent was waived by the ethics committee.

Additional information

Publisher’s note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Open Access This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made. The images or other third party material in this article are included in the article’s Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article’s Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit http://creativecommons.org/licenses/by/4.0/.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Ronicke, M., Sollfrank, L., Sagonas, I. et al. Feasibility and expert evaluation of a medical large language model in an interdisciplinary clinical board. Sci Rep 16, 30676 (2026). https://doi.org/10.1038/s41598-026-72397-y

Download citation

  • Received: 07 June 2026

  • Accepted: 16 September 2026

  • Published: 02 October 2026

  • Version of record: 02 October 2026

  • DOI: https://doi.org/10.1038/s41598-026-72397-y

Keywords

来源:Nature 健康护理主题 · nature.com