跳到正文
原文
Nature 健康护理主题·· 2026-10-03AI 评分27

四款大语言模型回答 Peyronie 病(PD)患者问题的专家评分质量与可读性比较

Comparison of expert-rated quality and readability of four large language models answering patient-oriented questions on Peyronie’s disease

AI 导读

一项观察性研究将 7 个基于 Google Trends 生成的 Peyronie 病(PD)患者问题以相同英文提示分别提交给 ChatGPT(GPT-5.5 Instant)、Gemini 3.5 Flash、DeepSeek-V4-Flash(Instant 模式)和 Grok(Auto 模式)。

正文

Scientific Reports (2026) Cite this article

We’re sharing this article early to provide faster access to peer-reviewed, accepted research. It is citable and carries a permanent DOI. This version is subject to further edits and will be replaced automatically by the final Version of Record. All legal disclaimers apply.

Abstract

Large language models (LLMs) are increasingly used by patients seeking information about medical conditions, including Peyronie’s disease (PD). However, the quality and readability of artificial intelligence (AI)-generated responses to patient-oriented PD questions remain unclear. To compare the responses generated by four widely used AI models for common patient-oriented questions about PD in terms of expert-rated quality, response length, and readability. In this observational study, seven patient-oriented questions on PD were generated based on Google Trends outputs. Identical English-language prompts were submitted once to four publicly available AI systems: ChatGPT powered by GPT-5.5 Instant, Gemini 3.5 Flash, DeepSeek-V4-Flash (Instant mode), and Grok (Auto mode). The resulting responses were independently evaluated in randomized order by three board-certified urologists using a predefined 4-point expert-rating scale. Response length was assessed using word count, while readability was evaluated using the Flesch Reading Ease Score (FRES), Flesch-Kincaid Grade Level (FKGL), Gunning Fog Score (GFS), Simple Measure of Gobbledygook (SMOG), and Readable Rating. Given the small number of paired observations, model comparisons were performed using exact Friedman tests; significant overall tests were followed by exact two-sided Wilcoxon signed-rank tests with Bonferroni correction. Reviewer-specific exact Friedman analyses did not detect statistically significant differences in expert-rated quality among the four AI models (all exact p ≥ 0.111). Inter-rater agreement was limited (single-measure ICC = 0.063, 95% CI, – 0.115 to 0.305), restricting the interpretation of comparative quality findings. Exact Friedman analyses demonstrated significant overall differences in word count, FRES, FKGL, GFS, SMOG, and Readable Rating (all exact p < 0.001). However, no pairwise comparison reached the Bonferroni-corrected significance threshold, and these analyses were severely constrained by the small number of paired observations; their non-significance should therefore not be interpreted as evidence that between-model differences were absent. Commonly used AI models showed significant overall differences in response length and readability when answering patient-oriented questions about PD, whereas reviewer-specific analyses did not detect statistically significant differences in expert-rated quality. However, the limited inter-rater agreement and the absence of external reference-standard verification preclude conclusions regarding equivalent or comparable model quality and warrant cautious interpretation of the expert-rated quality findings.

Abbreviations

AI:

Artificial intelligence

FEBU:

Fellow of the European Board of Urology

ICC:

Intraclass correlation coefficient

FKGL:

Flesch-Kincaid Grade Level

FRES:

Flesch Reading Ease Score

GFS:

Gunning Fog Score

LLM:

Large language model

PD:

Peyronie’s disease

SMOG:

Simple Measure of Gobbledygook

Funding

None.

Author information

Authors and Affiliations

  1. Department of Urology, Karasu State Hospital, Sakarya, Türkiye

    Anil Erdik

  2. School of Medicine, Department of Urology, Sakarya University, Sakarya, Türkiye

    Haci Ibrahim Cimen

  3. Department of Urology, Gerede State Hospital, Bolu, Türkiye

    Kemal Demirhan

  4. Department of Urology, Sakarya Training and Research Hospital, Sakarya, Türkiye

    Deniz Gul

Authors

  1. Anil Erdik
  2. Haci Ibrahim Cimen
  3. Kemal Demirhan
  4. Deniz Gul

Corresponding author

Correspondence to Haci Ibrahim Cimen.

Ethics declarations

Competing interests

The authors declare no competing interests.

Ethical approval

Ethical approval was obtained from the Sakarya University Faculty of Medicine Health Sciences Scientific Research Ethics Committee on May 21, 2026; decision/application number: E-590391-417. Written informed consent was obtained from all three expert reviewers before their participation in the expert-rating phase of the study. No patients, patient records, identifiable personal information, or biological samples were included in this study. The only human-derived data consisted of anonymized ratings provided by three board-certified urologists who evaluated the AI-generated responses. All procedures were performed in accordance with the relevant institutional guidelines and the Declaration of Helsinki.

Additional information

Publisher’s note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Open Access This article is licensed under a Creative Commons Attribution-NonCommercial-NoDerivatives 4.0 International License, which permits any non-commercial use, sharing, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if you modified the licensed material. You do not have permission under this licence to share adapted material derived from this article or parts of it. The images or other third party material in this article are included in the article’s Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article’s Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit http://creativecommons.org/licenses/by-nc-nd/4.0/.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Erdik, A., Cimen, H.I., Demirhan, K. et al. Comparison of expert-rated quality and readability of four large language models answering patient-oriented questions on Peyronie’s disease. Sci Rep (2026). https://doi.org/10.1038/s41598-026-72669-7

Download citation

  • Received: 13 July 2026

  • Accepted: 17 September 2026

  • Published: 03 October 2026

  • DOI: https://doi.org/10.1038/s41598-026-72669-7

Keywords

来源:Nature 健康护理主题 · nature.com