四款大语言模型回答 Peyronie 病(PD)患者问题的专家评分质量与可读性比较
Comparison of expert-rated quality and readability of four large language models answering patient-oriented questions on Peyronie’s disease
一项观察性研究将 7 个基于 Google Trends 生成的 Peyronie 病(PD)患者问题以相同英文提示分别提交给 ChatGPT(GPT-5.5 Instant)、Gemini 3.5 Flash、DeepSeek-V4-Flash(Instant 模式)和 Grok(Auto 模式)。
- Article
- Open access
- Published: 03 October 2026
Scientific Reports (2026) Cite this article
We’re sharing this article early to provide faster access to peer-reviewed, accepted research. It is citable and carries a permanent DOI. This version is subject to further edits and will be replaced automatically by the final Version of Record. All legal disclaimers apply.
Abstract
Large language models (LLMs) are increasingly used by patients seeking information about medical conditions, including Peyronie’s disease (PD). However, the quality and readability of artificial intelligence (AI)-generated responses to patient-oriented PD questions remain unclear. To compare the responses generated by four widely used AI models for common patient-oriented questions about PD in terms of expert-rated quality, response length, and readability. In this observational study, seven patient-oriented questions on PD were generated based on Google Trends outputs. Identical English-language prompts were submitted once to four publicly available AI systems: ChatGPT powered by GPT-5.5 Instant, Gemini 3.5 Flash, DeepSeek-V4-Flash (Instant mode), and Grok (Auto mode). The resulting responses were independently evaluated in randomized order by three board-certified urologists using a predefined 4-point expert-rating scale. Response length was assessed using word count, while readability was evaluated using the Flesch Reading Ease Score (FRES), Flesch-Kincaid Grade Level (FKGL), Gunning Fog Score (GFS), Simple Measure of Gobbledygook (SMOG), and Readable Rating. Given the small number of paired observations, model comparisons were performed using exact Friedman tests; significant overall tests were followed by exact two-sided Wilcoxon signed-rank tests with Bonferroni correction. Reviewer-specific exact Friedman analyses did not detect statistically significant differences in expert-rated quality among the four AI models (all exact p ≥ 0.111). Inter-rater agreement was limited (single-measure ICC = 0.063, 95% CI, – 0.115 to 0.305), restricting the interpretation of comparative quality findings. Exact Friedman analyses demonstrated significant overall differences in word count, FRES, FKGL, GFS, SMOG, and Readable Rating (all exact p < 0.001). However, no pairwise comparison reached the Bonferroni-corrected significance threshold, and these analyses were severely constrained by the small number of paired observations; their non-significance should therefore not be interpreted as evidence that between-model differences were absent. Commonly used AI models showed significant overall differences in response length and readability when answering patient-oriented questions about PD, whereas reviewer-specific analyses did not detect statistically significant differences in expert-rated quality. However, the limited inter-rater agreement and the absence of external reference-standard verification preclude conclusions regarding equivalent or comparable model quality and warrant cautious interpretation of the expert-rated quality findings.
Abbreviations
- AI:
-
Artificial intelligence
- FEBU:
-
Fellow of the European Board of Urology
- ICC:
-
Intraclass correlation coefficient
- FKGL:
-
Flesch-Kincaid Grade Level
- FRES:
-
Flesch Reading Ease Score
- GFS:
-
Gunning Fog Score
- LLM:
-
Large language model
- PD:
-
Peyronie’s disease
- SMOG:
-
Simple Measure of Gobbledygook
Funding
None.
Author information
Authors and Affiliations
Department of Urology, Karasu State Hospital, Sakarya, Türkiye
Anil Erdik
School of Medicine, Department of Urology, Sakarya University, Sakarya, Türkiye
Haci Ibrahim Cimen
Department of Urology, Gerede State Hospital, Bolu, Türkiye
Kemal Demirhan
Department of Urology, Sakarya Training and Research Hospital, Sakarya, Türkiye
Deniz Gul
Authors
- Anil Erdik
- Haci Ibrahim Cimen
- Kemal Demirhan
- Deniz Gul
Corresponding author
Correspondence to Haci Ibrahim Cimen.
Ethics declarations
Competing interests
The authors declare no competing interests.
Ethical approval
Ethical approval was obtained from the Sakarya University Faculty of Medicine Health Sciences Scientific Research Ethics Committee on May 21, 2026; decision/application number: E-590391-417. Written informed consent was obtained from all three expert reviewers before their participation in the expert-rating phase of the study. No patients, patient records, identifiable personal information, or biological samples were included in this study. The only human-derived data consisted of anonymized ratings provided by three board-certified urologists who evaluated the AI-generated responses. All procedures were performed in accordance with the relevant institutional guidelines and the Declaration of Helsinki.
Additional information
Publisher’s note
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
Rights and permissions
Open Access This article is licensed under a Creative Commons Attribution-NonCommercial-NoDerivatives 4.0 International License, which permits any non-commercial use, sharing, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if you modified the licensed material. You do not have permission under this licence to share adapted material derived from this article or parts of it. The images or other third party material in this article are included in the article’s Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article’s Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit http://creativecommons.org/licenses/by-nc-nd/4.0/.
About this article
Cite this article
Erdik, A., Cimen, H.I., Demirhan, K. et al. Comparison of expert-rated quality and readability of four large language models answering patient-oriented questions on Peyronie’s disease. Sci Rep (2026). https://doi.org/10.1038/s41598-026-72669-7
Received: 13 July 2026
Accepted: 17 September 2026
Published: 03 October 2026
DOI: https://doi.org/10.1038/s41598-026-72669-7
Keywords
来源:Nature 健康护理主题 · nature.com