ChatGPT与Gemini在泌尿系统结石治疗中的临床准确性和引用可靠性比较
A comparative expert evaluation of ChatGPT and Gemini on urolithiasis: clinical accuracy versus citation reliability
研究比较了ChatGPT 5.2 Pro和Gemini 3 Pro在泌尿系统结石治疗中的表现,发现Gemini在临床准确性和指南遵循性方面优于ChatGPT,但两者在引用可靠性方面均存在严重问题。
We’re sharing this article early to provide faster access to peer-reviewed, accepted research. It is citable and carries a permanent DOI. This version is subject to further edits and will be replaced automatically by the final Version of Record. All legal disclaimers apply.
Abstract
The rapid evolution of large language models (LLMs) presents opportunities and risks, including ‘hallucination’, for urological practice. This study compares the performance of ChatGPT 5.2 Pro and Gemini 3 Pro in answering urolithiasis queries, and evaluates the discrepancy between clinical accuracy and citation reliability. In this cross-sectional study conducted between January and February 2026, 65 clinical questions based on the 2025 European Association of Urology (EAU) guidelines were administered to both models. Two blinded, board-certified urologists independently evaluated the 130 generated responses. The outputs were scored on three separate dimensions. These were medical accuracy, guideline adherence and citation quality. Each dimension was scored on a 1–5 scale. Gemini 3 Pro demonstrated statistically higher scores compared to ChatGPT 5.2 Pro in terms of medical accuracy and guideline adherence (both p < 0.001). Gemini 3 Pro provided responses that were rated as clinically appropriate by experts at a rate of 98.5%, compared to 81.5% for GPT-5.2 Pro (p = 0.001). However, both models had critically flawed citation reliability. At the overall response level, ChatGPT generated high-risk responses in 95.4% of cases, compared to 69.2% for Gemini (p < 0.001). At the individual citation level, 45.8% of GPT-5.2 Pro’s citations and 24.8% of Gemini 3 Pro’s citations were entirely fabricated (score 1a). Meanwhile, 30.3% and 31.4% of citations, respectively, were real publications with incorrect or mismatched DOIs (score 1b). Notably, there was no statistical correlation between medical accuracy and citation quality (ChatGPT 5.2 Pro: rho = 0.033, p = 0.793; Gemini 3 Pro: rho=-0.195, p = 0.120). While Gemini 3 Pro provides highly accurate, guideline-adherent clinical information for urolithiasis, neither model currently serves as a reliable primary tool for academic literature retrieval. There is a stark discrepancy between the models’ high clinical accuracy and their critically poor citation reliability. Therefore, it is unsafe to rely on AI-generated medical content without strictly and independently verifying the underlying references for clinical practice.
Acknowledgements
We would like to thank Professor Hakan Türk for independently reviewing and clinically validating the question set.
Funding
The authors did not receive any funding for the submitted work.
Author information
Authors and Affiliations
Department of Urology, Uşak University Faculty of Medicine, Fevzi Çakmak, Gazi Boulevard Side Street No: 50, Uşak Centre, Uşak, 64300, Turkey
Yasin Aktaş & Adem Tunçekin
Department of Urology, Uşak Training and Research Hospital, Uşak, Turkey
Yasin Aktaş & Adem Tunçekin
Authors
- Yasin Aktaş
- Adem Tunçekin
Corresponding author
Correspondence to Yasin Aktaş.
Ethics declarations
Competing interests
The authors declare no competing interests.
Ethical approval
This observational study uses publicly available, AI-generated data and does not involve human participants, animals or patient records. Therefore, formal ethical approval has been waived.
Consent to participate
Not applicable.
Consent to publish
Not applicable.
Additional information
Publisher’s note
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
Rights and permissions
Open Access This article is licensed under a Creative Commons Attribution-NonCommercial-NoDerivatives 4.0 International License, which permits any non-commercial use, sharing, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if you modified the licensed material. You do not have permission under this licence to share adapted material derived from this article or parts of it. The images or other third party material in this article are included in the article’s Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article’s Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit http://creativecommons.org/licenses/by-nc-nd/4.0/.
About this article
Cite this article
Aktaş, Y., Tunçekin, A. A comparative expert evaluation of ChatGPT and Gemini on urolithiasis: clinical accuracy versus citation reliability. Sci Rep (2026). https://doi.org/10.1038/s41598-026-81085-w
Received: 30 July 2026
Accepted: 12 November 2026
Published: 18 November 2026
DOI: https://doi.org/10.1038/s41598-026-81085-w
Keywords
来源:Nature 健康护理主题 · nature.com