跳到正文
原文
Nature 健康护理主题·· 6 天前AI 评分43

大型语言模型在健康声明验证中的共识幻象

Agreement among large language models does not track medical truth in health claim verification

AI 导读

一项研究评估了10种大型语言模型在验证108条健康声明(40条真实、40条虚假、28条争议)时的表现,发现尽管前沿模型最准确,但无一模型可靠。多数投票未能提高准确性,且“争议”判定错误率高。研究指出,当前大型语言模型在健康声明验证中的共识并不反映医学真相。

正文

We’re sharing this article early to provide faster access to peer-reviewed, accepted research. It is citable and carries a permanent DOI. This version is subject to further edits and will be replaced automatically by the final Version of Record. All legal disclaimers apply.

Abstract

Large language models (LLMs) are increasingly proposed as tools to verify online health information. A common assumption from the self-consistency and ensemble literature is that agreement across models signals reliability. But it remains untested whether inter-model consensus tracks medical truth. We evaluated 10 LLMs spanning frontier, Russian-specific and medical-specialized families on RuHealth, a corpus of 108 short health claims (40 true, 40 false, 28 disputed) validated through multi-stage physician review. Each claim was classified as true, false, or disputed, in both Russian and a validated English translation, across decoding temperatures and 3 stochastic runs (\({\sim }\)90,000 judgements). Frontier models were most accurate (macro-F1 64–70) but no model was reliable: “disputed” was assigned to 59.8% of judgements versus a 26% true base rate, while false claims were systematically missed. Majority voting did not help. Three frontier models were unanimous on 90 of 108 claims, yet 31% of those unanimous verdicts were wrong, and 26 of 28 assigned “disputed” to claims that were actually true (14) or false (12). Thus, in this setting on health claims, agreement among contemporary LLMs is not evidence of medical correctness; this effect, which we term consensus mirage, proves to be robust to prompt wording and language.

Acknowledgements

We are deeply grateful to our clinical expert reviewers — Yuliya Klink, Natalia Usolova, and Sofia Menshikova — for their meticulous assessment of the stimulus material, and to Daria Morgacheva for her invaluable meta-review oversight.

Funding

The study was implemented in the framework of the Basic Research Program at HSE University (HSE-BR-2025-68).

Author information

Author notes

  1. E. Artemenko, O. Koltsova, S. Nikolenko: These authors contributed equally to this work.

Authors and Affiliations

  1. Social and Cognitive Informatics Laboratory, HSE University, Promyshlennaya St. 17, Saint Petersburg, 190008, Russia

    Elena Artemenko, Olessia Koltsova, Sergey Nikolenko, Anton Surkov & Anna Probst

Authors

  1. Elena Artemenko
  2. Olessia Koltsova
  3. Sergey Nikolenko
  4. Anton Surkov
  5. Anna Probst

Corresponding author

Correspondence to Elena Artemenko.

Ethics declarations

Competing interests

The authors declare no competing interests.

Additional information

Publisher's note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Open Access This article is licensed under a Creative Commons Attribution-NonCommercial-NoDerivatives 4.0 International License, which permits any non-commercial use, sharing, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if you modified the licensed material. You do not have permission under this licence to share adapted material derived from this article or parts of it. The images or other third party material in this article are included in the article’s Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article’s Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit http://creativecommons.org/licenses/by-nc-nd/4.0/.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Artemenko, E., Koltsova, O., Nikolenko, S. et al. Agreement among large language models does not track medical truth in health claim verification. Sci Rep (2026). https://doi.org/10.1038/s41598-026-79438-6

Download citation

  • Received: 12 August 2026

  • Accepted: 31 October 2026

  • Published: 14 November 2026

  • DOI: https://doi.org/10.1038/s41598-026-79438-6

Keywords

来源:Nature 健康护理主题 · nature.com