跳到正文
原文
Medical Xpress·· 2026-10-09AI 评分55

视角论文探讨 AI 健康建议致害后记录缺失带来的医疗风险与问责路径

Missing chatbot records leave doctors struggling to trace harm from AI health advice

AI 导读

一篇发表于 Nature Health 的视角论文指出,人们向 AI 聊天机器人获取健康建议时,对话通常留在 AI 公司平台,用户缺少向医生或监管机构追溯错误的记录,相关伤害因此难以被检测和修正。

正文
AI chatbot
Credit: Unsplash/CC0 Public Domain

As more people turn to AI chatbots for quick, 24/7 medical advice, a new perspective paper co-authored by faculty at Binghamton University, State University of New York, explores the risks of this practice that are largely invisible to users—and health care systems.

The paper, written by researchers at Binghamton University, Stanford University, Texas A&M University and Indiana University and published in Nature Health, examines what happens when a chatbot gives someone bad health advice.

The authors argue that the harm is hard to detect because the conversation stays on the AI company's platform, the person has no place to report the problem and outside researchers cannot measure the damage.

"We're getting medical advice from these chatbots, but nobody—literally no one—is looking at this. If there's an error or an issue, there's just no way for people to know," said Kaicheng Yang, an assistant professor in the School of Computing at Binghamton University's Thomas J. Watson College of Engineering and Applied Science and one of the paper's authors.

When harmful advice leaves no record

The researchers gave the example of a 60-year-old man who asked ChatGPT how to cut chloride from his diet and replaced table salt with sodium bromide. The man was hospitalized with bromide toxicity after experiencing hallucinations and paranoia.

The paper highlights that because the man received this advice from a chatbot and was unable to retrieve the conversation, his doctors had no way of knowing exactly what he was told, making it nearly impossible to piece together what happened.

The paper argues that AI companies lack a system that makes the health advice their chatbots generate for patients visible to others. This built-in, systemic lack of visibility actively prevents oversight. "Harms are not only possible, but structurally hidden from the clinicians, researchers, and regulators who could otherwise detect and correct them," the authors note.

Misinformation beyond chatbot conversations

The paper also emphasizes a key aspect of chatbots generating answers for people with health questions—they can be wrong. Chatbots can confidently generate erroneous answers, oversimplify information or draw from outdated information or false claims. However, the authors also note that those risks can extend to people who are not actively seeking answers.

With the prevalence of AI summaries in search engines and the integration of AI into social media platforms like X and Meta, you do not have to be actively searching for an answer to be given incorrect information.

"Sometimes I'm not even looking for health information, but just by browsing my social media feeds, it's there. It just shows up, and we believe that could have undesirable outcomes, especially if there's medical misinformation or state actors trying to manipulate the online discussion," Yang said.

Regardless of whether a person is actively looking for answers or comes across them incidentally, the paper's authors argue that the harm rarely leaves traces that clinicians, regulators or researchers can verify.

Opening health advice to outside scrutiny

To combat this, the authors propose several methods to increase transparency and accountability for AI companies. They note that AI companies should give users access to their own health conversations so they can share this information with clinicians, who can better investigate the pathway that led to a harmful event. They also recommend that those companies develop a disclosure system to allow health guidance to be flagged, reported and investigated.

They further recommend that social media companies take more responsibility by improving the clarity of labels on AI-generated health content and withholding that content until it is vetted by medical governing bodies. Similarly, search engines should use only vetted information in summaries. Policymakers, the authors suggest, should also extend physician malpractice liability to AI chatbot companies.

"We do not think this is something we should rely on the companies to do, because their incentive is always to make more money," Yang said. "Building such a system goes against that incentive. So we have to have some kind of third-party monitoring system, an independent evaluation."

Publication details

DeVerna, M.R., et al The Invisible Risks of AI-Generated Health Information, Nature Health (2026). DOI: 10.1038/s44360-026-00206-x

Journal information: Nature Health

Who's behind this story?

Sadie Harley

Sadie Harley

BSc Life Sciences & Ecology. Microbiology lab background with pharmaceutical news experience in oil, gas, and renewable industries. Full profile →

Robert Egan

Robert Egan

Bachelor's in mathematical biology, Master's in creative writing. Well-traveled with unique perspectives on science and language. Full profile →

Citation: Missing chatbot records leave doctors struggling to trace harm from AI health advice (2026, October 9) retrieved 9 October 2026 from https://medicalxpress.com/news/2026-10-chatbot-doctors-struggling-ai-health.html

This document is subject to copyright. Apart from any fair dealing for the purpose of private study or research, no part may be reproduced without the written permission. The content is provided for information purposes only.

来源:Medical Xpress · medicalxpress.com