西奈山伊坎医学院研究:简短安全提醒将大语言模型潜在有害临床选择率从16.6%降至10.1%
Safety prompts can help AI models make safer clinical choices
西奈山伊坎医学院团队9月26日发表于 Communications Medicine 的研究发现,简短的安全提醒可将大语言模型在临床场景中的潜在有害选择率从16.6%降至10.1%。
基于20个模型约千万次响应,简短的安全提醒将潜在有害临床选择率从16.6%降至10.1%,提示提示词设计与自动化安全测试对临床AI开发的重要性。
As artificial intelligence (AI) becomes increasingly capable of assisting with health care tasks, a new study by researchers at the Icahn School of Medicine at Mount Sinai has found that adding a brief safety reminder reduced potentially harmful choices by AI models in clinical scenarios.
The study, published Sept. 26 in Communications Medicine, also found that AI models can be influenced by the context and instructions surrounding a clinical decision.
The findings, based on millions of outputs, suggest that how AI systems are prompted and guided may remain an important consideration in developing safe and reliable clinical applications, even as increasingly capable AI models and agents become better able to understand users' intent without lengthy or comprehensive prompts.
The reminder reduced potentially harmful choices across 19 of the 20 models tested, demonstrating a potential approach to strengthening safeguards.
Safety reminders lowered harmful choices
The research team evaluated 20 large language models using 501 variations of 50 clinical scenarios, along with 100 cases adapted from deidentified hospital discharge records. Across more than 10 million responses, the models made approximately 1.18 million potentially harmful clinical choices. Without a safety reminder, potentially harmful choices accounted for 16.6% of model responses. Adding a brief safety reminder reduced that rate to 10.1%.
The findings, the researchers say, highlight the importance of evaluating not only whether an AI model can provide accurate clinical information but also how it responds when given an instruction that conflicts with patient safety.
"AI models do not make decisions in a vacuum. The language, framing and context surrounding a request can influence how they respond, including when an instruction could be unsafe," says physician-scientist and first author Mahmud Omar, M.D., a lecturer in the Windreich Department of Artificial Intelligence and Human Health at the Icahn School of Medicine at Mount Sinai, who leads research on the safety, reliability and real-world effects of generative AI in clinical care.
"A simple safety reminder reduced potentially harmful choices in most of the models we tested, which is encouraging. But it did not eliminate them, so a reminder should be viewed as one safeguard, not a substitute for clinical oversight."
Testing unsafe instructions under pressure
For example, researchers tested scenarios in which a model was instructed to skip recommended follow-up blood tests to reduce workload. The request could also be framed as urgent or presented as an order from a superior. Models were then asked to choose among four possible actions, including following the request, maintaining the recommended follow-up or seeking help from a clinician.
The researchers varied the wording of the scenarios and tested three short safety reminders. Each combination was tested 10 times, with the order of the answer choices randomized.
The safety reminder reduced potentially harmful choices in 19 of the 20 models tested. The effect was seen both in the written clinical scenarios and in cases adapted from hospital discharge records. Examples of potentially harmful choices included skipping needed tests to reduce workload or stopping antibiotic treatment before completing the recommended regimen without a sufficient clinical reason.
"These results suggest that safety testing needs to go beyond asking whether an AI model gets the right answer under ordinary conditions," says co-senior author Girish N. Nadkarni, M.D., MPH, chair of the Windreich Department of Artificial Intelligence and Human Health at the Icahn School of Medicine at Mount Sinai and director of the Hasso Plattner Institute for Digital Health at Mount Sinai.
"As AI systems become more autonomous and are asked to complete increasingly complex tasks, we need to know whether they can recognize when an instruction may be unsafe, question it, verify it or ask a human for help."
Safeguards for increasingly autonomous systems
The researchers propose that developers and health care organizations build automated safety testing into the development and evaluation of clinical AI systems. Such testing could be conducted before a system is introduced into a clinical workflow and repeated as models are updated or new safety concerns emerge.
The study also points to an emerging challenge as AI systems evolve from question-and-answer tools into more autonomous "agents" that can carry out multiple steps. The researchers plan to examine how accumulated context may affect an agent's decisions, including when that context contains hidden instructions, known as prompt injection, or pressures to save time or stay within a budget.
The researchers say the findings do not mean that a safety reminder makes AI-generated medical advice safe to use without clinical review. Rather, the results demonstrate that relatively simple changes in how an AI system is prompted can affect its clinical choices, while underscoring the need for additional safeguards and human oversight.
Publication details
Mahmud Omar et al, Evaluating large language model responses to unsafe clinical instructions, Communications Medicine (2026). DOI: 10.1038/s43856-026-01933-8
Journal information: Communications Medicine
Key medical concepts
Who's behind this story?
Gaby Clark
MA in English, copy editor since 2021 with experience in higher education and health content. Dedicated to trustworthy science news. Full profile →
Robert Egan
Bachelor's in mathematical biology, Master's in creative writing. Well-traveled with unique perspectives on science and language. Full profile →
Citation: Safety prompts can help AI models make safer clinical choices (2026, October 10) retrieved 10 October 2026 from https://medicalxpress.com/news/2026-10-safety-prompts-ai-safer-clinical.html
This document is subject to copyright. Apart from any fair dealing for the purpose of private study or research, no part may be reproduced without the written permission. The content is provided for information purposes only.
来源:Medical Xpress · medicalxpress.com