A new study by Mount Sinai researchers finds that AI chatbots are prone to spreading medical misinformation, but simple prompts can reduce errors significantly, emphasizing the need for stronger safeguards in health care.
Researchers at the Icahn School of Medicine at Mount Sinai have discovered that AI chatbots, commonly used in health care, are highly susceptible to spreading false medical information. This revelation underscores the urgent need for stronger safeguards to ensure these tools deliver accurate advice.
Their findings, published in the journal Communications Medicine, suggest that implementing a simple built-in warning prompt can significantly mitigate this risk.
Lead author Mahmud Omar, an independent consultant with the research team, explains the vulnerability observed.
“What we saw across the board is that AI chatbots can be easily misled by false medical details, whether those errors are intentional or accidental,” he said in a news release. “They not only repeated the misinformation but often expanded on it, offering confident explanations for non-existent conditions. The encouraging part is that a simple, one-line warning added to the prompt cut those hallucinations dramatically, showing that small safeguards can make a big difference.”
Experiment Details
The research team created fictional patient scenarios incorporating fabricated medical terms, such as made-up diseases or symptoms.
These scenarios were submitted to leading AI models. Initially, no additional guidance was provided, resulting in chatbots confidently producing erroneous information. However, when a one-line caution was added, reminding the AI that the information might be inaccurate, the rate of errors dropped significantly.
“Our goal was to see whether a chatbot would run with false information if it was slipped into a medical question, and the answer is yes,” added co-corresponding senior author Eyal Klang, chief of generative AI in the Windreich Department of Artificial Intelligence and Human Health at the Icahn School of Medicine at Mount Sinai.
“Even a single made-up term could trigger a detailed, decisive response based entirely on fiction. But we also found that the simple, well-timed safety reminder built into the prompt made an important difference, cutting those errors nearly in half,” he added. “That tells us these tools can be made safer, but only if we take prompt design and built-in safeguards seriously.”
Future Implications
The research team plans to apply this approach to real, anonymized patient records and test advanced safety prompts and retrieval tools. They believe their “fake-term” method could be a powerful tool for hospitals, tech developers and regulators to stress-test AI systems before clinical applications.
“Our study shines a light on a blind spot in how current AI tools handle misinformation, especially in health care,” co-corresponding senior author Girish N. Nadkarni, the chair of the Windreich Department of Artificial Intelligence and Human Health, said in the news release.
“It underscores a critical vulnerability in how today’s AI systems deal with misinformation in health settings. A single misleading phrase can prompt a confident yet entirely wrong answer. The solution isn’t to abandon AI in medicine, but to engineer tools that can spot dubious input, respond with caution, and ensure human oversight remains central. We’re not there yet, but with deliberate safety measures, it’s an achievable goal,” added Nadkarni, who is also the director of the Hasso Plattner Institute for Digital Health, and Irene and Dr. Arthur M. Fishberg Professor of Medicine at the Icahn School of Medicine at Mount Sinai and the chief AI officer for the Mount Sinai Health System.
Source: Mount Sinai School of Medicine