An MIT-led team has outlined a way to build “humble” medical AI systems that know when to pause, ask for more data and defer to human judgment. The goal is to turn AI from an oracle into a co-pilot that supports, rather than overrides, clinicians.
Artificial intelligence is rapidly moving into hospitals and clinics, promising faster diagnoses and more personalized treatments. But as AI tools become more powerful, researchers are warning that they can also become dangerously overconfident.
An international team led by MIT has now proposed a way to build medical AI systems that are not just smart, but “humble” — able to recognize when they are uncertain, flag their own limits and prompt clinicians to gather more information before acting.
The researchers argue that this shift in design could help prevent doctors from being steered in the wrong direction by AI systems that sound sure of themselves even when they are wrong. Instead of acting as black-box oracles, future systems would function more like thoughtful collaborators.
The goal is to change how clinicians relate to AI, according to senior author Leo Anthony Celi, a senior research scientist at MIT’s Institute for Medical Engineering and Science and a physician at Beth Israel Deaconess Medical Center, and an associate professor at Harvard Medical School.
“We’re now using AI as an oracle, but we can use AI as a coach. We could use AI as a true co-pilot. That would not only increase our ability to retrieve information but increase our agency to be able to connect the dots,” Celi said in a news release.
The new framework, published in BMJ Health and Care Informatics, lays out how developers can embed curiosity and humility into clinical decision support tools.
The team’s starting point is a problem that has already shown up in intensive care units and other high-stakes settings: when AI systems appear authoritative, doctors and patients tend to trust them, even if the recommendation clashes with clinical intuition.
Previous studies have found that ICU physicians sometimes defer to AI suggestions they see as reliable, despite their own doubts. That can be especially dangerous when the system is confidently wrong.
To counter this, the MIT-led group wants AI to behave less like a final decision-maker and more like a partner that supports human judgment. The aim is to design systems that keep people at the center, according to lead author Sebastián Andrés Cajas Ordoñez, a researcher at MIT Critical Data, a global consortium based in MIT’s Laboratory for Computational Physiology.
“We are trying to include humans in these human-AI systems, so that we are facilitating humans to collectively reflect and reimagine, instead of having isolated AI agents that do everything. We want humans to become more creative through the usage of AI,” Cajas Ordoñez said in the news release.
At the core of the proposed framework is a set of computational modules that can be added to existing AI models.
The first module forces the AI to evaluate how sure it really is when it makes a diagnostic prediction. Developed by consortium members Janan Arslan and Kurt Benke of the University of Melbourne, this component — called the Epistemic Virtue Score — acts as a kind of self-awareness check. It compares the model’s confidence with the actual uncertainty and complexity of the clinical situation.
If the system detects that it is more confident than the available evidence justifies, it does not simply push ahead with a firm recommendation. Instead, it can pause, flag the mismatch and suggest next steps, such as ordering specific tests, gathering more patient history or calling in a specialist.
In other words, the AI is designed not only to produce answers, but also to signal when those answers should be treated with caution.
Celi likens this to having another expert in the room.
“It’s like having a co-pilot that would tell you that you need to seek a fresh pair of eyes to be able to understand this complex patient better,” he said.
The team is now working to implement this framework in AI systems trained on large, real-world hospital databases that Celi and colleagues have helped build, including the widely used Medical Information Mart for Intensive Care (MIMIC) dataset from Beth Israel Deaconess Medical Center. They plan to introduce these “humble” AI tools to clinicians across the Beth Israel Lahey Health system.
The approach, the researchers say, could be applied to a range of medical AI tools, from systems that read X-rays and other images to models that help emergency room teams choose among treatment options.
The work is also part of a broader push by Celi and his collaborators to make AI in health care more inclusive and socially aware.
Many influential medical AI models are trained on publicly available U.S. data, which can embed a narrow view of disease and care. Patients who lack access to major hospitals — including many people in rural or underserved communities — may be missing from the datasets altogether. That can lead to tools that work well for some groups and poorly for others.
On top of that, most diagnostic AI systems are built on electronic health records, which were never designed for this purpose. These records often lack the rich context clinicians use in real life, such as social factors, patient preferences and nuanced clinical reasoning.
To address these gaps, MIT Critical Data runs workshops that bring together data scientists, clinicians, social scientists, patients and other stakeholders to co-design AI systems. Before anyone starts modeling, participants are asked to interrogate the data itself: what is missing, who is left out and how that might skew the results.
Celi emphasized this questioning is essential.
“We make them question the dataset. Are they confident about their training data and validation data? Do they think that there are patients that were excluded, unintentionally or intentionally, and how will that affect the model itself?” he said.
By combining technical safeguards like the Epistemic Virtue Score with social processes that surface bias and blind spots, the researchers hope to build AI that not only performs well on benchmarks but also earns trust in real-world care.
They stress that slowing down AI development is not the goal. Instead, they want to change how it is done.
“Of course, we cannot stop or even delay the development of AI, not just in health care, but in every sector. But, we must be more deliberate and thoughtful in how we do this,” Celi added.
For now, the framework offers a roadmap for turning AI from a rigid authority into a reflective partner — one that knows when to speak up, when to stay quiet and when to ask for help. If widely adopted, that shift could make future medical AI not only more powerful, but also more humane.