Artificial intelligence can now match physicians on clinical reasoning tasks, but researchers say strong benchmark scores are no substitute for rigorous safety evaluation and governance before these tools reach real patients.
Artificial intelligence systems are getting remarkably good at thinking through medical problems — in some cases keeping pace with experienced physicians. But a new expert commentary published in Science argues that impressive test scores are not the same as safe, effective patient care, and that health care AI is advancing faster than the guardrails designed to govern it.
Researchers at Flinders University in Australia reviewed recent evidence showing that advanced, reasoning-based AI models can work through diagnostic scenarios step by step, matching or even surpassing the diagnostic accuracy of trained doctors. The commentary, titled “AI can reason like a physician; what comes next?,” calls these developments genuinely promising — while sounding a clear alarm about premature deployment.
“AI systems have demonstrated that they can reason through clinical problems with similar performance to doctors, notably on the same scenarios used to train clinicians themselves. This presents genuine opportunities to support clinicians in the future,” senior author Ash Hopkins, an associate professor in the College of Medicine and Public Health, NHMRC Investigator and leader of Flinders’ Clinical Cancer Epidemiology Lab, said in a news release.
But the research team stresses that real-world medicine involves far more than answering text-based questions correctly. Physical examinations, listening to patients, interpreting social and medical context, and bearing professional accountability for outcomes are all essential elements that current AI systems cannot independently provide.
“Health care decisions are complex, high stakes, and deeply human, and accuracy alone, particularly on just text-based cases, does not make a system safe for patients,” co-author Erik Cornelisse, a doctoral candidate in Flinders’ College of Medicine and Public Health, said in the news release.
The Risks of Moving Too Fast
The commentary points to a well-documented historical pattern: algorithms deployed without sufficient testing can make outcomes worse, not better. Bias in training data, gaps in representation, and lack of real-world validation have all contributed to harm in previous cases where automated tools were rushed into clinical use.
“History shows that algorithms can worsen outcomes when deployed without sufficient safeguards and can amplify problems as easily as they solve them, particularly when systems are trained on incomplete or unrepresentative data,” Cornelisse added.
The Flinders team also notes that the legal, ethical and professional accountability structures surrounding medical AI are still being worked out. Who is responsible when an AI-assisted diagnosis goes wrong — the developer, the hospital, the physician? Those questions remain largely unanswered.
“Multiple stakeholders are currently working on the frameworks for AI in terms of legal, professional, or moral responsibility for its decisions, and presently there is a critical need for deliberate and controlled integration into clinical care,” Hopkins said.
Why It Matters for Students and Future Health Professionals
For students studying medicine, nursing, public health, or health informatics, these findings carry direct relevance. The next generation of clinicians will almost certainly work alongside AI tools — and understanding their limitations is just as important as knowing how to use them. The commentary argues that AI should be held to the same standards of supervision and evaluation as any human practitioner.
“We do not allow doctors to practise without supervision and evaluation, and AI should be held to comparable standards,” said Cornelisse.
Beyond the clinic, the debate over medical AI governance is shaping emerging careers in health policy, medical ethics, regulatory science, and health technology assessment — fields that are growing rapidly as these tools become more prevalent.
The Path Forward
The researchers are not calling for a halt to AI development in health care. Rather, they argue that enthusiasm must be paired with rigorous evaluation frameworks that measure what actually matters: improvements in real patient outcomes, not performance on standardized exams or curated datasets.
“Patients deserve technology that improves care in the real world, not systems that only look impressive in studies,” Hopkins added.
The team’s vision for responsible adoption is optimistic but conditional.
“With careful design, strong oversight, and rigorous evaluation, AI could become a powerful tool to deliver safer, fairer, and more effective care across health systems worldwide,” Hopkins concluded.
Source: Flinders University