A new augmented reality ultrasound system from MIT lets users see real-time 3D images through a VR headset, dramatically reducing the mental effort required to interpret scans. The technology could speed up training for future healthcare providers and improve accuracy in clinical settings.
A team of researchers at MIT has developed an augmented reality system that transforms the way medical ultrasound imaging works — overlaying a live, three-dimensional rendering of scanned tissue directly into a user’s field of view through a VR headset. The technology, published June 10 in Nature Communications Engineering, could have significant implications for medical education, clinical accuracy and the next generation of healthcare professionals.
The Problem With Traditional Ultrasound
Standard ultrasound imaging produces flat, two-dimensional slices of tissue that trained technicians must mentally stitch together into a coherent 3D picture — a process that takes years to master. Lead author and MIT graduate student Jason Hou described the challenge bluntly.
“It’s a difficult skill to master, and there are long learning curves,” Hou said in a news release. “The hardest thing is this mental tomography bottleneck where you’re trained to reconstruct the 2D slices in your 3D mental space. That is a cognitive burden that can lead to inaccuracies in scanning.”
That cognitive load can mean mistakes, missed details and slower workflows — problems that affect both patients and the healthcare providers trying to help them.
How the New System Works
The MIT team’s solution, called AR-VIU (augmented real-time volumetric imaging in ultrasound), combines a compact ultrasound probe with a 3D computer graphics engine called Unreal Engine. The probe — roughly the size of a small deck of cards — captures tissue data using an array arranged in a square formation, enabling it to record volumetric images rather than flat slices. That data is compressed and streamed directly into the graphics engine, which renders a full 3D model of the scanned area in real time.
When a user wears an AR/VR headset, they see that digital rendering superimposed over the actual location of the object being scanned — functioning, as the researchers described it, like X-ray vision. Shifting one’s head or changing angle updates the view instantly, giving the user an intuitive spatial understanding of what lies beneath the surface.
Because the probe uses fewer ultrasound elements than conventional 3D systems, it also requires less power and costs less to build, addressing one of the key barriers to wider adoption of 3D ultrasound technology.
Testing It Out: Novices vs. Experts
To evaluate the system, the researchers recruited 18 participants — nine experienced ultrasound professionals, including sonographers and physicians, and nine people with no prior ultrasound experience. Each participant completed identification and localization tasks using four different imaging setups: traditional 2D imaging on a screen, 3D imaging on a screen, 2D augmented reality, and the full AR-VIU system.
Tasks included identifying objects embedded in gelatin inside opaque containers and marking the precise location of a tissue phantom — a gel material engineered to mimic human tissue — to simulate needle placement during a biopsy.
The results were striking. Across the board, AR-VIU improved performance, but the effect was especially pronounced among beginners. Novices using AR-VIU performed nearly on par with seasoned experts, closing a gap that was wide when both groups used conventional 2D imaging.
“Overlaying images with the anatomy and providing 3D visual context makes ultrasound significantly easier for novices to understand,” added co-lead author and MIT graduate student Shrihari Viswanath.
Post-experiment interviews reinforced those findings. Most novices said they preferred the AR-VIU approach and found it made tasks more manageable. Experts, by contrast, generally favored the traditional 2D method they were trained on — but acknowledged the new system’s advantages in specific scenarios like biopsy needle placement and cardiac imaging.
What It Means for Healthcare Training and Practice
Senior author Canan Dagdeviren, an associate professor of media arts and sciences at MIT, emphasized the technology’s dual promise for education and clinical care.
“For training, this could make ultrasound more intuitive and more understandable. On the clinical side, it could be less time-consuming, more accurate, and also give health care providers more peace of mind. They wouldn’t have to wonder if they missed anything,” Dagdeviren said in the news release.
Dagdeviren also highlighted the system’s user-friendly design, adding: “The 3D system imposes less brain drain, it’s more intuitive, and it’s easier to understand what is happening in the targeted region.”
Why It Matters for Students
For students pursuing careers in medicine, nursing, radiology, or any clinical field that uses diagnostic imaging, this research signals a potential shift in how foundational skills are taught. If the lengthy, cognitively demanding process of learning to interpret 2D ultrasound images can be shortened through AR-assisted training, students entering clinical rotations could arrive better prepared — and more confident — sooner in their education.
Beyond training, the system’s potential to improve accuracy during procedures like biopsies has real-world stakes for patients. A tool that reduces the gap between novice and expert performance doesn’t just benefit learners — it could improve outcomes across the healthcare system.
The research team is continuing to refine image resolution and conduct further accuracy testing.
Source: MIT