SCIEMENT tackles congenital heart defects with Sony’s Spatial Reality Display
The SCIEMENT Viewtify solution transforms a series of medical images into a 3D computer graphic shape that doctors can examine as a surface in real-time, providing unprecedented, intuitive understanding.
How we helped
The challenge
- Enable pediatric cardiologists and cardiac surgeons to better visualize congenital heart defects.
- Empower neurosurgeons to better understand the location of cerebral aneurisms.
- Enable radiologists to better see the branching structure of complex blood vessels.
The solution
- Transform DICOM-standard 2D medical imaging into 3D computer graphics that can be manipulated, rotated, enlarged, and cut by any plane in real-time.
- Display the images in a glasses-free volumetric display that responds to head motion like a solid 3D object, with Sony’s Spatial Reality Display.1,2.
- Expected to help enable doctors to better plan treatment and test surgical cuts in 3D.3.
*1 Computer required with a recommended CPU of Intel Core i7-9700K @3.60GHz or faster; and a graphics card such as NVIDIA GeForce RTX 2070 SUPER or faster.
*2 See instruction manual accompanying the product for details on product use, incl. 3D feature.
*3 Viewtify and Sony’s Spatial Reality Display are not medical devices and to be used for research and educational purposes only.
The outcome
- Sony’s Spatial Reality Display provides much faster, more convenient 3D modeling than 3D printing.
- Enables highly accurate, impressively detailed, glasses-free 3D medical visualization.
- Expected to help set the stage for better patient outcomes.3
Quotes:
“This naked-eye stereoscopic solution will revolutionize the medical field.”
- Dr. Hirofumi Seo, M.D., Ph.D., President and CEO, SCIEMENT, Inc.
The challenge of 2D images
CT, MRI and echography are indispensable imaging tools that help guide medical practice. But they all generate 2D images of 3D subjects. According to Dr. Hirofumi Seo, president of SCIEMENT, “In pediatric cardiology, it’s difficult to imagine the accurate 3D structure from such images. And even when 3D computer graphics are synthesized, doctors are still looking at 2D displays, making it difficult to understand depth.”
SCIEMENT’s Viewtify solution is poised to transform medical 3D imaging. The system uses the Unreal Engine 4 computer graphics software to synthesize multiple 2D images into a 3D surface shape, which doctors can examine on Sony’s Spatial Reality Display. Says Dr. Seo, “Sony’s glasses-free, volumetric 3D display can show depth intuitively, so it’s easy to understand even complex structures.”
The power of 3D display
Starting from contrasted cardiac CT images, Viewtify and Sony’s Spatial Reality Display can help doctors understand the detailed anatomy of the infant heart. “Usually, congenital heart defects are classified by looking at echocardiography, but it is often difficult,” says Dr. Seo. “With the Viewtify solution, it’s very easy to classify each defect. And we can understand that even when the classification is the same, the actual shape or structure is different in each case.”3
The solution enables doctors to manipulate the 3D image in the computer, rotating, translating, zooming in or out, and make a cross section by any plane, all in real-time4. In addition, doctors can “look around” the 3D image on Sony’s Spatial Reality Display as if examining a solid object. Dr. Seo says, “With the Spatial Reality Display’s high resolution and eye tracking, we can really understand intracardiac structures.”
Dr. Seo reports tremendous interest among pediatric cardiologists and pediatric cardiac surgeons. Other potential applications include confirming the intervertebral gap for thoracic epidural anesthesia, understanding the shape of bone fragments in compound fractures, viewing the complex shape of blood vessels in cerebral aneurisms, plus forensic medicine and medical education.
*4 Recommend use of “high resolution, quality images” created using Unity or Unreal software.
3D display versus 3D printing
In medical practice and medical education, 3D display also offers practical advantages over 3D printing. “With 3D display, you can adjust the cross section, the look and the scaling as needed,” says Dr. Seo. “On a 3D printout, we can’t undo a cut and re-cut at another place. But with Sony’s Spatial Reality Display, we can (virtually) cut as many times as we want.” In addition, compared to 3D printing, 3D images are faster, consume no costly materials, need no physical storage, and lend themselves to making figures for papers and textbooks.
Engineering the solution
Dr. Seo considers 3D viewing to be a supplement, not a replacement for 2D medical monitors, such as the Sony surgical model display (LMD-X2705MD) that he uses side-by-side with the Spatial Reality Display. The Spatial Reality Display can only be viewed by one person at a time and is not designed for 2D medical images. “Medical doctors will always want to look at DICOM images at their very best,” Dr. Seo says.
In addition to his medical training, Dr. Seo has deep expertise in computer graphics. His visualized video “Multi-Scale Multi-Physics Heart Simulator” won the award for Best Visualization or Simulation at the international SIGGRAPH 2015 Computer Animation Festival. That was also the year he started working with Unreal Engine 43. The Viewtify solution converts DICOM-standard medical images into Unreal Engine 4 content. He says, “Getting 3D computer graphics onto Sony’s Spatial Reality Display was the easy part. You also need appropriate shading and lighting. And the real challenge was generating simultaneous versions for the 2D and 3D displays. The results were worth the effort. This naked-eye stereoscopic solution will revolutionize the medical field.”