Augmented and Virtual Reality – Radiology at the Center of Medical Technology

Augmented and virtual reality are among some of the newest technologies to reach a sophistication viable for medical use. Augmented reality (AR) is the integration of virtual components onto the background of reality, while virtual reality (VR) is a complete and isolated virtual display.

The most popular utilization of AR and VR is in entertainment devices, such as video games, but the versatility and responsiveness of the technologies have allowed for exciting developments in healthcare, as well. In some ways, it can be said that interventional radiologists have been using these techniques for their entire existence. In essence, anytime a fluoroscopic projection is used during a procedure, that is technically an augmentation of reality used to help visualize internal structures in real time.

Utilizing recent AR and VR technological advances, radiographic data can now be used to create a display with more integration and interaction. In addition to its utility in image-guided procedures, AR and VR have also found niche roles in the education of trainees and for direct benefits to patients, as well, though the technology is not without its drawbacks.

One of the most promising uses of AR technology, specifically, is procedural planning and guidance. In the operating room, AR displays are generated using advanced algorithms that interpret CT data into a 3D map of structures delineated by density (Tang et al.). This map can, then, be displayed in several ways, such as on a screen nearby, through a headset, or projected onto the patient themselves depending on the needs of the procedure. Then, a plan can be drawn onto the augmented surgical field before the first cut.

Not only can structures, such as vessels and tumors, be overlaid onto a display of the surgical field, but organ movements and deformations can be accounted for in real time to update the display. This gives some compensation for the lack of tactile feedback during laparoscopic procedures and allows for a better real-time visualization of underlying structures. Tang and colleagues demonstrated the value of these displays for hepatobiliary surgery, but these techniques can be applied to just about any surgical procedure that can be planned far enough in advance.

AR projectors can already integrate navigation aids in MRI-guided procedures to assist in carrying out the procedure such as Mewes et al. describes. Wherever radiographic information exists before or during a procedure, AR stands to improve the integration of that information in the OR, and radiologists are the gatekeepers of this technology with their knowledge of medical informatics and mastery of radiologic anatomy.

With AR and VR already in the public consciousness as powerful tools of media consumption, it should come as no surprise that they have been quickly modified to work as compelling interactive forms of education. Progress has been made in developing AR tools for the advance of “tele-mentoring,” the use of displays to mentor trainees at long distance (Andersen et al.). AR allows for mentors to annotate directly on the student’s display so the trainee is no longer required to shift attention between a separate screen and their work and the training procedure can be done collaboratively. This system would also allow a more experienced interventional radiologist or surgeon to assist one doing an unfamiliar procedure without the need for transportation of the expert to the patient.

VR works as a training tool similarly to how it is used to consume media: allowing for a virtual simulation of a subject rather than just a video. New software allows for VR headsets to display a procedure to trainees to observe from anywhere. These students can see the surgical field from a 360-degree view or from the surgeon’s own perspective as each step is narrated, providing more immersion and detail than any 2D animation or lecture could. Interventional radiologist Dr. Ziv Haskal is pioneering this technique to teach complicated trans-jugular intrahepatic portosystemic shunt (TIPS) placement surgery (Barney). He demonstrates that even the most complicated and difficult-to-shadow surgeries could be shared nationwide and simulated through simple VR sets as a powerful teaching aid.

VR sets also have the potential to directly benefit the patient through simulated education and therapeutic benefit. Amazingly, therapeutic VR simulations have been found to reduce a self-reported pain score especially among those with more severe initial pain (Spiegel et al.). This represents an alternative to pharmacological therapy thought to work by distracting from nociceptive perception by fully engaging the other senses. Furthermore, VR can be used to walk a patient through their day at the hospital from check-in to procedure.

Dr. Haskal demonstrated how VR can be used to prep a patient for the same procedure he was teaching before -- a TIPS placement -- with the viewer getting checked in, visiting with their doctor, and being taught the procedure through the VR medium (Davis). Simple VR simulations like these can be programmed into a smartphone that many patients already have and viewed with the addition of a foldable cardboard headset, allowing for these sessions to be conducted in the patient’s own home.

Unfortunately, a technology this new to the field comes with some limitations. AR specifically requires a more robust framework in place compared to VR. The tele-mentoring system suffers from a few significant draw backs: a frame rate which limits the system’s ability to accept annotations from a proctor, the current model can only relay still frames for communication, and audio communication was not integrated (Andersen et al.). Additionally, processing power is a constraining factor especially if a projection is desired due to the increased demand necessary to account for the changes in topography when the field distorts (Tang et al.). And with any additional piece of machinery comes the opportunity for malfunctions and the need for supplementary technical staff and funding.

There is also a significant buy-in in terms of costs and burden of learning for the software. Improvements are needed to make the systems more user-friendly before AR and VR are appealing to the less technologically inclined. There are also limitations to haptic realism in the programs and some develop motion sickness and cannot even use them. Despite all this, as the software becomes more robust and user-friendly, the versatility of AR and VR will quickly overcome its weaknesses.

AR and VR are the future of technology-integrated medicine. Using projectors or headsets to simulate internal structures or peek into a procedure in VR are things that seem in place in a science fiction novel, but that reality is closer than ever. The sheer utility of AR annotation in procedural planning and visualization of radiologic information in real-time is something that will drive the technology forward despite its current drawbacks. The field of radiology has many avenues to apply these technologies, with its wealth of procedures using radiographic displays and the benefit of AR displays to non-invasive surgeries.

With a variety of current applications including patient and trainee education, as well as procedural planning and guidance, the future directions that AR and VR can take medicine in are as varied as remote-assisted home treatments for patients and VR curriculums over specific chronic diseases for patients and students alike. The technology has the potential to integrate with many aspects of the daily medical procedures and medical education, and radiologists are in a position to take those steps first.

References

1. Andersen D, Popescu V, Cabrera ME, et al. Virtual annotations of the surgical field through an augmented reality transparent display. The Vis Comp. 2015;32(11):1481-1498.

2. Barney J. Virtual Reality Puts Physicians, Trainees – Even You – Right in the Operating Room. 2018, October 19 Retrieved from https://news.virginia.edu/content/virtual-reality-puts-physicians-trainees-even-you-right-operating-room

3. Davis J. Going virtual: How VR is guiding interventional radiology. 2018, September 18  Retrieved from https://www.elsevier.com/connect/going-virtual-how-vr-is-guiding-interventional-radiology

4. Mewes A, Heinrich F, Hensen B, et al. Concepts for augmented reality visualisation to support needle guidance inside the MRI. Health Tech Let. 2018;5(5):172-176.
5. Spiegel B, Fuller G, Lopez M, et al. Virtual reality for management of pain in hospitalized patients: A randomized comparative effectiveness trial. Plos One. 2019;14(8).

6. Tang R, Ma LF, Rong ZX, et al. Augmented reality technology for preoperative planning and intraoperative navigation during hepatobiliary surgery: A review of current methods. Hepat & Panc Dis Intl. 2018;17(2):101-112.