#332 First-in-man AI-assisted Augmented Reality Surgery Drs. Alex Mottrie & Jasper Hofman

This is one of the 2023 KS International Innovation Awards videos selected for inclusion in the Vattikuti Foundation – ORSI Humans at the Cutting Edge of Robotic Surgery Conference, October 6, 7 & 8, 2023 in Ghent, Belgium. Posting does not imply that is has been selected as a Finalist, just that the content will be discussed at the Conference.

From the entry: Objective: Several bottlenecks withhold the integration of Augmented Reality (AR) into daily surgical practice. One key inhibitor is the instrument occlusion caused by the augmentation. As such, occlusions can negatively impact surgical safety and as such deteriorate rather than improve surgical care. In this work, we develop and test a real-time de-occlusion algorithm which segments robotic instruments during Augmented Reality by means of artificial intelligence.

Methods: A binary segmentation model was trained on a dataset of 31812 images, sampled from 100 RAPN full-length RAPN videos. The model was subsequently used in a pipeline which simultaneously allows for 3D model rotation and alignment with the surgical scene. The application was tested during during three live surgeries: vascular stent removal, liver metastasectomy and partial nephrectomy.

Results: The pipeline shows robust real-time binary segmentation of 37 non-organic surgical items, which are never occluded during AR. The end-to-end latency was measured at 13 milliseconds. Figure 1 depicts the current state-of-the-art in AR in the top line, compared to our AI-assisted segmentation in the bottom line. Figure 1: Toggling on/off instrument occlusion during endoscopic echography for all 3 cases.

Conclusion: We perform the first-in-man AI assisted instrument de-occlusion during Augmented Reality Surgery. The pipeline is robust and can improve surgical safety and ergonomics. It paves the way towards acceptance of AR in minimally invasive surgery.

See more at: https://vattikutifoundation.com/videos/