3IT Project TeleSTAR: First Time Broadcast of Charité Surgery as AR-based 3D-Live Stream with Live Annotations

Berlin, 26 September 2020 – Our 3IT Partner Project TeleSTAR has transmitted the surgical insertion of a cochlea implant for the first time using an augmented reality based 3D live stream. 35 participants from TU Delft in the Netherlands, including ENT physicians and master students of the faculty “Biomedical Engineering”, were able to follow the surgery live at the Charité Berlin for teaching purposes. Importantly, the 3D live stream could be supplemented by simultaneous audio commentary and an intraoperative annotation mode using AR. The intraoperative annotation mode allows surgeons to add additional visual information such as sketches, references and image-based anatomy measurements to the live stream. In this way, every step of the surgery is explained and comprehensible. While live surgeries have long been part of medical education, innovative AR approaches offer new teaching methods.

The fully digital surgical microscope ARRISCOPEwas used for the visualizations. It enables high-resolution 3D images, a radiation-free and image-based measurement of the anatomy as well as the acquisition of multispectral image data.

“Especially in the current COVID 19 pandemic situation, this type of communication is the most appropriate and useful form of distance learning and communication,” commented Dr. Florian C. Uecker, Senior Consultant at the ENT Clinic of the Charité Berlin. “I am very glad that EIT-Health is supporting us in this project and that with the ARRISCOPE we have the world’s leading digital 3D microscope at our disposal”.

“Augmented reality using AI and computer vision algorithms in combination with digital processing chains are the key technologies for new intraoperative assistance instruments and the promotion of education through a new level of surgical transparency,” explains Dipl.-Inf. Jean-Claude Rosenthal, research associate at Fraunhofer HHI.

The surgical live stream lasted 90 minutes and was transmitted without interruption in a 10Mbit 3D stream including AR and depth visualization. TeleSTAR develops novel training concepts for clinical staff using mixed reality (MR) and AR-based approaches.

TeleSTAR is funded by the EIT-Health. EIT-Health is supported by the EIT, a body of the European Union.

Find all information on our 3IT-Project TeleSTAR here.

Find the Pressrelease to the 3D live surgery here.

Here are some impressions: