MultiARC

Input:
As input signals, three-dimensional digital microscopy and endoscopy is used. In addition, various light sources are used for the illumination of the surgical scene to capture tissue structures in different wavelengths.
To expand the surgical scene via remote interaction, the system will make  use of various 2D and 3D input devices and generate a time-synchronized augmented and mixed reality visualization.

Output:
For the output module, it has to be investigated to what extent it is possible to calculate an accurate high-resolution image by use of correlations between position, time and spectral range on all visualization devices. Nonetheless, the format agnostic imaging pipeline allows us an adaptive presentation of the underlying mutual information. Overall information related to the ongoing procedure might be shown on any external display. In addition, the binocular  or HMDs enable interactions via specific virtual objects for the communication between surgeons or for video streaming including audio comments and annotations to lecture rooms for trainee surgeons or medical students. Besides, it will be investigated whether stereoscopic displays and what type of AR/VR devices are predestined for remote telemedicine and teaching services.

Analysis:
For 3D reconstruction and surgical AR/MR applications a calibration of the stereoscopic system is crucial, in order to get a real world representation of the surgical scene and the underlying optical system. Therefore, a zoom-independent calibration scheme is defined, allowing image-based measurements resulting in metric and 3D data representation. This 3D data serves as a basis for high-level surgical AR/MR assistance functions for different surgical applications like measurements and size-correct prosthesis visualization as well as for tissue classification.

This project follows the idea of extracting multispectral tissue data, combining wide-band LEDs as an illumination unit and a RGB- camera as three-channel acquisition device. The scene will be illuminated using all available LEDs consecutively and acquire images synchronized to the illumination sequence. This illumination modality covers nearly the complete visual spectrum (approx. 400 nm to 780 nm).

For remote surgical assistance, it needs to be ensured that all surgeons can rely and work on consistent data. Furthermore, the entire pipeline needs to run in real-time without any noticeable latency. Meaning, the low-latency communication software needs to be able to handle interactive scribblings, large 3D data sets, preoperative data and vital parameters of the patient.

For any of these network scenarios, we address a low-latency interactive bi-directional communication user interface, which allows medicolegal consultations between surgical experts or detailed patient counseling before the intervention.

The project is divided into different work packages (WP 1 – WP 7), where each is dedicated to a single element of the processing chain. The project partners will contribute their expertise to different work packages and gather the results.

WP 1: Project management
WP 2: Imaging and data acquisition
WP 3: Analysis of measuring
WP 4: Multispectral tissue analysis
WP 5: Telepresence and teleconsultation
WP 6: AR/ VR visualization on different devices
WP 7: Evaluation and Integration into the clinical workflow

MultiARC @ Forum Digital Technologies

03 December 2019
Berlin, Germany

Expert Forum: AI in Minimally Invasive Surgery and Treatment

Keynote: Research in Digital Minimally Invasive Surgery at Fraunhofer HHI

Panel Discuission: The Role of AI in Digital Medicine and Surgery – Undererstimated or Overrated?

Find more information here.

MultiARC @ Miccai 2019

17-19 October 2019
Shenzhen, China

MultiARC succesfully participated at the 22^nd International Conference on Medical Image Computing and Computer Assisted Intervention in Shenzhen, China. The project contributed to the Endoscopic Vision Challenge from Intuitive Surgical and won the Runner-Up Award in the category ‘Lowest Mean Error’.

For more information, please click here.

MultiARC @ 3IT Summit

10 September 2019
Berlin, Germany

Talk: MultiARC – AI Assisted Surgery

For more information, please click here.

MultiARC Network Meeting

08 September 2019
Hamburg, Germany

Poster: MultiARC – Multimodal AR assisted in ENT surgery

MultiARC @ EMB Conference

23-27 July 2019
Berlin, Germany

At the EMB Conference 2019, MultiARC presented a stereo-endoscopic framework that allows real-time capable image-based measurements of anatomical structures. A 3D endoscope is used to reconstruct the oral cavity to measure orofacial cleft of five pediatric patients.

Endoscopic Single-Shot 3D Reconstruction of Oral Cavity
J.-C. Rosenthal, E. L. Wisotzky, P. Eisert, and F. C. Uecker

For more information, please click here.

MultiARC @ CARS 2019

18-21 June 2019
Rennes, France

MultiARC was represented in a number of talk sessions at the 33rd International Conference of Computer Assisted Radiology and Surgery.

• Talk in Session Joint ISCAS & CAD: Artificial Intelligence for Computer-aided Dagnosis and Minimally Invasive Therapy
  “A multispectral snapshot camera method to analyze optical tissue characteristics in vivo”

• Talk in Session Mixed reality for Surgical Simulation, Training and Education
  “An AR-Solution for Education and Consultation during Microscopic Surgery”

• “Introducing a zoom-independent calibration target for augmented reality applications using a digital surgical microscope”

For further information, please click here.

MultiARC @ IEEEVR 2019

23-27 March 2019
Osaka, Japan

MultiARC was presented at the 26th IEEE VR Conference on Virtual Reality and 3D User Interfaces at the IEEE VR Workshop on Applied VR for Enhanced Healthcare (AVEH).

Interactive & Multimodal-based AR for Remote Assistance using a Digital Surgical Microscope
E. L. Wisotzky, J.-C. Rosenthal, P. Eisert, A. Hilsmann, F. Schmid, M. Bauer, A. Schneider, F. C. Uecker

Fraunhofer HHI presented an interactive and multimodal-based augmented reality system for computer-assisted surgery in the context of ear, nose and throat (ENT) treatment. The proposed processing pipeline uses fully digital stereoscopic imaging devices, which support multispectral and white light imaging to generate high-resolution image data, and consists of five modules. Input/output data handling, a hybrid multimodal image analysis and a bi-directional interactive augmented reality (AR) and mixed reality (MR) interface for local and remote surgical assistance are of high relevance for the complete framework.

The hybrid multimodal 3D scene analysis module uses different wavelengths to classify tissue structures and combines this spectral data with metric 3D information. Additionally, we propose a zoom-independent intraoperative tool for virtual ossicular prosthesis insertion (e.g. stapedectomy) guaranteeing very high metric accuracy in sub-millimeter range (1/10 mm). A bi-directional interactive AR/MR communication module guarantees low latency, while consisting surgical information and avoiding informational overload. Display agnostic AR/MR visualization can show our analyzed data synchronized inside the digital binocular, the 3D display or any connected head-mounted-display (HMD). In addition, the analyzed data can be enriched with annotations by involving external clinical experts using AR/MR and furthermore an accurate registration of preoperative data. Such a collaborative surgical system will lead to a highly improved patient outcome through an easier tissue classification and reduced surgery risk.

For further information, please click here.

MultiARC @ SPIE Medical Imaging 2019

16-21 February 2019
San Diego, California, United States

SPIE Medical Imaging focuses on the latest innovations found in underlying fundamental scientific principles, technology developments, scientific evaluation, and clinical application. The symposium covers the full range of medical imaging modalities including image processing, physics, computer-aided diagnosis, perception, image-guided procedures, biomedical applications, ultrasound, informatics, radiology and digital pathology, with an increased focus on fast emerging areas like deep learning, AI, and machine learning.

Validation of two techniques for intraoperative hyperspectral human tissue determination
Eric L. Wisotzky, Fraunhofer Institute for Telecommunications Heinrich Hertz Institute, HHI (Germany) and Humboldt-Univ. zu Berlin (Germany); Benjamin Kossack, Fraunhofer Institute for Telecommunications Heinrich Hertz Institute, HHI (Germany); Florian C. Uecker, Philipp Arens, Steffen Dommerich, Charité Universitätsmedizin Berlin (Germany); Anna Hilsmann, Fraunhofer Institute for Telecommunications Heinrich Hertz Institute, HHI (Germany); Peter Eisert Fraunhofer Institute for Telecommunications Heinrich Hertz Institute, HHI (Germany) and Humboldt-Univ. zu Berlin (Germany)

For further information, please click here.

MultiARC @ Smart Data Forum

03 December 2018
Berlin, Germany

SIBB e.V. presents a monthly talk on Health-IT. Under the topic of Image Processing the researchers Jean-Claude Rosenthal and Eric Wisotzky of the Fraunhofer HHI gave a lecture on 3D analysis with measurement and hyperspectral analysis with tissue differentiation.

For more information, please click here.