The 3IT provides a patform for its partneres to see what’s new concerning everything immersive media as well as to come together and exchange new ideas, technological updates and develop newproject plans.
In order to overcome the limitations we all face during the past months, the 3IT has started to produce expert interviews, in which you can learn more about 3IT related topics.
3IT Project TeleSTAR Expert Interview: AR-Based 3D-Live Surgery Transmission
Expert Interview “Environment – the opening project of MaerzMusik” filmed with OmniCam & teases about the new 3IT project Virtual LiVe
Streaming of Interactive Volumetric Video – Expert Talk about the Upcoming Project at 3IT – VoluProf
The scalable, mirror-based multi-camera system OmniCam-360 is a Fraunhofer HHI technology that allows the recording of live video in360 degree panoramic format & 3D-video and -audio content. Its generated videos are optimal for immersive applications!
Due to the Corona pandemic, the opening project was not possible live at the Haus der Berliner Festspiele. However, with the OmniCam-360, a unique and immersive visual experience was created.
The project “Environment” explored the special pandemic situation and was created accordingly: The musicians spent 84 days together in quarantine and created a show that the audience was able to listen to live from their homes.
To find out how the OmniCam-360 works, contact us and schedule a meeting or a guided tour in the 3IT Showroom in Berlin!
On 23 March 2021, 3IT project manager Maria Ott presented our partner network at this year’s Innovatour Congress. The congress was dedicated to everything surrounding the latest innovations from the fields of technology, mobility, life science, fintech and insurtech, with the goal to bring together smaller and larger companies and players working in tech. From 23 to 26 March, the participants came together from diverse areas of the tech industry, ranging from young startups and innovation hubs, to large companies and research institutions.
On the first of four congress days, everything related to “Consumer Electronics” was the topic of discussion. Maria Ott gave a short insight into the Fraunhofer Heinrich Hertz Institute and its research fields and then focused on introducing the 3IT with its showroom, partners, projects, aims and the displayed cutting-edge demonstrators in the 3IT facilities.
In the presentation of projects and technologies at the 3IT, Maria Ott spoke about 360 degree panorama video and audio production, the most recent 3IT project TeleSTAR, which focuses on AR-based 3D live broadcasting from the ER, and the Streaming of Interactive Volumetric Video Production and gave an outlook about future projects.
In addition to 3IT and other companies, there were 3IT partners such as Sony and Fraunhofer IIS attending and presenting their projects as well. Innovatour’s goal of bringing together global technology companies and innovative startups, thus providing the opportunity to network and present interesting technologies was certainly a success! Many questions have been asked and answered and new connections have been made. The 3IT-Team is happy to have been able to join this exciting new event format!
For today’s Holocaust Remembrance Day, the 3IT would like to remind of a special project of the partners Fraunhofer HHI and Volucap: The first volumetric interview with a contemporary witness of the Holocaust, filmed in 2019 in the Volucap studio in Potsdam-Babelsberg and titled “ERNST GRUBE – THE LEGACY”.
The short film has been realized as a cooperation of UFA, under the direction of UFA Technology head and producer Ernst Feiler as well as UFA SHOW & FACTUAL producer Philipp Grieß, together with the expert Dr. Oliver Schreer, Head of Immersive Media & Communication group at Vision & Imaging Technologies department at Fraunhofer Heinrich Hertz Institute.
As the last survivors of the horrors of National Socialism are aging, thus it becomes more and more important to find solutions to preserve their voices, in order to keep them alive also for subsequent generations. Holocaust survivor Ernst Grube’s account has been recorded with the help of volumetric video technology and its underlying technology ‘3D Human Body Reconstruction’, which was developed by Fraunhofer HHI. In Volucap GmbH’s volumetric studio, which has made a name of its own, Ernst Grube narrates his personal experience in NS-Germany and his imprisonment in the concentration camp Theresienstadt. Through 16 camera pairs as well as an important amount of computer power, a lifelike three-dimensional depiction of Ernst Grube’s witness report was created, allowing for a direct integration in a virtual world.
The portrayal of Ernst Grube does not only enable the maintenance of a singular voice, it even approaches this heavy topic as a virtual reality experience. This is in particular relevant to reach also a younger audience, to whom the NS’ atrocities and the Holocaust might seem distant. The short film “Ernst Grube – the Legacy” is thus a pivotal contribution to a vivid German Culture of Remembrance.
The intersection of cutting-edge technology and social relevant topics further proves the relevance of immersive imaging technologies and calls for new perspectives on how the respective technology can contribute to society. Find out more about “Ernst Grube – The Legacy” in this interesting talk.
Berlin, 15 November 2020 – For the second time, 3IT’s partner project TeleSTARsuccessfully transmitted the surgery of a cochlear implantation using an augmented reality based 3D live stream from ENT clinics of Berlin’s renowned hospital university Charité. This time, the stream was broadcasted to four lecture halls Europe-wide, hence, it permitted 45 viewers to witness the surgery live. Students from the ENT department of the hospital ErasmusMC in Rotterdam, one of the most prestigious research hospitals in Europe, as well as from the living lab of trauma surgery in the University hospital of Ludwig-Maximiliam University in Munich were able to follow the surgery for teaching purposes live with a latency of 800ms. Moreover, other surgeons from the field of visceral and transplant surgery as well as medical master students could survey the surgery in 3IT’s 3D-cinema, equally with a latency of 800ms. The additional broadcast to Charité’s Intranet gave students the advantage to observe the surgery live with a latency of mere 20ms. Subsequent to the live streaming, 67 viewers watched the 2D YouTube stream, with a latency of 8s.
This time, an improved audio concept helped to relief the operating senior surgeon, Dr. Florian Uecker. Once again, the 3D stream was supported by synchronous audio commentary and an intraoperative annotation mode using AR. Thus, every single step was conveyed by explanations, making them more comprehensible for the students. By means of the intraoperative annotation mode, surgeons were able to append visual information to the live stream, in the form of sketches, references and anatomy measurements. Even more, the students had the opportunity to ask questions from the various remote lecture halls to the second assistant surgeon Dr. Sophia Häusler.
As the OR is a time and resource constrained environment, live surgeries enjoy a long tradition within medical education, already. However, the integration of innovative AR approaches, facilitates groundbreaking new teaching methods. The live transfer did not solely enable a larger public to monitor the surgery live. In fact, in times of Covid19, this new format of teaching becomes pivotal, rendering possible high quality distant learning without putting anybody at unnecessary risk.
The insertion of a cochlear implant is a standard operation in ENT surgery, in order to treat deaf people. The implementation is advantageous in order to demonstrate the use of AR-technology and to test it, as the surgery can be made within a short time. The fully digital surgical microscope “ARRISCOPE” of the company Munich Surgical Imaging (formerly ARRI Medical) was used for the visualizations, as it renders possible high-resolution 3D images, a radiation-free and image-based measurement of the anatomy and the acquisition of multispectral image data.
TeleSTAR is funded by the EIT-Health. EIT-Health is supported by the EIT, a body of the European Union.
Find all information on our 3IT-Project TeleStar here.
Find the press release to the 3D live surgery here.
Expert opinions
Dr. R.M. (Mick) Metselaar Mick, ENT surgeon at Erasmus MC: “I’m definitely hoping for a follow-up.”
Prof. Dr. Reiner Kunz, visceral surgeon, Berlin-Tempelhof: “Congratulations on this absolutely impressive demonstration of the Arriscope System on the occasion of cochlear implantation. In my opinion, extremely good image, excellent 3-dimensionality and the additional attributes such as ROI markings, tissue recognition etc. were demonstrated very effectively and in my opinion already very well suited for routine operation.”
Prof. Dr. John v.d. Dobbelsteen, faculty of Biomedical Engineering at TU Delft: “Students and clinical trainees need to experience the full spectrum of modern medical technology. TeleSTAR offers surgical trainees and medical technology developers this important exposure”
N.N., Biomedical Engineer, TU Delft “Amazing experience to observe a surgery like this. I found it very useful to be able to ask the surgeon exactly where we should be focusing on, this was only further helped by the ability for the surgeons to point out exact locations using the annotation mode.”
Dr. med. Florian C. Uecker, leading senior physician, ENT clinic at Charité Berlin, Virchow-Campus “This technology provides an unparalleled quality of remote teaching and communication, which is particularly valuable given the current COVID-19 pandemic”
Dr. med. Steffen Dommerich, leading senior physician, ENT clinic at Charité ,Campus Mitte “A surgical broadcast with all this additional information makes the whole thing appear much more descriptive and increases the learning experience.”^
Dr. med. Philipp Arens, ENT senior physician, ENT clinic at Charité, Campus Mitte “A fascinating system, which gives a deep insight into the patient that is almost the same as if you were operating yourself”.
Dr. PM Ute Morgenstern, Biomedical Engineer, TU Dresden: “This is really a very good way to provide a realistic environment for medical students far away from the surgical location, especially if the analogue training possibilities are now limited due to Corona. Above all – what interests us as biomedical engineers – the possibility to convey technical information about the medical environment in such a way that engineering students or practitioners can see exactly the interface between their research/development and the clinical user and can therefore approach their own work with more knowledge, motivation and criticism.”
From 11 to 12 November 2020, the first hybrid version of the annual Mediatech Hub Conference MTH took place – produced live in Babelsberg’s studios and accessible through an online platform. The MTH Conference is Germany’s only business-to-business event dedicated to media technology, examining the potential, the applications and challenges of MediaTech in the context of both economy and entertainment.
During the two conference days, more than 800 visitors participated at the #mthcon2020’s interactive online platform. Attendees could experience a program amounting to 30 hours with more than 75 speakers on three stages supplemented by workshops on the production of series, metadates, micro mobility and digital ecosystems. In the Startup Pitch Arena, innovative Media-Tech-Startups were able to present their business models, while the Cinemalovers Cinema showed six curated movies.Networking was made possible in the Networking Carousel.
The MTH’s opening talks threw a glance behind the scenes of the technology sector, taking into consideration, how humans could use media technology, in order to create a fair and human society. The conference’s first day was inaugurated by Murat Sönmez, director of the Center for the Fourth Industrial Revolution at the World Economic Forum, demanding for a “transparent and integrative way” in order to fully realize “the potential of the Fourth Industrial Revolution”. Kenza Ait Si Abbou, author and Senior Manager Robots and AI at 3IT’s partnerDeutsche Telekom, launched the MTH’s second day with a presentation on how Artificial Intelligence is biased and thus reproduces racism.
Central topics on the entertainment stage picked up the film industry’s digitization, which has been shown to be accelerated by the pandemic. While the crisis due to Corona is challenging the sector in a radical manner, it has been met with creative visions, according to the speakers.
The Industry Stage of the MTH highlighted the potential of media technology in the domains of mobility, health, the event sector and industry 4.0, in particular Virtual Reality and Artificial Intelligence. According to Michel Reilhac, curator of the Venice Biennale’s VR competition, the future lies in ‘phygital’ events that is hybrid formats combining physical and virtual. Moreover, in the E-Health-Track, health experts discussed the potential of immersive technologies to support both the treatment of patients and the fight against pandemics. Their conclusion was that health technologies had to be designed patient-centered.
In the Track Mobility, Kirsten Degner of the Digital Media Hub Mobility gave insights on the studies of micro-mobility, while Marius Swoboda of Rolls-Royce switched live into the VR-Cave of the Berliner Werk, in order to demonstrate how industrial processes can be optimized thanks to immersive technologies.
Berlin, 26 September 2020 – Our 3IT Partner Project TeleSTAR has transmitted the surgical insertion of a cochlea implant for the first time using an augmented reality based 3D live stream. 35 participants from TU Delft in the Netherlands, including ENT physicians and master students of the faculty “Biomedical Engineering”, were able to follow the surgery live at the Charité Berlin for teaching purposes. Importantly, the 3D live stream could be supplemented by simultaneous audio commentary and an intraoperative annotation mode using AR. The intraoperative annotation mode allows surgeons to add additional visual information such as sketches, references and image-based anatomy measurements to the live stream. In this way, every step of the surgery is explained and comprehensible. While live surgeries have long been part of medical education, innovative AR approaches offer new teaching methods.
The fully digital surgical microscope “ARRISCOPE” was used for the visualizations. It enables high-resolution 3D images, a radiation-free and image-based measurement of the anatomy as well as the acquisition of multispectral image data.
“Especially in the current COVID 19 pandemic situation, this type of communication is the most appropriate and useful form of distance learning and communication,” commented Dr. Florian C. Uecker, Senior Consultant at the ENT Clinic of the Charité Berlin. “I am very glad that EIT-Health is supporting us in this project and that with the ARRISCOPE we have the world’s leading digital 3D microscope at our disposal”.
“Augmented reality using AI and computer vision algorithms in combination with digital processing chains are the key technologies for new intraoperative assistance instruments and the promotion of education through a new level of surgical transparency,” explains Dipl.-Inf. Jean-Claude Rosenthal, research associate at Fraunhofer HHI.
The surgical live stream lasted 90 minutes and was transmitted without interruption in a 10Mbit 3D stream including AR and depth visualization. TeleSTAR develops novel training concepts for clinical staff using mixed reality (MR) and AR-based approaches.
TeleSTAR is funded by the EIT-Health. EIT-Health is supported by the EIT, a body of the European Union.
Find all information on our 3IT-Project TeleSTAR here.
Find the Pressrelease to the 3D live surgery here.
Volumetric 3D images and anatomical models are playing an increasingly significant role in medical training, complex surgery planning and in the communication of information to patients. The combination of the 3D PluraView and Vesalius3D from PS-Medtech and a laptop or tower-format station presents a truly affordable 3D workplace solution for doctor’s clinics and hospitals. 3IT’s long-time partner Schneider Digital has developed a ground-breaking benchmark for 3D-stereo visualization in medical applications: The 3D PluraView with Vesalius3D stereo monitors. Combining dual screens and a passive beam-splitter mirror, the 3D PluraView provides a realistic and high-resolution 3D experience, tantamount to a detailed holographic representation. Supported by the large 27” and 28” PluraView monitors in 2.5K and 4K (UHD) resolution, the 3D PluraView offers both, a superior viewing area as well as high brightness and contrast.
Vesalius3D is an easy-to-handle new software application for high-quality 3D visualisation and navigation through personalized, patient-specific, anatomical structures. Through the stereo 3D, Vesalius3D is adding in depth perception to an otherwise flat perspective. Allowing 3D visualisation on a single screen, it enables to “fly” through a patient’s vessels to explore them in detail. An integrated infrared tracking system ties live stereo rendering of 3D volume data to intuitive 3D navigation. Tracked gloves, tracked headsets, a footswitch, a harmonic Synergy Knife with a PST Pico (a very small tracker for easy integration in confined spaces) as well as VR-scalpels allow for ‘hands-on’ experience. These additional tools render the moving and modification of stereoscopic image information more intuitive than it would be possible with a standard mouse or keyboard.
Medical decision-making, previously based on a doctor’s experience, is now trained through real-time 3D visualisation, rendering Vesalius 3D a pivotal tool during team meeting or the planning for complex surgeries. Please find a video here.
On 23 September of this year, 3IT’s long-time partner media:net berlin-brandenburg invited for a “Political Morning with Christian Gaebler, head of Senatskanzlei Berlin” at the 3IT. After a welcome by Dr.-Ing. Ralf Schäfer, Head of Video division at the Fraunhofer Heinrich Hertz Institute, and Andrea Peters (media:net berlinbrandenburg e.V.), the two moderators Claudia Lehmann (CEO MAZ & MOVIE) and Christopher Hana (CEO The Esports Observer) took the floor to talk with Chrisitan Gaebler about the future of Berlin’s media industry in the face of the crisis caused by Covid19.
During the in-depth session, Christian Gaebler emphasized Berlin’s quick reaction to the difficult situation by providing the media industry with emergency aids. He argued that in particular 3IT’s long-time partner MedienBoard Berlin Brandenburg had to be mentioned in a highly positive light due to its fast and flexible response to the crisis and purposeful support of the media and film industries.
In the discussion with the moderators, the question arose on how Berlin-Brandenburg as an attractive site for the movie and film industry could be secured on a long-term and sustainable basis.
As a reaction to the current situation, as per Gaebler, the film sector would need to embrace new developments and direct its sponsorships accordingly. An example, he argued, was the “Digital Film Production”, which had been presented shortly with 3IT’s partner Medienboard, in order to consolidate and sustain Berlin’s competitiveness in the domain of filmmaking with digital key technologies, such as CGI (Computer Generated Imagery) and VFX (Visual Effects). Next to these very reasonable funding models, he further resumed, for an enduring film sector, qualified workforce would be mandatory, as well as efficient operations, and good sites, which does not necessarily imply the state’s intervention. Gaebler referred to the game industry as a very positive example, which emerged out of an active Start-Up-Scene and thanks to positive conditions without any government aids, had converted Berlin into one of the hotspots of the Games- and Esports-Scenes. He emphasised the duty of the Senate Chancellery to carry on in providing the space and foundations necessary to guarantee further expansion and growth. Subsequently, he admitted that in terms of digitization, there would be a huge room for improvement, in particular with regards to the development of broad band.
from left: Marc Reznicek (CINIQ/Fraunhofer HHI); Dr. Ralf Schäfer (Fraunhofer HHI); Christian Gaebler (Berlin Senate Chancellery); Maria Ott (3IT/ Fraunhofer HHI)
In his outlook at the end of the talk, Christian Gaebler pointed out further important issues that needed to be addressed by the senate in the future: improving the working conditions in film and television and the technical support of non-commercial local radio stations.
Finally, the moderators opened the general Q&A session and the participants were given the opportunity to address their questions personally to the head of the Senate. This resulted in another interesting exchange.
The 3IT-Team was very happy to host its partner media:net berlinbrandenburg’s event. We are looking forward to more fruitful events like this in the future.
For more information on the event, check out media:net’s homepage.
IBC is the world’s most influential media, entertainment and technology show. Six leading international bodies are the partners behind IBC, representing both exhibitors and visitors.
This year’s IBC took place in a digital format due to restrictions in connection with the ongoing pandemic. Fraunhofer HHI has participated in a presentation of new technologies with major involvement of the 3IT partners Telekom and Volucap.
The presentation of the new technologies was brought to visitors via a 3D rendered digital booth, where everyone was able to get detailed information on the technological development of the Production and Streaming of Interactive Volumetric Video and the cutting-edge implementations in the new video coding standard Versatile Video Coding (VVC).
Daily interactive webinar sessions added to the format. Overall, the fair can be considered a very successful and innovative format challenging current restrictions and showing how technology is able to contribute to society’s needs.
Interested in checking out our digital booth? It is still digitally available here. All further in-depth information on our exhibits can be found here.
