In the “Virtual LiVe” project, the three Fraunhofer Institutes FOKUS, HHI and IIS are working with the creative industry and companies from the event sector. Together, they realized their first experimental laboratory concert within the framework of the project. On December 11 2021, the highly anticipated event took place at different locations simultaneously, showcasing the benefits of the implementation of immersive media technology into classic event formats.
The past couple of months have been tremendously exciting within the project Virtual LiVe. All sections were operating at full speed in order to ensure a smooth and technically accurate workflow. How did it all start? In the first quarter of this year, the project partners Fraunhofer FOKUS, Fraunhofer HHI and Fraunhofer IIS set out to extend and enhance hybrid live events with cutting-edge immersive technologies. The ambitious goal promised the exploration of new shores: In order to blur the boundaries between physical and digital perception, high-end Fraunhofer technologies were to be implemented and intertwined, thus improving the digitization of classic event formats.
Together, the project partners invited all the stakeholders at the beginning of the project, analysed dozens of questionnaires and conducted workshops and colloquia to determine the exact needs of the industry. The findings led to the realization of the project’s first hybrid live event: In collaboration with the artist The Dark Tenor the event finally took place in the Kesselhaus at Kulturbrauerei Berlin, in the Planetarium Bochum, in the digital cinema at Fraunhofer IIS and online.
The whole setup of the concert was a one-of-a-kind event and received an overwhelmingly warm response from the various audiences at different locations.
Kesselhaus at Kulturbrauerei – on Site Location with a Digital Twist
The artist Billy Andrews – The Dark Tenor – gave the premiere concert of his Classic Roxx Tour at the Kesselhaus in Berlin. About 150 guests enjoyed the show on site with The Dark Tenor’s full band line-up – a first since the start of the pandemic. The outstanding novelty: He was accompanied by the artists Queenz of Piano, who played from StudioMix; a different location in Berlin. They were incorporated into the concert via an ultra-low latency audio connection utilizing the Digital Stage platform. This enabled the artists to play music together live “on stage”, even though they were not located in the same place. The Queenz of Piano were additionally projected from their studio onto a screen in the Kesselhaus. The musical and visual interplay provided a truly immersive, hybrid experience as the ensemble played together from different locations.
“The Classic RoXX Tour premiere was a resounding success and, in collaboration with the Fraunhofer Institutes, has created new possibilities for how music and art can co-exist over the Internet despite being at a distance from each other. I’m excited to see that technology is helping artists interact. It will definitely become an integral part of what we do at some point!” (Billy Andrews, The Dark Tenor)
„This was certainly something very special, even for the Kesselhaus. For the first time, musicians from different places made music together here at the same time, as if they were standing on the same stage. In addition to the spectators present here, hundreds more watched the show online and participated vigorously. A great experience.“ (Sören Birke, Kesselhaus Berlin)
“With the use of Fraunhofer technologies and those of the partners, we have managed to create a very special concert experience – unique and first of its kind. We are proud to have realized this project with these partners and very happy about the great feedback from experts, artists, but also from the actual fans.” (Stephan Steglich, Fraunhofer FOKUS)
© Paul Hahn/ Fraunhofer FOKUS
© Paul Hahn/ Fraunhofer FOKUS
Planetarium Bochum – Live Stream of the Concert in 360 Degrees on a Planetarium Dome
Hybrid live events offer participants the opportunity to attend performances by artists even if they are unable to travel to the venue in person – for many diverse reasons. In the context of this validation project, the project partners have gone one step further and opened up new venues for cross connected artistic performances, by means of their technological expertise.
The event in Berlin’s Kesselhaus was recorded using the panorama camera OmniCam, a scalable, mirror-based multi-camera system that, together with its Real Time Stitching Engine (RTSE), is capable to capture a high-resolution 360° video stream and transmit it in real time.
At the Planetarium Bochum, video images from 4K closeup cameras were combined with those from the 360° camera using a special DomePlayer and geometrically adjusted in real time for the dome shape, to then be displayed in the planetarium’s 6Kx6K dome projection using the 11 present high-resolution projectors. Depending on the song, this projection was further enriched with planetarium-specific content such as dome illumination effects or the razor-sharp starry sky of the Zeiss star projector. All video media displayed on the projection were interactively animated in their properties in real time, such as position, rotation, size, zoom or transparency. Depending on the situation and song mood, this approach provided a suitable and varied image that could visually even exceed the on-site experience.
To complete the immersive experience, the concert sound was produced with MPEG-H Audio. A 3D-Audio mix was created in real time. With the help of a Spatial Audio Designer an MPEG-H scene was authored, then encoded and pushed to the cloud. The audio and video stream were decoded in the planetarium and then distributed to more than 60 loudspeakers in the entire planetarium dome. The different video and audio streams were synchronized frame-accurately to ensure a perfect audio-visual experience.
„The 360° live concert was impressive – as an audience member in the Planetarium Bochum, you got a very direct impression of the atmosphere in the “Kesselhaus”! The interaction of 360° video and 3D audio created a highly immersive feeling in the dome. This was confirmed by our audience, which also asked for more events like this. The collaboration with the Fraunhofer Institutes in this experimental concert setting was consistently congenial, creative, and technically impressive.“ (Susanne Hüttemeister, Planetarium Bochum)
„Immersive rooms like planetariums are excellent platforms to create the feeling of finding oneself at the live location since we have the possibility to present immersive video and 3D-Audio in the highest quality to the audience. Such a hybrid social event, where the line between the real event and the remote audience is blurred, could be a solution for the “new normal” and contribute to overcome constraints of pandemic conditions or simply when the real event is fully booked. Even more, it transforms our perception of live events.“ (Christian Weißig, Fraunhofer HHI)
“It was a big challenge for all of us to make this high-profile experience possible, as it was the first time – world wide – that a concert was streamed live and displayed in 360° on a dome with this variety of functions and top notch audio-video quality! We are all the happier that everything worked out as planned and that the audience feedback was so overwhelmingly positive!“ (Manuel Schiewe, Fraunhofer FOKUS)
“A live concert is a perfect match for a 3D audio production. Listeners that can reproduce 3D-Audio – which can be as easy as using headphones – can enjoy the ambience of the concert hall and the noise of the crowd. The Dark Tenor fired up his audience and motivated them to sing along. This auditory experience has been successfully transported to audiences at the various other locations.” (Ulli Scuda, Fraunhofer IIS)
©Philipp Plum/ Fraunhofer FOKUS 
©Philipp Plum/ Fraunhofer FOKUS
Interactive Concert Streaming to Remote Locations
More than 600 participants from 16 countries followed the concert virtually via the Internet. For a truly hybrid experience it was important to give viewers at home the opportunity to enjoy the concert in a way that comes as close as possible to a live experience. The FAMIUM-DASH solution that was used here allows video streams to be transmitted with low-latency, adaptive quality, including auditory or visual feedback. In this way, the remote audience was not only able to view the concert online with a small delay of a few seconds, but also to send their applause in real time to the Kesselhaus via digital emojis and video chats. A Public Fan Channel, which was displayed on a big screen in the Kesselhaus during the concert, emphasized the interconnectedness of the different audiences. Additionally, viewers were able to setup their own private video chat rooms and invite their friends to watch the concert together. The FAMIUM SAND solution provides advanced streaming analytics and video player coordination and ensured a smooth playback of the low-latency video stream in the context of this hybrid event.
Additionally, the 360-degree panorama live stream of the OmniCam that was transmitted to the planetarium was also provided online for audiences at home in a resolution of up to 4K. With this additional stream, the viewers at home were able to navigate in the panorama of the live concert at Kesselhaus and immerse themselves into the show.
MPEG-H audio production can be rendered by a wide range of playback devices, once an audio mix is produced. For on-site monitoring of the encoded signal, a tablet was used to receive the stream from the internet and decode it for binaural headphone reproduction. At the same time, the very same bit stream was used to send to a digital cinema in Erlangen, where project partners watched the show with 3D sound. The laboratory concert demonstrated to what extent a production workflow that applies top edge technologies is able to provide a truly immersive real time content for audiences on site, on remote event locations such as planetariums or cinemas and at home.
“Soon after we started the live transmission of the concert stream, viewers immediately started providing feedback by clicking on the several feedback buttons and asked to enter our moderated public Fan Channel to appear on stage in the Kesselhaus. They were able to experience a smooth low-latent video playback of the live concert on their player platform of choice, including Web-Browsers on mobile and desktop devices, setup-boxes, TVs via HbbTV, and even the dome in the Planetarium in Bochum.“ (Louay Bassbouss, Fraunhofer FOKUS)
© Fraunhofer FOKUS 
© Fraunhofer FOKUS
What’s next?
The laboratory concert was a great success. The different technologies were put to the test in this experimental interaction and showed how to push the boundaries of hybrid live events. Currently, the second validation project is already in planning, so look forward to more exciting news from the Virtual LiVE project.
“With our collaborative lab concert, we have proven that hybrid live events are technically feasible. Now, together with our partners, we need to transform the technical complexity of such events into easily manageable and scalable solutions in order to provide artists and event organizers a tool for future and commercially successful hybrid productions.” (Robert Seeliger, Fraunhofer FOKUS)
„We are already planning further to investigate technological vanguard set ups for fancy ideas like volumetric recording at concerts and trade shows. This will allow the audience equipped with head mounted displays to virtually move around. They will decide individually from which direction they experience the concert or show floor. We are looking forward to sharing our findings with you!” (Siegfried Fößel, Fraunhofer IIS)
Did you miss the concert or the livestream? Watch the announcement video of the event and stay tuned for the video of the live event!
_____________________________________________________________________________
About Virtual LiVe
The Fraunhofer Institute for Open Communication Systems FOKUS, the Fraunhofer Institute for Telecommunications – Heinrich Hertz Institute HHI and the Fraunhofer Institute for Integrated Circuits IIS are collaborating on the project Virtual LiVe – Virtualization of Live Events through Audiovisual Immersion. The project is hosted by the 3IT – Innovation Center for Immersive Imaging Technologies – and supported by the ‘KMU-akut’ program “Research for SMEs” of the Fraunhofer-Gesellschaft. The aim of the project is to digitally complement classic event formats by using new immersive media technologies (e.g. 3D-Audio, 360°-Video streaming, interactive low-latency streaming and advanced stream analytics) and to add value compared to typical video streams via the Internet. Find more information on the project here.