The goal of the project VoluProf, Volumetric professor for omnipresent and user-optimized teaching in mixed reality, is to develop a Mixed Reality application that is suitable for everyday use within the context of knowledge transfer. The idea is to create a novel and more efficient way of direct, interactive and user-oriented teaching via photo- and audiorealistic avatars of real lecturers for remote learning.
The project is funded by the Federal Ministry of Education and Research (BMBF).
Traditional online learning, which provides content via static video and audio scripts, is limited in its effects since learners can hardly interact with teachers and give any feedback. Mixed Reality (MR) combined with animatable volumetric video has the potential to provide an immersive and interactive learning medium that enables a completely new level of learning experience.
VoluProf aims to develop a user-oriented, interactive MR application that enables discourses between teachers, represented as avatars, and users by means of multimodal chatbots. To achieve this, high-quality volumetric videos of the teachers are generated and animated as a basis for interaction. Voice of the avatar is generated realistically from a text provided by the teacher. The avatar is then streamed to the MR headsets of the learners over next-generation mobile networks using low-latency streaming techniques. The development is continuously accompanied by user studies that investigate the requirements and effects of the new teaching medium. Ethical, legal and social implications are monitored and continously integrated into the technical development.
The duration of the project is set for September 2021 – August 2024
The project partners want to create a space of future learning where both traditional lecturing and online learning environments are combined in a way, where lecturers can be experienced not only on video but also virtually and physically. In other words, the user participates in a lesson with a virtual professor by means of Augmented Reality glasses for example in their living room, while also being able to interact with them via eye contact and verbal communication.
The aim is to develop the following key features:
- Generating a photorealistic and animatable volumetric representation of the lecturer.
- Location-independent applicability through transmission via 5G networks and direct integration into mixed reality devices on the market, as well as subjective optimization of image quality at minimal bit rate.
- Natural and fast synthesis of the lecturer’s original voice.
- Answering ethical, legal and social questions about the implications of the new way of knowledge transfer.
- Media reception and effects research for the analysis and optimization of user acceptance.
- Educational research on the optimal and user-adaptive use of the new way of knowledge transfer
Is responsible for the development of the location-independent applicability through transmission via 5G networks and direct integration into Mixed Reality devices on the market, as well as subjective optimization of image quality at a minimal bit rate. Moreover, Fraunhofer HHI is concerned with the creation of photorealistic virtual humans and their animation based on volumetric video data. In particular, the scientists will work on methods for context-based synthesis of natural motion, gestures, and facial animation. Another focus in the project will be the perceived quality of image immersion in VoluProf. This includes the evaluation of the system in user studies, the modeling of the relevant perceptual aspects, and the integration of these models into the system.
Deutsche Telekom is Europe’s largest telecommunications company and will contribute its expertise in building and operating IP networks, especially 5G networks and the cloudification of network components. As an innovation driver, Deutsche Telekom has been active for years in the area of Mixed Reality with various partners and has developed a 5G Low Latency Edge Cloud Testbed and, together with the project partner FhG HHI and other partners, especially startups, has gained valuable experience in volumetric streaming and other media-relevant scenarios over 5G networks and will thus contribute its testbed as a basis for the planned developments in the project as well as the expertise acquired in the process.
Volucap GmbH specializes in the production and post-production of immersive film projects as well as research, development and distribution of specialized video technologies. Embedded in the infrastructure of the largest film studio in Europe, Studio Babelsberg, Volucap GmbH offers companies, SMEs and research institutions in particular access to the technology of volumetric captures of 3D video models: Volumetric captures (Volucaps). In this project Volucap will contribute high quality volumetric video material of the lecturer.
The product portfolio of Aristech GmbH is based on proprietary developments in the field of speech technology software. This includes applications in the area of speech synthesis (TTS), speech recognition (ASR) as well as semantic analysis of spoken and written language. Aristech’s TTS technology is characterized in particular by the high naturalness of the speech output. This is achieved by a combination of domain-specific recording scripts of the speech talent, high accuracy of linguistic post-processing and the use of machine learning algorithms. As a final product, the voices are indistinguishable in quality from classical speech sampling while guaranteeing the full flexibility of speech synthesis.
The University of Rostock is responsible for Media Research and Ethics Research within the VoluProf project. At the Institute for Media Research and the Faculty of Theology, a great deal of preliminary work has been done on this project. From the perspective of communication and media studies, work, publications, and research focus on the interplay of media, media appropriation, and society with a focus on reception and impact research of specific user groups, mobile media use, and the use of immersive media. Furthermore, interdisciplinary research on mediatized science communication is conducted at the Institute with its Chair of Communication and Media Studies. One focus of the work is the methodological triangulation of qualitative methods of empirical user research, combined with new methods of digital humanities. Current research projects deal with the representation and impact of diversity and gender in digital and audiovisual media as well as with new methods of empirical media impact research involving innovative, digital and interactive technologies. The Faculty of Theology is responsible for analyzing the ethical, legal, and social implications (ELSI) of the VoluProf application from the user perspective. The faculty based Junior Professorship of Ethics has worked in the fields of Ethics of Technology and Ethics of Medicine for several years. Current research is focused on the Ethics of Artificial Intelligence and the Ethics of Information and Communication Technologies. All aspects of ELSI will be identified with the help of a literature review, workshops, qualitative interviews, participant observation studies, and group discussions. This expertise will be theoretically processed, discussed and evaluated throughout the project, and integrated into the specific research design and technical development process.
The Educational Technology Lab at DFKI works on a broad range of research topics related to digitization (especially AI) and education. This includes work on chatbot-based interaction in learning contexts as well as projects that investigate methods of personalization and analytics in learning contexts. A central field of application is higher education – here, for example, methods for automated recognition of learner creativity in online learning environments are currently being developed with DFG funding; with BMBF funding, projects are currently being carried out on personalized competence development through scalable mentoring processes and on digital support for learning groups in online education. The preliminary work achieved in the previous and current projects and the expertise built up can be contributed to the project, e.g. for the didactic conception of the example scenario and for the technical-didactic design of the interaction concept with the avatar.
Conferences, Journals, Book chapters
Serhan Gül, Cornelius Hellge, Peter Eisert: Latency Compensation Through Image Warping for Remote Rendering-based Volumetric Video Streaming. Proceedings of the IEEE International Conference on Image Processing (ICIP), October 2022.
Dr. H.J. van Gils-Schmidt, JProf. J.-C. Põder, Dr. A. Räder, J. Wegner, MA: A Mixed-Method Approach to Investigate Value Change by Technological Innovations (in context of an AI-based Mixed Reality Education Application). Workshop Machines of Change: Robots, AI and Value Change, TU Delft, the Netherlands (Online), 1–3 February, 2022.
Jangwoo Son, Yago Sanchez, Christian Hampe, Dominik Schnieders, Thomas Schierl, Cornelius Hellge: L4S congestion Control algorithm for interactive low latency applications over 5G. Submitted to ICME 2022.
Esther Greussing, Franziska Gaiser, Stefanie Helene Klein, Carolin Straßmann, Carolin Ischen, Sabrina Eimler, Katharina Frehmann, Miriam Gieselmann, Charlotte Knorr, Angelica Lermann Henestrosa, Andy Räder & Sonja Utz (2022): Researching interactions between humans and machines: methodological challenges. In: Publizistik. Vierteljahreshefte für Kommunikationsforschung. DOI: 10.1007/s11616-022-00759-3.
Peter Eisert, Oliver Schreer, Ingo Feldmann, Cornelius Hellge, Anna Hilsmann: Volumetric Video – Acquisition, Interaction, Streaming and Rendering, Immersive Video Technologies, pp. 289-326, Academic Press, UK, ISBN: 978-0-323-91755-1, September 2022.
MWC 2023, Mixed Reality Video Streaming with L4S (planned)
MWC 2022, Mixed Reality Streaming over 5G
IBC 2022, Interactive Volumetric Video Streaming over 5G
Cornelius Hellge: Interactive Volumetric Video Streaming over 5G using L4S, IEEE BTS Pulse, July 14 2022. (https://www.5g-mag.com/post/12-14-07-22-ieee-bts-pulse).
Cornelius Hellge: Holoportation & Avatars, FutureHotels Innovation Breakfast, June 2022. (https://www.linkedin.com/company/futurehotel/).
Watch this summary about the project’s aims by Cornelius Hellge, project leader of VoluProf (German language).