Enhancing workspace and referential awareness in XR collaboration

Context and Objectives

Augmented Reality (AR) has been largely explored for remote assistance under a wide range of applications domains, such as industrial maintenance or home assistance. In such a context, one user, in an AR capable system, can asks guidance from remote users having visual access to the physical environment. To ensure optimal collaboration, two main requirements must be fulfilled, first, remote users must be aware of the physical workspace (workspace awareness) and second, users must be aware of the other users (user awareness).

The awareness of the workspace enables the remote collaborators to understand and assess the environment in which the AR user is located. A wide range of works have explored how to present AR workspaces to remote users [Assaf et al.]. Most of the works focuses on static 3D reconstructions [Kumaravel et al.], virtual proxies [Oda et al.], light fields [Mohr et al.] or real-time video feeds [Fages et al.]. All methods present several trade-offs, based on the size of the reconstructed workspace, the freedom that remote users have to navigate through the reconstructed workspace, the fidelity of the reconstructed workspace or the required preparation and instrumentation. While video provides high fidelity with minimal instrumentation, the navigation capability for remote users is limited and only provide a partial view of the workspace. In contrast, static reconstructions have high fidelity, high navigation freedom and a large reconstructed workspace, at the expense of complex instrumentation. However, due to the difficulty to ensure free exploration for remote collaborators, workspace awareness still remains an open problem, and it is typically supported either by virtual replicas/reconstructions or video feeds [4]. However, with the appearance of 3D Gaussian Splatting (3DGS) methods [Kerbl et al.] [Meuleman et al.], real-time and high-fidelity reconstruction of physical workspaces is nowadays possible. However, there are still little research on how the online reconstruction process can be executed in real time without any instrumentation and still ensure a smooth collaboration among different actors. In this context, the first objective of the PhD is to evaluate incremental reconstruction methods in order to efficiently support workspace awareness even in the context of partial or uncertain reconstructions, and propose interaction methods to incrementally refine the dynamic reconstruction. Furthermore, a secondary goal in this context is the perceptual assessment of the reconstructed models to ensure that 3DGS methods do not introduce perceptual biases.

Second, users should be able to be aware of other users and should be able to communicate with traditional interaction modalities, such as voice, gaze and gestures [Johnson et al.]. However, the interaction capabilities of VR/AR users will differ, while the AR user will be constrained by the physical workspace, remote collaborators have “super powers” (e.g., can teleport or go through virtual objects). Precisely, we will focus on referential awareness, in which remote collaborators use various means to refer to the objects required to accomplish a physical task [Argelaguet et al., Dix]. Users rely on methods that requires the least effort to enable others to understand their references [Clark and Wilkes-Gibbs]. Typically, collaborators prefer to use short phrases, alternate descriptions, deictic expressions, and simple gestures (like pointing) to efficiently refer to objects without lengthy verbal descriptions [Kraut et al.]. Deictic expressions are used to point to things, people, or locations and contain words like “this”, “here”, or “that”. These expressions are also often combined with gestures such as pointing. For example, someone could point at an object and say “bring that one over here” instead of verbally describing what and where the object is. However, in the context of dynamic and potentially uncertain reconstructions, the object referred to might not be accessible (e.g., hidden, unreconstructed) to all collaborators. The final goal of the PhD will be to propose interaction techniques in order to ease referential awareness among collaborators in the context of dynamic environment reconstructions. The user representation will be key element to ensure referential awareness as it provides visual feedback to the collaborators (e.g., posture, gaze, gestures), thus providing an efficient information exchange [Bai et al.]. In addition to referential awareness, the user representation can have a large impact on other user experience dimensions such as social presence [Oh et ak.]. Yet, in the context of the remote AR/VR collaboration, simple user representations are typically considered [Piumsomboon et al.] [Bai et al.]. A transversal objective will be to assess the role of the user representation and interaction capabilities for referential awareness in an asymmetric VR/AR context.

Organization

The first three months of the PhD candidate will be do a thorough analysis of the state of the art and appropriate a VR 3DGS-based rendering prototype already developed in the team. The next twelve months will be focused on reconstruction fidelity assessment and explore perceptual issues of the 3DGS reconstruction. The next twelve months will be focused on the study of referential awareness in the 3DGS reconstruction in a context of asymmetric VR/AR interaction. The final six months will be focused on the writing of the manuscript and to the preparation of the defense, while finishing the initiated contributions.

Ecosystem

This PhD is anchored in the main research objectives of the Seamless team (created end 2024), focusing on the improvement of cross-reality interactions. This PhD will benefit from the EquipEx+ Continuum project in which the Seamless team is involved in. Precisely from VR and AR platforms at Inria Center at Rennes University/IRISA.

Profile of the applicant

Applicant must be holder of a Masters in Computer Science or equivalent, with in-depth knowledge in Human-Computer Interaction and/or in Virtual Reality. Knowledge about machine learning algorithms and signal processing would be positively appreciated, while not mandatory.