Creating socially reactive virtual characters for enhanced social interactions in Virtual Reality

Context

Creating compelling immersive situations requires to deal with several important elements. Amongst these, simulating and displaying convincing virtual humans naturally interacting with users is crucial to replicate real-life situations and foster collaboration, due to the importance of interpersonal interactions in our everyday life. In the last decade, great advances have been seen in the quality of such virtual characters, such as in terms of appearance (e.g. Metahuman characters) or the ability to control characters in game-like situations (e.g., Motion Matching [But15]). However, when it comes to interactions with users, such virtual characters typically only display limited, and often scripted, behaviours. For instance, virtual pedestrians are typically steered to avoid other characters and users using crowd simulation methods (e.g RVO [VLM08]), but will not react appropriately if users suddenly startle them, intrude their personal space, or act inappropriately. 
The goal of this PhD is therefore to create virtual characters that display more appropriate reactions to a number of social interactions in immersive situations. The general idea is that virtual characters are often unresponsive to interactions, that would be considered unacceptable in real situations, which is detrimental for the believability of the immersive situations. A typical demonstrator of this work would be a virtual character demonstrating its annoyance at someone intruding their personal space, e.g. if coming too close, both through their body gesture and facial expressions. Such reactive characters are crucial for creating credible immersive situations, in line with the concept of Plausibilty Illusion presented by Slater et al. [Sla09,SSC10] (i.e., “the overall credibility of the scenario (in particular in relation to how a similar situation might be in physical reality”). 

Objectives

To reach this goal, this PhD will focus on two main research axes: 1) identifying the minimal set of reactive capabilities that a virtual character should possess for displaying believable social interactions, and 2) proposing novel methods to endow virtual characters with such natural reactions. 
In the first axis of the PhD, the objective will be to identify the minimum set of reactive capabilities that a virtual character should display to naturally react to users. This will require to first establish a taxonomy of such reactive capabilities, including body, facial, emotional, vocal or even physical reactions to users. For instance, previous work investigated the possibility of enabling virtual characters to touch users depending on the context of the interaction [CPT21], which is seldom accounted for while having the potential to provide novel interpersonal interactions towards users. Similarly, virtual characters should be able to positively or negatively emotionally react to user interactions, both through their posture and facial expressions. In addition, identifying the reactive capabilities that virtual characters should display also raises the question of predicting the users’ intention through their motion, and therefore identifying key characteristic elements of the interaction, such as differentiating between the action of giving an object to a virtual character (for collaboration) vs. poking the virtual character (potentially inappropriate). 
In the second axis of the PhD, the objective will be to propose novel methods to endow virtual characters with novel reactive abilities. The difficulty in proposing such methods lies in their ability to generalize to a large range of reactions, to avoid simply scripting a number of predefined interactions. To investigate such questions, we propose to explore novels paradigms where users themselves are able to demonstrate to the virtual character how to react to specific situations, therefore enabling virtual characters to continuously learn from such experiences. We are particularly interested in exploring the possibility to take inspiration from virtual body ownership switching experiences [OPS15], in order to let users experience and demonstrate their own reactions to previous interactions they performed. Such an approach also raises challenges related to learning reactions from a limited number of examples, to ensure that we introduce relevant variations in the interaction abilities learned by virtual characters so that they do not repeat exactly the same motion every time they encounter the same situation. Such approaches also need to be evaluated to demonstrate their benefits in terms of user interactions with such virtual characters, as well as in terms of benefits on the overall user immersion in such virtual environments. 

Overall, the results of this PhD will contribute to creating more compelling and reactive virtual characters, as well as to better understand the limits of appropriate and inappropriate interactions between real and virtual persons in immersive situations. The results will also be shared through demonstrators displaying virtual characters reacting more naturally at user social interactions, as well as learning novel reactions from user demonstrations. 

Project Organisation

This PhD will be supervised by members of the Virtus team, who have complimentary expertise in studying, modeling and evaluating virtual humans, especially by leveraging Virtual Reality, and will take place in the joined Inria / IRISA research centre located in Rennes. 

The project will last for the 3 years of the PhD and should involve:

• Year 1: literature review including a taxonomy of reactive actions; proposition of a first set of minimal reactive capabilities and development of an initial reactive model (demonstrator D0)
• Year 2: evaluation of the initial reactive model; generalization of the reactive model, including exploring a virtual body ownership switching paradigm to enable users to demonstrate expected reactions (demonstrator D1)
• Year 3: evaluation of the generalized reactive model; finalization of the demonstrators (Dfinal)