Autonomous Agents and Avatars in REVERIEs Virtual Environment Fons - - PowerPoint PPT Presentation

▶
autonomous agents and avatars in reverie s virtual
SMART_READER_LITE
LIVE PREVIEW

Autonomous Agents and Avatars in REVERIEs Virtual Environment Fons - - PowerPoint PPT Presentation

May 06, 2023 148 likes •367 views

Autonomous Agents and Avatars in REVERIEs Virtual Environment Fons Kuijk (CWI), Konstantinos C. Apostolakis, Petros Daras (CERTH), Brian Ravenet (TPT), Haolin Wei, David S. Monaghan (DCU) REVERIE is a Web3D 2015 Real and Virtual

slide-1
SLIDE 1

Real and Virtual Engagement in Realistic Immersive Environments

Autonomous Agents and Avatars in REVERIE’s Virtual Environment

Fons Kuijk (CWI),
 Konstantinos C. Apostolakis, Petros Daras (CERTH),
 Brian Ravenet (TPT),
 Haolin Wei, David S. Monaghan (DCU)

Web3D 2015

REVERIE is a

slide-2
SLIDE 2

Real and Virtual Engagement in Realistic Immersive Environments

Siege of Haarlem (1572)

Web3D 2015

Real and Virtual Engagement in Realistic Immersive Environments

slide-3
SLIDE 3

Real and Virtual Engagement in Realistic Immersive Environments

Teylers Museum (founded in 1784)

Web3D 2015

Real and Virtual Engagement in Realistic Immersive Environments

slide-4
SLIDE 4

Real and Virtual Engagement in Realistic Immersive Environments

The Hiding Place (1940-1945)

Web3D 2015

Real and Virtual Engagement in Realistic Immersive Environments

slide-5
SLIDE 5

Real and Virtual Engagement in Realistic Immersive Environments

Web3D 2015

Real and Virtual Engagement in Realistic Immersive Environments

slide-6
SLIDE 6

Real and Virtual Engagement in Realistic Immersive Environments

Frans Hals Museum

Web3D 2015

Real and Virtual Engagement in Realistic Immersive Environments

slide-7
SLIDE 7

Real and Virtual Engagement in Realistic Immersive Environments

These are facts and stories told on educational trips done in 1966-1967 Educational trips are not easy to organize

In REVERIE, an ambient, content-centric Internet- based immersive environment, people can work, meet, participate in live events, socialize and share experiences as they do in real life

REVERIE can be used for virtual educational trips

(and for a 3D hangout and for Simon Says, and for …)

Web3D 2015

Real and Virtual Engagement in Realistic Immersive Environments

slide-8
SLIDE 8

Real and Virtual Engagement in Realistic Immersive Environments

An educational trip to the European Parliament

Web3D 2015

Real and Virtual Engagement in Realistic Immersive Environments

slide-9
SLIDE 9

Real and Virtual Engagement in Realistic Immersive Environments

An educational trip to a virtual gallery

Web3D 2015

Real and Virtual Engagement in Realistic Immersive Environments

slide-10
SLIDE 10

Real and Virtual Engagement in Realistic Immersive Environments

Objectives of REVERIE’s Virtual Environment

  • No need for fancy equipment.
  • Users free to move and look around.
  • Spatial audio.
  • Virtual humans have natural behaviour:
  • gaze
  • movements
  • gestures
  • social interaction
  • User driven and autonomous virtual characters

plausibly interact.

Web3D 2015

Real and Virtual Engagement in Realistic Immersive Environments

slide-11
SLIDE 11

Real and Virtual Engagement in Realistic Immersive Environments

REVERIE Architecture

Web3D 2015

Real and Virtual Engagement in Realistic Immersive Environments

slide-12
SLIDE 12

Real and Virtual Engagement in Realistic Immersive Environments

User Analysis

WHAT Facial expressions for

  • Puppeting (face)
  • Arousal & Valence
  • Agree or Disagree
  • Attention

Body gestures for

  • Puppeting (body)
  • Navigation

Speech for

  • Simple dialog

Web3D 2015

Real and Virtual Engagement in Realistic Immersive Environments

HOW User data captured by:

  • webcam
  • (Kinect)
  • microphone
slide-13
SLIDE 13

Real and Virtual Engagement in Realistic Immersive Environments

REVERIE Architecture

Web3D 2015

Real and Virtual Engagement in Realistic Immersive Environments

slide-14
SLIDE 14

Real and Virtual Engagement in Realistic Immersive Environments

Reasoning Framework for Autonomy

Web3D 2015

Real and Virtual Engagement in Realistic Immersive Environments

Gaze & Attention Detection User State Emotion & Affect Identification Facts from the World Keyword Spotting Dialogue State World State Agent & Group State Avatar Behavior Proposer Agent Behavior Proposer Gaze & Attention Behavior Assesment Affect & Social Assessment Obstacle Avoidance & Path Finding Behavior Selection Planner & Realizer Speech Navigation Animation Rendering Capturing Script Proposer Analyzers Interpreters Proposers Realizers States

Legend

Rules & Attitude Scripts

slide-15
SLIDE 15

Real and Virtual Engagement in Realistic Immersive Environments

Configurable Clients

Web3D 2015

Real and Virtual Engagement in Realistic Immersive Environments User Client Animation Engine Render module User Client Reasoning & Navigation Server Navigation Server Reasoning Framework Animation Engine Render module Global Parameters Pub/Sub User Client Reasoning & Navigation Server Navigation Server Reasoning Framework Animation Engine Render module Global Parameters Pub/Sub Gaze Analysis User Client Reasoning & Navigation Server Navigation Server Reasoning Framework Animation Engine Render module Global Parameters Pub/Sub Gaze Analysis Face Analysis User Client Reasoning & Navigation Server Navigation Server Reasoning Framework Animation Engine Render module Global Parameters Pub/Sub Gaze Analysis Face Analysis Body Analysis User Client Reasoning & Navigation Server Navigation Server Reasoning Framework Animation Engine Render module Global Parameters Pub/Sub Gaze Analysis Face Analysis Speech Analysis Body Analysis User Client User Client User Client User Client User Client User Client Navigation Server Reasoning Framework Animation Engine Render module Global Parameters Pub/Sub Gaze Analysis Face Analysis Speech Analysis Body Analysis Animation Engine Render module Gaze Analysis Face Analysis Speech Analysis Body Analysis Gaze Analysis Face Analysis Speech Analysis Body Analysis

WebSockets

slide-16
SLIDE 16

Real and Virtual Engagement in Realistic Immersive Environments

Reasoning in REVERIE

Comes in levels: Agents: fully autonomous

Web3D 2015

Real and Virtual Engagement in Realistic Immersive Environments

Behaviour Planner & Realizer Socio-emotional attitude Intention Nonverbal signals! (e.g. head upward, smile)

Avatars: system adds autonomy where needed

  • gaze
  • gestures
  • pose based on user interaction.
slide-17
SLIDE 17

Real and Virtual Engagement in Realistic Immersive Environments

Autonomous Gaze

Web3D 2015

Real and Virtual Engagement in Realistic Immersive Environments

Attentive: Following user camera: Not attentive: (may lead to agent reaction)

slide-18
SLIDE 18

Real and Virtual Engagement in Realistic Immersive Environments

Autonomous Gesture

Web3D 2015

Real and Virtual Engagement in Realistic Immersive Environments

Request to speak

slide-19
SLIDE 19

Real and Virtual Engagement in Realistic Immersive Environments

Options for Navigation

User controlled
 (small incremental steps):

  • Keyboard
  • Mouse
  • Kinect

Autonomous (path planning):

  • Map
  • Follow Me

Web3D 2015

Real and Virtual Engagement in Realistic Immersive Environments

slide-20
SLIDE 20

Real and Virtual Engagement in Realistic Immersive Environments

Conclusion

  • Agents react on users’ activity.
  • User control over avatar is configurable (the

reasoning adds autonomy where needed).

  • Agents and avatars have human-like behavior.
  • Users feel being immersed.
  • REVERIE components
  • can be deployed on regular computer system;
  • are well suited for web-based communication.

Web3D 2015

Real and Virtual Engagement in Realistic Immersive Environments

slide-21
SLIDE 21

Real and Virtual Engagement in Realistic Immersive Environments

More detailed information: http://www.reveriefp7.eu/

Web3D 2015

Real and Virtual Engagement in Realistic Immersive Environments