Impact of 3D bookmarks on Navigation and Streaming in a Networked - - PowerPoint PPT Presentation

Impact of 3D bookmarks on Navigation and Streaming in a Networked Virtual Environment

Impact of 3D bookmarks on Navigation and Streaming in a Networked Virtual Environment

Thomas Forgione1 thomas.forgione@irit.fr Axel Carlier1 axel.carlier@enseeiht.fr Géraldine Morin1 morin@enseeiht.fr Wei Tsang Ooi2 weitsang@nus.edu.sg Vincent Charvillat1 charvi@enseeiht.fr

1Université de Toulouse – IRIT 2National University of Singapore

May 11, 2016

• T. Forgione, A. Carlier, G. Morin, W.T. Ooi, V. Charvillat

MMSys 2016 1 / 36

Impact of 3D bookmarks on Navigation and Streaming in a Networked Virtual Environment

NVE

Networked Virtual Environment 3D environment Streaming Free flying camera 5 degrees of freedom (x, y, z, θ and φ)

Figure: An NVE from Sketchfab

• T. Forgione, A. Carlier, G. Morin, W.T. Ooi, V. Charvillat

MMSys 2016 2 / 36

Impact of 3D bookmarks on Navigation and Streaming in a Networked Virtual Environment

NVE

Networked Virtual Environment 3D environment Streaming Free flying camera 5 degrees of freedom (x, y, z, θ and φ) P x y z T

φ θ Figure: Spheric coordinate system

• T. Forgione, A. Carlier, G. Morin, W.T. Ooi, V. Charvillat

MMSys 2016 2 / 36

Impact of 3D bookmarks on Navigation and Streaming in a Networked Virtual Environment

Outline

1

Navigation in NVE Controls 3D Bookmarks

2

Quality of Experience Experimental setup Results

3

Impact on Quality of Service Streaming policies Comparison

• T. Forgione, A. Carlier, G. Morin, W.T. Ooi, V. Charvillat

MMSys 2016 3 / 36

Impact of 3D bookmarks on Navigation and Streaming in a Networked Virtual Environment Navigation in NVE

Outline

1

Navigation in NVE Controls 3D Bookmarks

2

Quality of Experience Experimental setup Results

3

Impact on Quality of Service Streaming policies Comparison

• T. Forgione, A. Carlier, G. Morin, W.T. Ooi, V. Charvillat

MMSys 2016 4 / 36

Impact of 3D bookmarks on Navigation and Streaming in a Networked Virtual Environment Navigation in NVE Controls

State of the art : difficulty of navigation

Ruddle et al. [2000] Some interactions can get the user lost in the 3D environment Hyperlinks as such are unusable Jankowski and Hachet [2015] Difficult with a 2D screen Strongly task-dependent

• T. Forgione, A. Carlier, G. Morin, W.T. Ooi, V. Charvillat

MMSys 2016 5 / 36

Impact of 3D bookmarks on Navigation and Streaming in a Networked Virtual Environment Navigation in NVE Controls

Keyboard controls : difficulty of navigation

Esc F1 F2 F3 F4 F5 F6 F7 F8 F9 F10 F11 F12 ‘ 1 2 3 4 5 6 7 8 9

• =

\ Del

Lock

/ *

• Tab

Q W E R T Y U I O P [ ] 7 8 9 Caps A S D F G H J K L ; ’ 4 5 6 ⇑ Shift Z X C V B N M , . / ⇑ Shift ↑ 1 2 3 Ctrl  Alt Alt  Ctrl ← ↓ → . Enter

+

Enter

Figure: Keyboard controls

P x y z T

φ θ Figure: Spheric coordinate system

• T. Forgione, A. Carlier, G. Morin, W.T. Ooi, V. Charvillat

MMSys 2016 6 / 36

Impact of 3D bookmarks on Navigation and Streaming in a Networked Virtual Environment Navigation in NVE Controls

Difficulty of navigation

• T. Forgione, A. Carlier, G. Morin, W.T. Ooi, V. Charvillat

MMSys 2016 7 / 36

Impact of 3D bookmarks on Navigation and Streaming in a Networked Virtual Environment Navigation in NVE 3D Bookmarks

Arrow bookmarks

Figure: Bookmarks as arrows

Arrow curved according to the current position

• f the camera

Shows path to targeted bookmark 3D data from McGuire [2011]

• T. Forgione, A. Carlier, G. Morin, W.T. Ooi, V. Charvillat

MMSys 2016 8 / 36

Impact of 3D bookmarks on Navigation and Streaming in a Networked Virtual Environment Navigation in NVE 3D Bookmarks

Fly-to transition

Figure: Smooth move from the current position to the bookmark

• T. Forgione, A. Carlier, G. Morin, W.T. Ooi, V. Charvillat

MMSys 2016 9 / 36

Impact of 3D bookmarks on Navigation and Streaming in a Networked Virtual Environment Navigation in NVE 3D Bookmarks

Viewport bookmarks

Figure: Bookmarks as viewports

Computer vision style bookmark Optical center Image plane

• T. Forgione, A. Carlier, G. Morin, W.T. Ooi, V. Charvillat

MMSys 2016 10 / 36

Impact of 3D bookmarks on Navigation and Streaming in a Networked Virtual Environment Quality of Experience

Outline

1

Navigation in NVE Controls 3D Bookmarks

2

Quality of Experience Experimental setup Results

3

Impact on Quality of Service Streaming policies Comparison

• T. Forgione, A. Carlier, G. Morin, W.T. Ooi, V. Charvillat

MMSys 2016 11 / 36

Impact of 3D bookmarks on Navigation and Streaming in a Networked Virtual Environment Quality of Experience Experimental setup

Evaluating the impact of bookmarks with MicroWorkers

Crowdsourcing People were recruited on microworkers.com More than 100 started the experiment 51 of them completed it, and were paid Traces of all experiments were recorded in order to be replayed to simulate streaming

• T. Forgione, A. Carlier, G. Morin, W.T. Ooi, V. Charvillat

MMSys 2016 12 / 36

Impact of 3D bookmarks on Navigation and Streaming in a Networked Virtual Environment Quality of Experience Experimental setup

Problem : task design

Without task, people won’t browse our 3D scenes They have nothing to do They are probably not interested in our data 3D browsing is difficult

= ⇒ They will browse for a few seconds and leave = ⇒ We will not see the impact of bookmarks

Our solution Ask people to perform a task Find hidden objects in the scene

• T. Forgione, A. Carlier, G. Morin, W.T. Ooi, V. Charvillat

MMSys 2016 13 / 36

Impact of 3D bookmarks on Navigation and Streaming in a Networked Virtual Environment Quality of Experience Experimental setup

Figure: A coin hidden behind a curtain

• T. Forgione, A. Carlier, G. Morin, W.T. Ooi, V. Charvillat

MMSys 2016 14 / 36

Impact of 3D bookmarks on Navigation and Streaming in a Networked Virtual Environment Quality of Experience Experimental setup

Task design

One micro-job Initial questionnaire (age, gender, etc...), tutorial 3 random tasks Final questionnaire to evaluate QoE One random task Select 1 3D scene out of 3 (loaded beforehand) Select 1 UI out of 3 (no bookmarks, 10 viewports, 10 arrows) Select 8 coins randomly out of 50 (positioned beforehand) No correlation between bookmarks and coin positions

• Success

At least 6 coins found among the 8 hidden coins in the scene

• T. Forgione, A. Carlier, G. Morin, W.T. Ooi, V. Charvillat

MMSys 2016 15 / 36

Impact of 3D bookmarks on Navigation and Streaming in a Networked Virtual Environment Quality of Experience Results

Stats on 51 users

BM type #Exp Mean # coins # completed Mean time No bookmarks 51 7.08 18 4:16 min Arrows 51 7.39 27 2:33 min Viewports 51 7.51 30 2:16 min

Table: Analysis of the sessions length and users success by type of bookmarks

BM type Total distance Fly-to distance Ratio No bookmarks 610.80 N/A N/A Arrows 586.30 369.77 63% Viewports 546.96 332.72 61 %

Table: Analysis of the length of the paths by type of bookmarks

• T. Forgione, A. Carlier, G. Morin, W.T. Ooi, V. Charvillat

MMSys 2016 16 / 36

Impact of 3D bookmarks on Navigation and Streaming in a Networked Virtual Environment Quality of Experience Results

Questionnaire

Questions Answers What was the difficulty level without bookmark? 3.04 / 5 ±0.31 (99%) What was the difficulty level with bookmark? 2.15 / 5 ±0.30 (99%) Did the bookmarks help you to find the coins? 42 Yes, 5 No Did the bookmarks help you to browse the scene? 49 Yes, 2 No Do you think bookmarks can be helpful? 49 Yes, 2 No Which bookmark style do you prefer and why? 32 Arrows, 7 Viewports Did you enjoy the experiment ? 36 Yes, 3 No

Table: List of questions in the questionnaire and summary of answers.

= ⇒ Quality of experience is better with bookmarks

• T. Forgione, A. Carlier, G. Morin, W.T. Ooi, V. Charvillat

MMSys 2016 17 / 36

Impact of 3D bookmarks on Navigation and Streaming in a Networked Virtual Environment Quality of Experience Results

From users’ paths to system queries

10 20 30 40 50 60 0.2 0.4 0.6 Time (in s) Percentage of triangles Without bookmarks With bookmarks

Figure: Percentage of polygons queried after a certain period of time

Higher demand to the server with bookmarks Potential impact on QoS

• T. Forgione, A. Carlier, G. Morin, W.T. Ooi, V. Charvillat

MMSys 2016 18 / 36

Impact of 3D bookmarks on Navigation and Streaming in a Networked Virtual Environment Quality of Experience Results

Impact of bookmarks on navigation Faster navigation Better performance w.r.t. the task Good QoE, task perceived as simpler Higher demand to the server

• T. Forgione, A. Carlier, G. Morin, W.T. Ooi, V. Charvillat

MMSys 2016 19 / 36

Impact of 3D bookmarks on Navigation and Streaming in a Networked Virtual Environment Impact on Quality of Service

Outline

1

Navigation in NVE Controls 3D Bookmarks

2

Quality of Experience Experimental setup Results

3

Impact on Quality of Service Streaming policies Comparison

• T. Forgione, A. Carlier, G. Morin, W.T. Ooi, V. Charvillat

MMSys 2016 20 / 36

Impact of 3D bookmarks on Navigation and Streaming in a Networked Virtual Environment Impact on Quality of Service

Streaming simulation

Client Server

Figure: Streaming model

• T. Forgione, A. Carlier, G. Morin, W.T. Ooi, V. Charvillat

MMSys 2016 21 / 36

Impact of 3D bookmarks on Navigation and Streaming in a Networked Virtual Environment Impact on Quality of Service

Streaming simulation

Client Server C l i e n t i n f

• Camera position

Frustum of the camera Bookmark clicked if any Client info :

Figure: Streaming model

• T. Forgione, A. Carlier, G. Morin, W.T. Ooi, V. Charvillat

MMSys 2016 21 / 36

Impact of 3D bookmarks on Navigation and Streaming in a Networked Virtual Environment Impact on Quality of Service

Streaming simulation

Client Server C l i e n t i n f

• Camera position

Frustum of the camera Bookmark clicked if any Client info : Compute faces to send according to a streaming policy

Figure: Streaming model

• T. Forgione, A. Carlier, G. Morin, W.T. Ooi, V. Charvillat

MMSys 2016 21 / 36

Impact of 3D bookmarks on Navigation and Streaming in a Networked Virtual Environment Impact on Quality of Service

Streaming simulation

Client Server C l i e n t i n f

• Camera position

Frustum of the camera Bookmark clicked if any Client info : Compute faces to send according to a streaming policy P

• l

y g

• n

s

Figure: Streaming model

• T. Forgione, A. Carlier, G. Morin, W.T. Ooi, V. Charvillat

MMSys 2016 21 / 36

Impact of 3D bookmarks on Navigation and Streaming in a Networked Virtual Environment Impact on Quality of Service

Streaming simulation

Client Server C l i e n t i n f

• Camera position

Frustum of the camera Bookmark clicked if any Client info : Compute faces to send according to a streaming policy P

• l

y g

• n

s Update camera (Evaluate rendering)

Figure: Streaming model

• T. Forgione, A. Carlier, G. Morin, W.T. Ooi, V. Charvillat

MMSys 2016 21 / 36

Impact of 3D bookmarks on Navigation and Streaming in a Networked Virtual Environment Impact on Quality of Service

Streaming simulation

Client Server C l i e n t i n f

• Camera position

Frustum of the camera Bookmark clicked if any Client info : Compute faces to send according to a streaming policy P

• l

y g

• n

s Update camera (Evaluate rendering) C l i e n t i n f

• Figure: Streaming model
• T. Forgione, A. Carlier, G. Morin, W.T. Ooi, V. Charvillat

MMSys 2016 21 / 36

Impact of 3D bookmarks on Navigation and Streaming in a Networked Virtual Environment Impact on Quality of Service

Rendering evaluation

One 3D model is fully loaded at the beginning One 3D model is loading progressively We compute the rendering on the fully loaded model R = Pixelsalready loaded face Pixelsall faces

Figure: A loading 3D scene with R = 0.9

• T. Forgione, A. Carlier, G. Morin, W.T. Ooi, V. Charvillat

MMSys 2016 22 / 36

Impact of 3D bookmarks on Navigation and Streaming in a Networked Virtual Environment Impact on Quality of Service Streaming policies

A naive streaming policy

C : Frustum culling / Backface culling Send only polygons that may intersect the frustum of the camera Send only polygons that are oriented towards the camera Sort polygons by distance to the camera

• T. Forgione, A. Carlier, G. Morin, W.T. Ooi, V. Charvillat

MMSys 2016 23 / 36

Impact of 3D bookmarks on Navigation and Streaming in a Networked Virtual Environment Impact on Quality of Service Streaming policies

C : Frustum culling / Backface culling

Figure: Frustum culling

• T. Forgione, A. Carlier, G. Morin, W.T. Ooi, V. Charvillat

MMSys 2016 24 / 36

Impact of 3D bookmarks on Navigation and Streaming in a Networked Virtual Environment Impact on Quality of Service Streaming policies

Two fold issue

1 2 3 4 5 6 7 8 0.6 0.7 0.8 0.9 1 Time (in s) R

Figure: Evaluation of the naive streaming policy

• T. Forgione, A. Carlier, G. Morin, W.T. Ooi, V. Charvillat

MMSys 2016 25 / 36

Impact of 3D bookmarks on Navigation and Streaming in a Networked Virtual Environment Impact on Quality of Service Streaming policies

Two fold issue

1 2 3 4 5 6 7 8 0.6 0.7 0.8 0.9 1 Time (in s) R C only

Figure: Evaluation of the naive streaming policy

• T. Forgione, A. Carlier, G. Morin, W.T. Ooi, V. Charvillat

MMSys 2016 25 / 36

Impact of 3D bookmarks on Navigation and Streaming in a Networked Virtual Environment Impact on Quality of Service Streaming policies

Server-side rendering Find visible triangles Sort them by projected area

Figure: Server-side rendering

• T. Forgione, A. Carlier, G. Morin, W.T. Ooi, V. Charvillat

MMSys 2016 26 / 36

Impact of 3D bookmarks on Navigation and Streaming in a Networked Virtual Environment Impact on Quality of Service Streaming policies

Impact of server-side rendering

0.2 0.4 0.6 0.8 1 1.2 1.4 1.6 1.8 2

·104

0.2 0.4 0.6 0.8 1 Number of Triangles Received R Culling / distance sort Precomputed faces / area sort Great impact on QoS Not scalable : the server cannot do rendering for everyone But we can do it on bookmarks =

⇒ Precomputation

• T. Forgione, A. Carlier, G. Morin, W.T. Ooi, V. Charvillat

MMSys 2016 27 / 36

Impact of 3D bookmarks on Navigation and Streaming in a Networked Virtual Environment Impact on Quality of Service Streaming policies

Impact of server-side rendering

0.2 0.4 0.6 0.8 1 1.2 1.4 1.6 1.8 2

·104

0.2 0.4 0.6 0.8 1 Number of Triangles Received R Culling / distance sort Precomputed faces / area sort Great impact on QoS Not scalable : the server cannot do rendering for everyone But we can do it on bookmarks =

⇒ Precomputation

• T. Forgione, A. Carlier, G. Morin, W.T. Ooi, V. Charvillat

MMSys 2016 27 / 36

Impact of 3D bookmarks on Navigation and Streaming in a Networked Virtual Environment Impact on Quality of Service Streaming policies

A short review on prefetching

Park et al. [2001] Predict the user’s interest in an object Sort objects by interest and prefetch Hung and Liu [2006] Cluster the navigation path of users Predict future navigation path Prefetch objects near predicted path Zhou et al. [2015] Learn object access patterns Predict objects that are likely to be accessed and prefetch

• T. Forgione, A. Carlier, G. Morin, W.T. Ooi, V. Charvillat

MMSys 2016 28 / 36

Impact of 3D bookmarks on Navigation and Streaming in a Networked Virtual Environment Impact on Quality of Service Streaming policies

C : Frustum culling / Backface culling Send only polygons that may intersect the frustum of the camera Send only polygons that are oriented towards the camera FD : Fetch destination Send the triangles that are seen by the bookmark clicked PP : Prefetch predicted Send the triangles that are seen by the bookmarks that might be clicked

• T. Forgione, A. Carlier, G. Morin, W.T. Ooi, V. Charvillat

MMSys 2016 29 / 36

Impact of 3D bookmarks on Navigation and Streaming in a Networked Virtual Environment Impact on Quality of Service Streaming policies

C : Frustum culling / Backface culling Send only polygons that may intersect the frustum of the camera Send only polygons that are oriented towards the camera FD : Fetch destination Send the triangles that are seen by the bookmark clicked PP : Prefetch predicted Send the triangles that are seen by the bookmarks that might be clicked

• T. Forgione, A. Carlier, G. Morin, W.T. Ooi, V. Charvillat

MMSys 2016 30 / 36

Impact of 3D bookmarks on Navigation and Streaming in a Networked Virtual Environment Impact on Quality of Service Streaming policies

PP : Prefetch Predicted Try to guess the next bookmark clicked knowing the previous one

1 2 3 4 5 6 7 8 9 10 11 1 2 3 4 5 6 7 8 9 10 11 Previous recommendation clicked Next recommendation clicked

Figure: Patterns of access for a specific scene

• T. Forgione, A. Carlier, G. Morin, W.T. Ooi, V. Charvillat

MMSys 2016 31 / 36

Impact of 3D bookmarks on Navigation and Streaming in a Networked Virtual Environment Impact on Quality of Service Streaming policies

PP : Prefetch Predicted Try to guess the next bookmark clicked knowing the previous one

1 2 3 4 5 6 7 8 9 10 11 1 2 3 4 5 6 7 8 9 10 11 Previous recommendation clicked Next recommendation clicked

Figure: Patterns of access for a specific scene

Frustum culling Bookmark j Bookmark i

Figure: Prefetching packet

• T. Forgione, A. Carlier, G. Morin, W.T. Ooi, V. Charvillat

MMSys 2016 31 / 36

Impact of 3D bookmarks on Navigation and Streaming in a Networked Virtual Environment Impact on Quality of Service Comparison

Quality of rendering after a click on a bookmark

1 2 3 4 5 6 7 8 0.6 0.7 0.8 0.9 1 Time (in s) R

Figure: Comparison of streaming policies

• T. Forgione, A. Carlier, G. Morin, W.T. Ooi, V. Charvillat

MMSys 2016 32 / 36

Impact of 3D bookmarks on Navigation and Streaming in a Networked Virtual Environment Impact on Quality of Service Comparison

Quality of rendering after a click on a bookmark

1 2 3 4 5 6 7 8 0.6 0.7 0.8 0.9 1 Time (in s) R C only

Figure: Comparison of streaming policies

• T. Forgione, A. Carlier, G. Morin, W.T. Ooi, V. Charvillat

MMSys 2016 32 / 36

Impact of 3D bookmarks on Navigation and Streaming in a Networked Virtual Environment Impact on Quality of Service Comparison

Quality of rendering after a click on a bookmark

1 2 3 4 5 6 7 8 0.6 0.7 0.8 0.9 1 Time (in s) R C only V-PP

Figure: Comparison of streaming policies

• T. Forgione, A. Carlier, G. Morin, W.T. Ooi, V. Charvillat

MMSys 2016 32 / 36

Impact of 3D bookmarks on Navigation and Streaming in a Networked Virtual Environment Impact on Quality of Service Comparison

Quality of rendering after a click on a bookmark

1 2 3 4 5 6 7 8 0.6 0.7 0.8 0.9 1 Time (in s) R C only V-PP V-FD

Figure: Comparison of streaming policies

• T. Forgione, A. Carlier, G. Morin, W.T. Ooi, V. Charvillat

MMSys 2016 32 / 36

Impact of 3D bookmarks on Navigation and Streaming in a Networked Virtual Environment Impact on Quality of Service Comparison

Quality of rendering after a click on a bookmark

1 2 3 4 5 6 7 8 0.6 0.7 0.8 0.9 1 Time (in s) R C only V-PP V-FD V-PP+FD

Figure: Comparison of streaming policies

• T. Forgione, A. Carlier, G. Morin, W.T. Ooi, V. Charvillat

MMSys 2016 32 / 36

Impact of 3D bookmarks on Navigation and Streaming in a Networked Virtual Environment

Improvement with prediction is not significant

Conclusion on policies Frustum culling is not satisfying Prefetching is better than frustum culling but not significantly (due to low predictability in 3D) Fetch destination is really efficient Fetch destination + Prefetching is barely better

• T. Forgione, A. Carlier, G. Morin, W.T. Ooi, V. Charvillat

MMSys 2016 33 / 36

Impact of 3D bookmarks on Navigation and Streaming in a Networked Virtual Environment

Conclusion

Bookmarks’ impact on NVE Good for QoE Drawback on QoS Prerendering the bookmarks can limit this drawback Future work Optimize chunk size allocated for prefetching Adapt fly-to transition to bandwidth Study the influence of the bookmarks’ positioning on user browsing pattern

• T. Forgione, A. Carlier, G. Morin, W.T. Ooi, V. Charvillat

MMSys 2016 34 / 36

Impact of 3D bookmarks on Navigation and Streaming in a Networked Virtual Environment

Questions ?

• T. Forgione, A. Carlier, G. Morin, W.T. Ooi, V. Charvillat

MMSys 2016 35 / 36

Impact of 3D bookmarks on Navigation and Streaming in a Networked Virtual Environment

Shao-Shin Hung and Damon Shing-Min Liu. Using prefetching to improve walkthrough latency: Research articles. Comput.

• Animat. Virtual Worlds, 17(3-4):469–478, July 2006. ISSN

1546-4261. doi: 10.1002/cav.v17:3/4. URL http://dx.doi.org/10.1002/cav.v17:3/4. Jacek Jankowski and Martin Hachet. Advances in interaction with 3d environments. Computer Graphics Forum, 34(1):152–190, 2015. Morgan McGuire. Computer graphics archive, August 2011. URL http://graphics.cs.williams.edu/data. Sungju Park, Dongman Lee, Mingyu Lim, and Chansu Yu. Scalable data management using user-based caching and prefetching in distributed virtual environments. In Proceedings

• f the ACM Symposium on Virtual Reality Software and

Technology, pages 121–126, Banff, Canada, 11 2001. Roy A Ruddle, Andrew Howes, Stephen J Payne, and Dylan M

• Jones. The effects of hyperlinks on navigation in virtual
• environments. International Journal of Human-Computer
• T. Forgione, A. Carlier, G. Morin, W.T. Ooi, V. Charvillat

MMSys 2016 36 / 36