SLIDE 1
High-Fidelity Light Field VR Playback Using NVIDIA GPUs
Tim Milliron, Vice President of Engineering Nikhil Karnad, Architect for Image-Based Rendering
1
2
High-Fidelity Light Field VR Playback Using NVIDIA GPUs Tim - - PowerPoint PPT Presentation
High-Fidelity Light Field VR Playback Using NVIDIA GPUs Tim - - PowerPoint PPT Presentation
High-Fidelity Light Field VR Playback Using NVIDIA GPUs Tim Milliron, Vice President of Engineering Nikhil Karnad, Architect for Image-Based Rendering 1 An Introduction to Light Field 2 THE LIGHT FIELD HOW IT WORKS KEY BENEFITS Capture
SLIDE 2
SLIDE 3
3
CONFIDENTIAL
THE LIGHT FIELD
KEY BENEFITS
⨠
Every pixel contains color, brightness and depth properties (RGBZ)
⨠
Every scene becomes a 3D model vs. a flat 2D image
⨠
Each captured perspective encodes proper view-dependent illumination HOW IT WORKS
⨠
Capture the natural flow of every ray of light
⨠
Leverages either hundreds of individual cameras or millions of microscopic lenses
⨠
Requires tightly coupled hardware and software
SLIDE 4
4
Light Field for VR
SLIDE 5
5
THE LIGHT FIELD VOLUME
⨠ Capture ray data from every angle at all
locations entering a given volume at high frame rate
⨠ Generate virtual views
from any point within the volume, facing any direction, with any field of view.
⨠ Breakthrough sense of presence & realism
for live action VR
SLIDE 6
6
THE LIGHT FIELD FOR VR - 6DOF
⨠ Parallax
– Ability to “see behind objects”
⨠ View Dependent Illumination
– Specular highlights, reflections, …
⨠ Truly Correct Stereo
– Any viewing orientation - even when the viewer’s head is sideways – Any inter-ocular distance - adjustable
- n-the-fly
SLIDE 7
7
Lytro Immerge
SLIDE 8
LYTRO IMMERGE
Capture
- Configurable multi-
camera system
Processing
- Color matching
- Depth estimation
Rendering
- Rendering process to
generate final assets for efficient playback and compression
Playback
- Light-field playback in
leading HMDs
SLIDE 9
CONFIDENTIAL
9
PRODUCTION-READY
⨠ Planar configuration comprised of
95 individual cameras
⨠ Nodal capture in 5 “wedges” ⨠ 475 cameras used to synthesize
a full 360 view
⨠ Generates a 1-meter-wide “Viewing Volume” ⨠ Designed for modern high-end production ⨠ Director & film-crew can be behind the camera ⨠ Works well in practical on-set conditions ⨠ Highest resolution camera on the market today -
up to 8k 360 resolution
SLIDE 10
10
CREATING A 360 LIGHT FIELD VOLUME
Lytro Immerge Planar Configuration captures the environment from one direction creating a “wedge”
CONFIDENTIAL
1
Rotate to Capture “Wedges” five rotations to film a full 360 view of the environment
2 3 4 5
Light Field Merging reconstructs the data into a 360 Light Field volume, enabling 6DoF movement for viewers during playback
SLIDE 11
11
What’s the catch?
SLIDE 12
What’s the catch?
12
SLIDE 13
100-1,000x data
13
The Bad News
SLIDE 14
14
The Good News
SLIDE 15
15
The Good News
SLIDE 16
16
Light Field Capture & Playback is (barely) within reach today…
SLIDE 17
17
Lytro Immerge Rendering & Playback with NVIDIA GPUs
SLIDE 18
18
Light Field rendering: Requirements
⨠ Need two views, one per eye ⨠ High quality throughout the 6-
DoF viewing volume
⨠ Close objects need to shift
relative to far ones
⨠ Fill in occlusions seamlessly ⨠ Illumination variation across
views should not be lost
⨠ Low latency, typically 90+ fps
SLIDE 19
19
Light Field rendering: Requirements
Viewing volume
DoF: Degrees of freedom
⨠ Need two views, one per eye ⨠ High quality throughout the 6-
DoF viewing volume
⨠ Close objects need to shift
relative to far ones
⨠ Fill in occlusions seamlessly ⨠ Illumination variation across
views should not be lost
⨠ Low latency, typically 90+ fps
SLIDE 20
20
Viewing volume
Light Field rendering: Requirements
DoF: Degrees of freedom
⨠ Need two views, one per eye ⨠ High quality throughout the 6-
DoF viewing volume
⨠ Close objects need to shift
relative to far ones
⨠ Fill in occlusions seamlessly ⨠ Illumination variation across
views should not be lost
⨠ Low latency, typically 90+ fps
SLIDE 21
21
Viewing volume
Light Field rendering: Requirements
DoF: Degrees of freedom
⨠ Need two views, one per eye ⨠ High quality throughout the 6-
DoF viewing volume
⨠ Close objects need to shift
relative to far ones
⨠ Fill in occlusions seamlessly ⨠ Illumination variation across
views should not be lost
⨠ Low latency, typically 90+ fps
SLIDE 22
22
Viewing volume
Light Field rendering: Requirements
⨠ Need two views, one per eye ⨠ High quality throughout the 6-
DoF viewing volume
⨠ Close objects need to shift
relative to far ones
⨠ Fill in occlusions seamlessly ⨠ Illumination variation across
views should not be lost
⨠ Low latency, typically 90+ fps
DoF: Degrees of freedom
SLIDE 23
23
Viewing volume
⨠ Need two views, one per eye ⨠ High quality throughout the 6-
DoF viewing volume
⨠ Close objects need to shift
relative to far ones
⨠ Fill in occlusions seamlessly ⨠ Illumination variation across
views should not be lost
⨠ Low latency, typically 90+ fps
Light Field rendering: Requirements
DoF: Degrees of freedom
SLIDE 24
24
Light Field rendering: Background
1 2 3 4 5
Theoretical two-plane Light Field parametrization
⨠ Each wedge is a planar camera array ⨠ Captured rays sample the plenoptic
function – Light slab [Levoy and Hanrahan, 1996] – Lumigraph [Gortler et al., 1996] – Digital Light Field Photography [Ng, 2006] – Light Field Camera Design [Wei et al., 2015]
SLIDE 25
25
Light Field rendering: Background
⨠ Each wedge is a planar camera array ⨠ Captured rays sample the plenoptic
function – Light slab [Levoy and Hanrahan, 1996] – Lumigraph [Gortler et al., 1996] – Digital Light Field Photography [Ng, 2006] – Light Field Camera Design [Wei et al., 2015]
Lytro Light Field camera 2012 Lytro ILLUM 2014
SLIDE 26
26
Lytro Immerge rendering: Practice
⨠ LOTS of captured rays!
– Hundreds of cameras – Video capture rate – Tens of billions of rays per sec
⨠ Per-ray payload multiplies
data size
1 2 3 4 5
Subset of the Light Field captured for Hallelujah
SLIDE 27
27
Lytro Immerge rendering: Practice
⨠ We use NVIDIA GPUs to crunch
these massive datasets
⨠ Assuming we could sift through
lots of rays per second during HMD render…
⨠ Upload to GPU would still be a
bottleneck – Bandwidth requirement of hundreds of GB/s
1 2 3 4 5
Image courtesy NVIDIA http://images.nvidia.com/pascal/img/titanx/titanx-design.png HMD: Head-mounted display
SLIDE 28
28
Lytro Immerge rendering: Acceleration
⨠ Goal
– Unburden the GPU from having to consider the entire dataset
⨠ Solution
– A proprietary acceleration structure that caches rays
Camera center
- f perspective
Eye center
- f perspective (in HMD)
HMD: Head-mounted display
SLIDE 29
29
Lytro Immerge playback: Example
Left-eye and right-eye views rendered to the HMD
1 2
Limited field of view capture across two wedges
HMD: Head-mounted display
SLIDE 30
30
Lytro Immerge playback: Pipeline
⨠ Load: Cache allows order-of-magnitude improvement
– Under 10 GB/s
⨠ Decode: Proprietary compression technique for further bandwidth
reduction – Both CPU and GPU used
⨠ Shaders: Compute and graphics ⨠ OpenGL and DirectX implementations
SLIDE 31
31
Lytro Immerge playback: Performance
⨠ Require 90 fps or higher, i.e., 11 ms or faster ⨠ Careful balance between compute and graphics ⨠ Average GPU render times for Hallelujah
– 980 Ti : 10.3 ms – Titan X Maxwell : 9.2 ms – Titan X Pascal : 6.5 ms
SLIDE 32
32
Acknowledgements
⨠ Representing the work of the entire LYTRO IMMERGE team ⨠ Shoutouts specifically for this section of the talk
– Kurt Akeley, Trevor Carothers, Zeyar Htet, Derek Pang, Mike Ma, Alex Song, Cathy Ashenbremer
⨠ Contact information
– Nikhil Karnad <nkarnad@lytro.com> – Tim Milliron @timmilliron
SLIDE 33
CONFIDENTIAL
33
CONFIDENTIAL
QUESTIONS & ANSWERS
SLIDE 34
High-Fidelity Light Field VR Playback Using NVIDIA GPUs
Tim Milliron, Vice President of Engineering Nikhil Karnad, Architect for Image-Based Rendering
34