60 MS TO GET IT RIGHT GAZE-CONTINGENT RENDERING & HUMAN - - PowerPoint PPT Presentation

▶
60 ms to get it right
SMART_READER_LITE
LIVE PREVIEW

60 MS TO GET IT RIGHT GAZE-CONTINGENT RENDERING & HUMAN - - PowerPoint PPT Presentation

Nov 22, 2022 268 likes •596 views

60 MS TO GET IT RIGHT GAZE-CONTINGENT RENDERING & HUMAN PERCEPTION Dr. Rachel Albert, GTC San Jose 2019 GOAL: IMMERSIVE VR Requirements to mimic real-world human visual experience Wide color gamut / high dynamic range High frame

slide-1
SLIDE 1
  • Dr. Rachel Albert, GTC San Jose 2019

60 MS TO GET IT RIGHT

GAZE-CONTINGENT RENDERING & HUMAN PERCEPTION

slide-2
SLIDE 2

2

GOAL: IMMERSIVE VR

Requirements to mimic real-world human visual experience

  • Wide color gamut / high dynamic range
  • High frame rate, Low latency
  • Wide Field of View (FoV)
  • High spatial resolution
slide-3
SLIDE 3

3

GOAL: IMMERSIVE VR

Requirements to mimic real-world human visual experience

  • Wide color gamut / high dynamic range
  • High frame rate, Low latency
  • Wide Field of View (FoV)
  • High spatial resolution
slide-4
SLIDE 4

4

GOAL: IMMERSIVE VR

Requirements to mimic real-world human visual experience

  • Wide color gamut / high dynamic range
  • High frame rate, Low latency
  • Wide Field of View (FoV)
  • High spatial resolution

Computation Cost Bandwidth Power/Heat/Weight

slide-5
SLIDE 5

5

HUMAN VISUAL PERCEPTION IS LIMITED

Detection Threshold

slide-6
SLIDE 6

6

HUMAN VISUAL PERCEPTION IS LIMITED

Detection Threshold Resolution Threshold

slide-7
SLIDE 7

7

Resolvable spatial resolution decreases in the periphery as a function of eccentricity.

Figure from Anderson et al. 1991

HUMAN VISUAL PERCEPTION IS NON-UNIFORM

Achromatic Acuity Chromatic Acuity

slide-8
SLIDE 8

8

ONE SOLUTION: GAZE-CONTINGENT IMAGERY

slide-9
SLIDE 9

9

SYSTEM REQUIREMENTS FOR GAZE-CONTINGENT RENDERING

Multi-Resolution Display (Varjo) Display-side decompression (Google/LG) Wireless Image Compression? Variable-Rate Shading (NVIDIA Turing GPU) Rendering Optimizations (Patney et al. 2016) Eye Tracking (Pupil Labs, Tobii)

slide-10
SLIDE 10

10

SACCADIC EYE MOVEMENTS & SACCADIC OMISSION

  • During a saccade, the eye moves

at 500 deg/sec

  • Latency may cause the

peripheral image to be visible

  • BUT

, Saccadic omission blocks perception of this image How long is the omission period for rendered scenes?

slide-11
SLIDE 11

11

EXPERIMENT: LATENCY REQUIREMENTS FOR GAZE-CONTINGENT RENDERING

  • Eye tracking latency only
  • Free viewing, 4 seconds per trial
  • Task: Yes/No Response

“Did you see any artifacts?”

  • Amount of peripheral degradation varied based on response

1 up/ 1 down adaptive staircase

slide-12
SLIDE 12

12

EXPERIMENTAL DESIGN

Gaussian

Fovea Size

Full-Resolution Area 5°, 10°, 20° Ecc.

Eye-Tracking Latency

Added Latency 0, 10, 20, 40, 80, 150 ms

Foveation Technique

Peripheral Rendering Method Subsampling, Gaussian, fCPS Gaussian

slide-13
SLIDE 13

13

“MOTION TO PHOTON” SACCADE START TO SCREEN UPDATE

Gaussian

  • HTC Vive + SMI Eye Tracker
  • NVIDIA Titan X (Pascal) GPU + Falcor rendering framework
slide-14
SLIDE 14

14

RESULTS

slide-15
SLIDE 15

15

INTERACTION BETWEEN TECHNIQUE & ECCENTRICITY

slide-16
SLIDE 16

16

MAIN EFFECT OF LATENCY (THRESHOLD AT 50-70 MS)

slide-17
SLIDE 17

17

LATENCY EXPERIMENT TAKEAWAY POINTS

  • Temporally stable foveation methods support more

foveation at larger eccentricities

  • All foveation methods and eccentricities show

significantly worse performance above 50-70 ms latency

slide-18
SLIDE 18

18

PERIPHERALLY DEGRADING IMAGES WITH COLOR

Reduce Saturation Reduce Bit Depth

slide-19
SLIDE 19

19

PERIPHERALLY DEGRADING IMAGES WITH COLOR

Reduce Saturation Reduce Bit Depth

slide-20
SLIDE 20

20

EXPERIMENT: REDUCE BIT DEPTH WHEN DOES THE VIEWER SEE FOVEATION?

slide-21
SLIDE 21

21

RESULT: LOW SPATIAL FREQUENCY CONTENT

slide-22
SLIDE 22

22

RESULT: MID SPATIAL FREQUENCY CONTENT

slide-23
SLIDE 23

23

RESULT: HIGH SPATIAL FREQUENCY CONTENT

slide-24
SLIDE 24

24

COLOR EXPERIMENT TAKEAWAY POINTS

  • Some bit depth reduction is not noticeable in the

periphery

  • The level of acceptable bit depth reduction is highly

dependent on spatial frequency of the content

slide-25
SLIDE 25

25

BIGGEST CHALLENGE: TEXT

High Contrast, High Spatial Frequency Content

Blurring breaks contrast constancy Aliasing produces flicker

slide-26
SLIDE 26

26

PERIPHERAL SPATIAL METAMERS

Freeman & Simoncelli, 2011 Original Image Metamer #1 Metamer #2

slide-27
SLIDE 27

27

  • Lossy compression of

spatial information

  • Peripheral perception

is complicated

PERIPHERAL SPATIAL METAMERS

slide-28
SLIDE 28
  • Dr. Rachel Albert, GTC San Jose 2019

60 MS TO GET IT RIGHT

GAZE-CONTINGENT RENDERING & HUMAN PERCEPTION

slide-29
SLIDE 29
  • Dr. Rachel Albert, GTC San Jose 2019

60 MS TO GET IT RIGHT

GAZE-CONTINGENT RENDERING & HUMAN PERCEPTION

THANK YOU

slide-30
SLIDE 30
slide-31
SLIDE 31

31

EXPERIMENTAL DESIGN

  • Task – move eyes and head to try to see artifacts

4 second stimulus duration per trial Yes/No Response (“Did you see any artifacts?”)

  • Amount of foveation varied based on responses

1 up/ 1 down adaptive staircase Foveation level = shading rate per 2^N pixels (higher is better)

  • Controlled for actual system latency, head & eye motion
slide-32
SLIDE 32

32

SYSTEM REQUIREMENTS FOR GAZE-CONTINGENT RENDERING

Eye Tracking

  • Accurate
  • Fast

Rendering

  • Cheap, Fast
  • Perceptually

Matched Display

  • Low Persistence
  • Low Latency