Real-time 3D Eyelids Tracking From Semantic Edges Quan Wen, Feng - - PowerPoint PPT Presentation

Real-time 3D Eyelids Tracking From Semantic Edges

Quan Wen, Feng Xu, Ming Lu, Jun-Hai Yong Tsinghua University

• 1-

Background

Facial capture and animation is crucial in many applications

Face capture in computer games Face animation in movies

• 2-

Background

Facial tracking focus less on the eyes

[Cao et al. 2015] [Hsieh et al. 2015] [Liu et al. 2015] [Bouaziz et al. 2013] [Cao et al. 2014] [Li et al. 2013]

• 3-

Background

Facial organs tracking

[Edwards et al. 2016] [Wu et al. 2016] [Bérard et al. 2016] [Wood et al. 2016] [Bermano et al. 2015] [Wang et al. 2016] [Wen et al. 2016]

• 4-

Our Work

A real-time 3D eyelids tracking system

• 5-

Overview

Input image

?

Eyelid Reconstruction

• 6-

Overview

Input image Eyelid features Eyelid Reconstruction

• 7-

Overview

Input image Eyelid features Eyelid models Eyelid Reconstruction

• 8-

Overview

Face & Eyeball Fitting [Wen et al. 2016] Input Depth Face & Eyeball Result Curve-based Eyelid Reconstruction Final Result Two Eyelid Linear Models Eyelid Edge Detection & Identification Input Color Edge Result Edge Maps for Training

• 9-

Overview

Eyelid Edge Detection & Identification Face & Eyeball Fitting [Wen et al. 2016] Input Color Edge Result Input Depth Face & Eyeball Result Edge Maps for Training Curve-based Eyelid Reconstruction Final Result Two Eyelid Linear Models

• 10-

Overview

Eyelid Edge Detection & Identification Face & Eyeball Fitting [Wen et al. 2016] Input Color Input Depth Face & Eyeball Result Edge Maps for Training Edge Result Curve-based Eyelid Reconstruction Eyelid Result Two Eyelid Linear Models

• 11-

Overview

Eyelid Edge Detection & Identification Face & Eyeball Fitting [Wen et al. 2016] Input Color Input Depth Edge Maps for Training Edge Result Curve-based Eyelid Reconstruction Final Result Face & Eyeball Result Two Eyelid Linear Models

• 12-

Overview

Eyelid Edge Detection & Identification Face & Eyeball Fitting [Wen et al. 2016] Input Color Edge Result Input Depth Face & Eyeball Result Edge Maps for Training Curve-based Eyelid Reconstruction Final Result Two Eyelid Linear Models

• 13-

Eyelid Edge Detection and Identification

Face & Eyeball Fitting [Wen et al. 2016] Input Depth Face & Eyeball Result Curve-based Eyelid Reconstruction Final Result Two Eyelid Linear Models Eyelid Edge Detection & Identification Input Color Edge Result Edge Maps for Training

• 14-

Semantic Eyelid Edges

Main features of the eyes: double-fold, top eyelid, bottom eyelid, bulge

• 15-

Semantic Eyelid Edges

Main features of the eyes: double-fold, top eyelid, bottom eyelid, bulge

• 16-

Semantic Eyelid Edges

Main features of the eyes: double-fold, top eyelid, bottom eyelid, bulge

• 17-

Semantic Eyelid Edges

Main features of the eyes: double-fold, top eyelid, bottom eyelid, bulge

• 18-

Semantic Eyelid Edges

Main features of the eyes: double-fold, top eyelid, bottom eyelid, bulge

• 19-

Semantic Eyelid Edges

Main features of the eyes: double-fold, top eyelid, bottom eyelid, bulge

• 20-

Network

DNN in HED

HED [Xie and Tu 2015]

1-channel Sigmoid Cross-entropy Loss

• 21-

Network

DNN in HED

HED [Xie and Tu 2015]

1-channel Sigmoid Cross-entropy Loss Training Set

··· ···

• 22-

Network

DNN in HED

HED [Xie and Tu 2015]

1-channel Sigmoid Cross-entropy Loss Training Set

··· ···

Network output

• 23-

Network

DNN in HED

HED [Xie and Tu 2015]

1-channel Sigmoid Cross-entropy Loss

Proposed eyelid edge detection and identification

Proposed DNN 4-channel Sigmoid Cross-entropy Loss

• 24-

Proposed DNN 4-channel Sigmoid Cross-entropy Loss

Network

DNN in HED

HED [Xie and Tu 2015]

1-channel Sigmoid Cross-entropy Loss Training Set

··· ···

No double-fold No bulge

• 25-

Proposed DNN 4-channel Sigmoid Cross-entropy Loss

Network

DNN in HED 1-channel Sigmoid Cross-entropy Loss Network output

• 26-

Eyelid Edge Detection and Identification

Results

• 27-

Eyelid Linear Models

Eyelid Edge Detection & Identification Face & Eyeball Fitting [Wen et al. 2016] Input Color Edge Result Input Depth Face & Eyeball Result Edge Maps for Training Curve-based Eyelid Reconstruction Final Result Two Eyelid Linear Models

• 28-

Shape Linear Rig

Position Contour shape Double-fold Bulge

Eyelid shape categories

• 29-

Shape Linear Rig

𝑐0

𝑗𝑒

(basic) 𝑐11

𝑗𝑒

(contour: downturned) 𝑐21

𝑗𝑒

(double-fold: single) 𝑐23

𝑗𝑒

(bulge: parallel)

Linear rig 𝐶𝑗𝑒 𝐶𝑗𝑒 = 𝑐𝑙

𝑗𝑒|𝑙 = 0, … , 𝑂𝑗𝑒 − 1 , 𝑂𝑗𝑒 = 29 𝑂𝑗𝑒 number of 𝑐𝑙

𝑗𝑒

𝑐𝑙

𝑗𝑒 models in 𝐶𝑗𝑒

• 30-

Shape Linear Rig

𝑐0

𝑗𝑒

(basic) 𝑐21

𝑗𝑒

(double-fold: single) 𝑐23

𝑗𝑒

(bulge: parallel)

Synthesized shape model of a specific user 𝐹𝑂 = 𝑐0

𝑗𝑒 + 𝑙=1 𝑂𝑗𝑒−1

𝑥𝑙

𝑗𝑒(𝑐𝑙 𝑗𝑒 − 𝑐0 𝑗𝑒) 𝑐11

𝑗𝑒

(contour: downturned)

• 31-

Shape Linear Rig

𝑐0

𝑗𝑒

(basic) 𝑐21

𝑗𝑒

(double-fold: single) 𝑐23

𝑗𝑒

(bulge: parallel)

Synthesized shape model of a specific user 𝐹𝑂 = 𝑐0

𝑗𝑒 + 𝑙=1 𝑂𝑗𝑒−1

𝑥𝑙

𝑗𝑒(𝑐𝑙 𝑗𝑒 − 𝑐0 𝑗𝑒) 𝑐0

𝑗𝑒 basic model in 𝐶𝑗𝑒

𝑐11

𝑗𝑒

(contour: downturned)

• 32-

Shape Linear Rig

𝑐0

𝑗𝑒

(basic) 𝑐21

𝑗𝑒

(double-fold: single) 𝑐23

𝑗𝑒

(bulge: parallel)

Synthesized shape model of a specific user 𝐹𝑂 = 𝑐0

𝑗𝑒 + 𝑙=1 𝑂𝑗𝑒−1

𝑥𝑙

𝑗𝑒(𝑐𝑙 𝑗𝑒 − 𝑐0 𝑗𝑒) 𝑐𝑙

𝑗𝑒 shape models in 𝐶𝑗𝑒

𝑐0

𝑗𝑒 basic model in 𝐶𝑗𝑒

𝑐11

𝑗𝑒

(contour: downturned)

• 33-

Shape Linear Rig

𝑐0

𝑗𝑒

(basic) 𝑐21

𝑗𝑒

(double-fold: single) 𝑐23

𝑗𝑒

(bulge: parallel)

Synthesized shape model of a specific user 𝐹𝑂 = 𝑐0

𝑗𝑒 + 𝑙=1 𝑂𝑗𝑒−1

𝑥𝑙

𝑗𝑒(𝑐𝑙 𝑗𝑒 − 𝑐0 𝑗𝑒) 𝑥𝑙

𝑗𝑒 weight of 𝑐𝑙 𝑗𝑒

𝑐11

𝑗𝑒

(contour: downturned) 𝑐𝑙

𝑗𝑒 shape models in 𝐶𝑗𝑒

𝑐0

𝑗𝑒 basic model in 𝐶𝑗𝑒

• 34-

Shape Linear Rig

Synthesized shape model of a specific user

𝐹𝑂 User-specific shape model

= +

𝑙=1 𝑂𝑗𝑒−1

𝑥𝑙

𝑗𝑒(

− )

𝑐0

𝑗𝑒

(basic) 𝑐0

𝑗𝑒

(basic) 𝑐k

𝑗𝑒

(shape)

• 35-

Pose Linear Rig

𝑐0

𝑓𝑦𝑞

(basic) 𝑐3

𝑓𝑦𝑞

(inner close) 𝑐5

𝑓𝑦𝑞

(outer close)

Generic linear rig 𝐶𝑓𝑦𝑞 𝐶𝑓𝑦𝑞 = 𝑐𝑙

𝑓𝑦𝑞|𝑙 = 0, … , 𝑂𝑓𝑦𝑞 − 1 , 𝑂𝑓𝑦𝑞 = 23 𝑐𝑙

𝑓𝑦𝑞 models in 𝐶𝑓𝑦𝑞

𝑂𝑓𝑦𝑞 number of 𝑐𝑙

𝑓𝑦𝑞

𝑐1

𝑓𝑦𝑞

(close)

• 36-

Pose Linear Rig

Generic pose rig 𝐶𝑓𝑦𝑞

Pose models 𝑐𝑙

𝑓𝑦𝑞

Basic model 𝑐0

𝑓𝑦𝑞

• 37-

Pose Linear Rig

Generic pose rig 𝐶𝑓𝑦𝑞

Pose models 𝑐𝑙

𝑓𝑦𝑞

Basic model 𝑐0

𝑓𝑦𝑞

User-specific basic model 𝐹𝑂

• 38-

Pose Linear Rig

Generic pose rig 𝐶𝑓𝑦𝑞

Pose models 𝑐𝑙

𝑓𝑦𝑞

Basic model 𝑐0

𝑓𝑦𝑞

User-specific pose rig 𝐶𝑓𝑦𝑞′

User-specific pose models 𝑐𝑙

𝑓𝑦𝑞′

User-specific basic model 𝐹𝑂

?

• 39-

Pose Linear Rig

Generic pose rig 𝐶𝑓𝑦𝑞 User-specific pose rig 𝐶𝑓𝑦𝑞′

User-specific pose models 𝑐𝑙

𝑓𝑦𝑞′

User-specific basic model 𝐹𝑂

Deformation transfer

Pose models 𝑐𝑙

𝑓𝑦𝑞

Basic model 𝑐0

𝑓𝑦𝑞

• 40-

Pose Linear Rig

User-specific eyelid model in tracking 𝐹𝑄 = 𝑐0

𝑓𝑦𝑞′ + 𝑙=1 𝑂𝑓𝑦𝑞−1

𝑥𝑙

𝑓𝑦𝑞(𝑐𝑙 𝑓𝑦𝑞′ − 𝑐0 𝑓𝑦𝑞′) 𝑐0

𝑓𝑦𝑞′

(basic) 𝑐3

𝑓𝑦𝑞′

(inner close) 𝑐5

𝑓𝑦𝑞′

(outer close) 𝑐1

𝑓𝑦𝑞′

(close)

• 41-

Pose Linear Rig

User-specific eyelid model in tracking 𝐹𝑄 = 𝑐0

𝑓𝑦𝑞′ + 𝑙=1 𝑂𝑓𝑦𝑞−1

𝑥𝑙

𝑓𝑦𝑞(𝑐𝑙 𝑓𝑦𝑞′ − 𝑐0 𝑓𝑦𝑞′) 𝑐0

𝑓𝑦𝑞′ basic model in 𝐶𝑓𝑦𝑞′

𝑐0

𝑓𝑦𝑞′

(basic) 𝑐3

𝑓𝑦𝑞′

(inner close) 𝑐5

𝑓𝑦𝑞′

(outer close) 𝑐1

𝑓𝑦𝑞′

(close)

• 42-

Pose Linear Rig

User-specific eyelid model in tracking 𝐹𝑄 = 𝑐0

𝑓𝑦𝑞′ + 𝑙=1 𝑂𝑓𝑦𝑞−1

𝑥𝑙

𝑓𝑦𝑞(𝑐𝑙 𝑓𝑦𝑞′ − 𝑐0 𝑓𝑦𝑞′) 𝑐𝑙

𝑓𝑦𝑞′ pose models in 𝐶𝑓𝑦𝑞′

𝑐0

𝑓𝑦𝑞′ basic model in 𝐶𝑓𝑦𝑞′

𝑐0

𝑓𝑦𝑞′

(basic) 𝑐3

𝑓𝑦𝑞′

(inner close) 𝑐5

𝑓𝑦𝑞′

(outer close) 𝑐1

𝑓𝑦𝑞′

(close)

• 43-

Pose Linear Rig

User-specific eyelid model in tracking 𝐹𝑄 = 𝑐0

𝑓𝑦𝑞′ + 𝑙=1 𝑂𝑓𝑦𝑞−1

𝑥𝑙

𝑓𝑦𝑞(𝑐𝑙 𝑓𝑦𝑞′ − 𝑐0 𝑓𝑦𝑞′) 𝑥𝑙

𝑓𝑦𝑞 weight of 𝑐𝑙 𝑓𝑦𝑞′

𝑐0

𝑓𝑦𝑞′

(basic) 𝑐3

𝑓𝑦𝑞′

(inner close) 𝑐5

𝑓𝑦𝑞′

(outer close) 𝑐1

𝑓𝑦𝑞′

(close) 𝑐𝑙

𝑓𝑦𝑞′ pose models in 𝐶𝑓𝑦𝑞′

𝑐0

𝑓𝑦𝑞′ basic model in 𝐶𝑓𝑦𝑞′

• 44-

Curve-based Eyelid Reconstruction

Eyelid Edge Detection & Identification Face & Eyeball Fitting [Wen et al. 2016] Input Color Input Depth Edge Maps for Training Edge Result Curve-based Eyelid Reconstruction Final Result Face & Eyeball Result Two Eyelid Linear Models

• 45-

Curve-based Eyelid Reconstruction

Minimize the inconsistency between the projected eyelid model and the real image

• 46-

Minimize the inconsistency between the projected eyelid model and the real image

Curve-based Eyelid Reconstruction

eyelid edge vertices

• 47-

Minimize the inconsistency between the projected eyelid model and the real image

Curve-based Eyelid Reconstruction

eyelid edge vertices semantic edges

• 48-

Correspondence obtaining

3D landmarks

double-fold top eyelid bottom eyelid bulge

Label 3D edge vertices on the eyelid model as 3D landmarks

• 49-

Curve Fitting

weighted least square fitting

Fit four polynomials according to the semantic edge map

• 50-

Correspondence obtaining

Obtain 2D landmarks according to relative curve length

𝑤𝑢 𝑤𝑢−1 𝑣𝑢−1 𝑣𝑢

𝑣𝑢 2D landmark 𝑤𝑢 3D landmark 𝑚 curve length

𝑡 ∗ 𝑚 𝜌 𝑤𝑢 , 𝜌 𝑤𝑢−1 = 𝑚 𝑣𝑢, 𝑣𝑢−1

𝜌 projection matrix

• 51-

Correspondence obtaining

Obtain 2D landmarks according to relative curve length

𝑤𝑢−1 𝑤𝑢 𝑡 ∗ 𝑚 𝜌 𝑤𝑢 , 𝜌 𝑤𝑢−1 = 𝑚 𝑣𝑢, 𝑣𝑢−1 projected 3D edge

• 52-

Correspondence obtaining

𝑤0

Curve length ratio 𝑡

𝑤𝑈−1 𝑡 = 𝑚 𝑣𝑈−1, 𝑣0 𝑚 𝜌 𝑤𝑈−1 , 𝜌 𝑤0 𝑣0 𝑣𝑈−1

𝑣0, 𝑣𝑈−1 end points on 2D curve 𝑤0, 𝑤𝑈−1 end points on 3D curve 𝑚 curve length 𝜌 projection matrix

𝜌 𝑤𝑈−1 𝜌 𝑤0

• 53-

Correspondence obtaining

Obtain 2D landmarks according to relative curve length

𝑤𝑢−1 𝑤𝑢

𝑣𝑢−1 𝑣𝑢 𝜌(𝑤𝑢−1) 𝜌(𝑤𝑢)

projected 3D edge 𝑡 ∗ 𝑚 𝜌 𝑤𝑢 , 𝜌 𝑤𝑢−1 = 𝑚 𝑣𝑢, 𝑣𝑢−1

• 54-

Correspondence obtaining

Obtain 2D landmarks according to relative curve length

𝑤𝑢−1 𝑤𝑢 correspondences 𝑡 ∗ 𝑚 𝜌 𝑤𝑢 , 𝜌 𝑤𝑢−1 = 𝑚 𝑣𝑢, 𝑣𝑢−1

• 55-

Eyelid Reconstruction

𝑤𝑢−1 𝑤𝑢

Minimize the distances between the projected 3D eyelid landmarks and the 2D eyelid landmarks

arg min

𝑥 𝑗=1 4 𝑢∈𝑇𝑗

𝛽𝑢

𝑗

𝜌 𝑤𝑢

𝑗 𝑥, 𝐶

− 𝑣𝑢

𝑗 2 2

projected 3D landmarks 2D landmarks

• 56-

Eyelid Reconstruction

𝑤𝑢−1 𝑤𝑢

Minimize the distances between the projected 3D eyelid landmarks and the 2D eyelid landmarks

arg min

𝑥 𝑗=1 4 𝑢∈𝑇𝑗

𝛽𝑢

𝑗

𝜌 𝑤𝑢

𝑗 𝑥, 𝐶

− 𝑣𝑢

𝑗 2 2

𝐶, 𝑥 eyelid linear rig and weights

projected 3D landmarks 2D landmarks

• 57-

Eyelid Reconstruction

𝑤𝑢−1 𝑤𝑢

Minimize the distances between the projected 3D eyelid landmarks and the 2D eyelid landmarks

arg min

𝑥 𝑗=1 4 𝑢∈𝑇𝑗

𝛽𝑢

𝑗

𝜌 𝑤𝑢

𝑗 𝑥, 𝐶

− 𝑣𝑢

𝑗 2 2

𝐶, 𝑥 eyelid linear rig and weights 𝑇𝑗 correspondence pairs of edge 𝑗

projected 3D landmarks 2D landmarks

• 58-

Eyelid Reconstruction

𝑤𝑢−1 𝑤𝑢

Minimize the distances between the projected 3D eyelid landmarks and the 2D eyelid landmarks

arg min

𝑥 𝑗=1 4 𝑢∈𝑇𝑗

𝛽𝑢

𝑗

𝜌 𝑤𝑢

𝑗 𝑥, 𝐶

− 𝑣𝑢

𝑗 2 2

𝐶, 𝑥 eyelid linear rig and weights 𝑇𝑗 correspondence pairs of edge 𝑗 𝛽𝑢

𝑗

weight of correspondence

projected 3D landmarks 2D landmarks

• 59-

Solve for the optimal weights 𝑥𝑝𝑞𝑢

Eyelid Reconstruction

arg min

𝑥 𝑗=1 4 𝑢∈𝑇𝑗

𝛽𝑢

𝑗

𝜌 𝑤𝑢

𝑗 𝑥, 𝐶

− 𝑣𝑢

𝑗 2 2

𝐶, 𝑥 eyelid linear rig and weights 𝑇𝑗 correspondence pairs of edge 𝑗 𝛽𝑢

𝑗

weight of correspondence

initial weights 𝑥0

• 60-

Eyelid Reconstruction

arg min

𝑥 𝑗=1 4 𝑢∈𝑇𝑗

𝛽𝑢

𝑗

𝜌 𝑤𝑢

𝑗 𝑥, 𝐶

− 𝑣𝑢

𝑗 2 2

𝐶, 𝑥 eyelid linear rig and weights 𝑇𝑗 correspondence pairs of edge 𝑗 𝛽𝑢

𝑗

weight of correspondence

Solve for the optimal weights 𝑥𝑝𝑞𝑢

• ptimal weights

𝑥𝑝𝑞𝑢

• 61-

Eyelid Reconstruction

Integrate into a face and eyeball fitting result [Wen et al. 2016]

• 62-

Solve for the optimal 𝑥𝑗𝑒

Shape Reconstruction

3D landmarks 2D landmarks

arg min

𝑥𝑗𝑒 𝑗=1 4 𝑢∈𝑇𝑗

𝛽𝑢

𝑗

𝜌 𝑤𝑢

𝑗 𝑥𝑗𝑒, 𝐶𝑗𝑒

− 𝑣𝑢

𝑗 2 2

𝐶𝑗𝑒 shape linear rig 𝑥𝑗𝑒 weights for 𝐶𝑗𝑒

• 63-

Solve for the optimal 𝑥𝑗𝑒

Shape Reconstruction

arg min

𝑥𝑗𝑒 𝑗=1 4 𝑢∈𝑇𝑗

𝛽𝑢

𝑗

𝜌 𝑤𝑢

𝑗 𝑥𝑗𝑒, 𝐶𝑗𝑒

− 𝑣𝑢

𝑗 2 2

𝐶𝑗𝑒 shape linear rig projected 3D landmarks 2D landmarks 𝑥𝑗𝑒 weights for 𝐶𝑗𝑒 3D landmarks

• 64-

Shape Reconstruction

arg min

𝑥𝑗𝑒 𝑗=1 4 𝑢∈𝑇𝑗

𝛽𝑢

𝑗

𝜌 𝑤𝑢

𝑗 𝑥𝑗𝑒, 𝐶𝑗𝑒

− 𝑣𝑢

𝑗 2 2

𝐶𝑗𝑒 shape linear rig 𝑥𝑗𝑒 weights for 𝐶𝑗𝑒

Solve for the optimal 𝑥𝑗𝑒

• 65-

Pose Reconstruction

Solve for the optimal 𝑥𝑓𝑦𝑞

2D landmarks 3D landmarks

arg min

𝑥𝑓𝑦𝑞 𝑗=1 4 𝑢∈𝑇𝑗

𝛽𝑢

𝑗

𝜌 𝑤𝑢

𝑗 𝑥𝑓𝑦𝑞, 𝐶𝑓𝑦𝑞′

− 𝑣𝑢

𝑗 2 2

𝐶𝑓𝑦𝑞′ user-specific pose linear rig 𝑥𝑓𝑦𝑞 weights for 𝐶𝑓𝑦𝑞′

• 66-

Pose Reconstruction

arg min

𝑥𝑓𝑦𝑞 𝑗=1 4 𝑢∈𝑇𝑗

𝛽𝑢

𝑗

𝜌 𝑤𝑢

𝑗 𝑥𝑓𝑦𝑞, 𝐶𝑓𝑦𝑞′

− 𝑣𝑢

𝑗 2 2

𝐶𝑓𝑦𝑞′ user-specific pose linear rig 𝑥𝑓𝑦𝑞 weights for 𝐶𝑓𝑦𝑞′

Solve for the optimal 𝑥𝑓𝑦𝑞

projected 3D landmarks 2D landmarks 3D landmarks

• 67-

Pose Reconstruction

arg min

𝑥𝑓𝑦𝑞 𝑗=1 4 𝑢∈𝑇𝑗

𝛽𝑢

𝑗

𝜌 𝑤𝑢

𝑗 𝑥𝑓𝑦𝑞, 𝐶𝑓𝑦𝑞′

− 𝑣𝑢

𝑗 2 2

𝐶𝑓𝑦𝑞′ user-specific pose linear rig 𝑥𝑓𝑦𝑞 weights for 𝐶𝑓𝑦𝑞′

Solve for the optimal 𝑥𝑓𝑦𝑞

• 68-

Performance

GPU thread 2 GPU thread 1

* Test Environment: 3.6GHz eight-core CPU, 16G RAM, NVIDIA Geforce GTX 980

• 69-

Performance

30ms GPU thread 2 GPU thread 1 Face and eyeball fitting

* Test Environment: 3.6GHz eight-core CPU, 16G RAM, NVIDIA Geforce GTX 980

• 70-

Performance

30ms 11ms GPU thread 2 GPU thread 1 Face and eyeball fitting Edge detection, identification and curve fitting

* Test Environment: 3.6GHz eight-core CPU, 16G RAM, NVIDIA Geforce GTX 980

• 71-

Performance

30ms 11ms 7ms GPU thread 2 GPU thread 1 Face and eyeball fitting Edge detection, identification and curve fitting Eyelid correspondence update and energy minimization

* Test Environment: 3.6GHz eight-core CPU, 16G RAM, NVIDIA Geforce GTX 980

• 72-

Performance

GPU thread 2 GPU thread 1 Total: 37ms

* Test Environment: 3.6GHz eight-core CPU, 16G RAM, NVIDIA Geforce GTX 980

• 73-

Comparisons

• 74-

Live Results

• 75-

Results of Internet Images

• 76-

Limitations

Eyelid tracking in challenging lighting conditions Depth requirement for the tracking system More shape variations and wrinkle details

• 77-

Conclusion

Real-time eyelid reconstruction

Eyelid linear models Eyelid edge detection & identification

DNN

Curve-based Eyelid reconstruction

• 78-

Thank you

http://feng-xu.com/projects/Realtime3DEyelids/ (Training set and eyelid model are available now)