Face Alignment in the Wild Mirrorability and Sensitivity Heng Yang - - PowerPoint PPT Presentation

Face Alignment in the Wild

Mirrorability and Sensitivity Rainbow Group, Computer Laboratory Heng Yang yanghengnudt@gmail.com

Outline

• Introduction
• Face alignment mirrorability
• Face alignment sensitivity
• Conclusion and discussion

Introduction

Woodrow Wilson "Woody" Bledsoe (Nov.12, 1921 – Oct. 4, 1995) During 1964 and 1965, Bledsoe, along with Helen Chan and Charles Bisson, worked on using the computer to recognize human faces Man-Machine System Users are asked to localize a set of facial landmarks of the photograph

“This recognition problem is made difficult by the great variability in head rotation and tilt, lighting intensity and angle, facial expression, aging, etc.” —Woody Bledsoe, 1966

Automatic Facial Landmarks Localization

Introduction

Detection Alignment Recognition

Name: Kate Emotion: Happy Gender: Female Attractiveness: 5 Race: Caucasian Age: ~30 Occupation: ? ….

Introduction

[Yaniv Taigman et al. DeepFace: Closing the Gap to Human-Level Performance in Face Verification,CVPR2014] [Tal Hassner et al. Effective Face Frontalization in Unconstrained Images, CVPR2015]

• Alignment is an essential step to the success of face recognition.

Introduction

1 2 3 4 5 6 7 8

1 2 3 4 5 6 7 8 9 1011 12 13 14 1 2 3 4 5 6 7 8 9 10 1 2 3 4 5 6 7 8

1 2 3 4 5 6 7 8 9 10 11 12 13 14

View1 View2 View3 View4 View5

• Conceptually it is very similar to many other problems

[Yang and Patras TIP’14] [Yang, Zhang and Robinson Arxiv1509.04954] [Liu and Belhumeur, ICCV’13] [Yang and Ramanan cvpr’11]

Introduction

https://sites.google.com/site/yanghengcv/face-alignment

• AAM [T. Cootes et al. 98]
• CLM [T. Cootes et al. 06]

Academia

Industry Sensetime, Face++, Linkface, Orbeus, CloudWalk，BUPTAPI, 腾讯优图，汉王人脸通，一登… FaceMark, eyedea，sightcorp, ci2cv, Kairos, betaface, Luxand FaceSDK, Cognitec, ayonix, Omron，Affectiva…

Introduction

CONS PROS ✔ ✔

It is easy to augment training samples It is very efficient in both training and testing ✔ It is simple and more accurate ✔ # of landmarks has little impact on running speed, with a vector representation It is very sensitive to initialization

✖

It is an open system and usually requires additional model to check the reliability

✖

• Cascaded face alignment framework

Face Alignment Mirrorability

[Heng Yang and Ioannis Patras CVPR 15.]

• Mirrorability: the ability of a model/algorithm to preserve the mirror

symmetry when applied on an image and its mirror image.

• NB. NOT USING

SYMMETRY CONSTRAINS Why only mirror? It is easy to avoid training data bias.

Face Alignment Mirrorability

[Heng Yang and Ioannis Patras CVPR 15.]

• Mirror error:

Big mirror error Small mirror error

Face Alignment Mirrorability

• Property #1: Mirror error is calculated in a blind way—no ground truth is needed
• Property #2 :Mirror error shows high correlation with the ground truth alignment error

[X. P. Burgos-Artizzu, P. Perona and P. Dollár, "Robust face landmark estimation under occlusion ", ICCV2013.]

Face Alignment Mirrorability

• Mirror error is calculated in a blind way—no ground truth is needed
• Mirror error shows high correlation with the ground truth alignment error

Face alignment by random ferns

Face alignment by random trees

[X. P. Burgos-Artizzu, P. Perona and P. Dollár, "Robust face landmark estimation under occlusion ", ICCV2013.] [Kazemi, Vahid and Sullivan, Josephine: "One Millisecond Face Alignment with an Ensemble of Regression Trees",CVPR,2014.]

Face Alignment Mirrorability

• Similar observation is found in human pose estimation

[F. Wang and Y. Li. Beyond physical connections: Tree models in human pose estimation. CVPR, 2013]

Face Alignment Mirrorability

• How does it happen in regression forests based method?

x1 x2 x1-x2 > t ?

Face Alignment Mirrorability

• How does it happen in regression forests based method?

Face Alignment Mirrorability

[Karel Lenc, Andrea Vedaldi, Understanding image representations by measuring their equivariance and equivalence ", CVPR 2015.]

• How does it happen in Convolutional Neural Netork method?

Face Alignment Mirrorability

• Application #1: difficult samples selection

Compared to samples selected with ground truth. Compared to samples selected w/o ground truth.

• 1. The samples that we have selected are truly ‘difficult’
• 2. Selected difficult samples show high consistency across different approaches.

Face Alignment Mirrorability

Feedback (Mirror error?) “open” system

Face Alignment Mirrorability

• Application #2: re-start cascaded face alignment

[Kazemi, Vahid and Sullivan, Josephine: "One Millisecond Face Alignment with an Ensemble of Regression Trees",CVPR,2014.] [X. P. Burgos-Artizzu, P. Perona and P. Dollár, "Robust face landmark estimation under occlusion ", ICCV2013.]

Good Warning Bad

Face Alignment Mirrorability

• Application examples

Face Alignment Mirrorability

• Mirrorability in other computer vision problems

104.1 37.03 11.11 3.854 110.1 46.26

http://vision.ucsd.edu/~pdollar/toolbox/doc/index.html https://how-old.net/ http://places.csail.mit.edu/demo.html

Face Alignment Sensitivity

• Motivation #1: face detection is not always consistent in consecutive frames.
• Motivation #2: face detection might vary from training to testing

GT HeatHunter Viola&J

• nes

IBUG HOG+SVM

Face Alignment Sensitivity

• A new evaluation metric over a testing set:

Face Alignment Sensitivity

• Empirical study #1: synthesised face center shifts

Viola&J

• nes

Shift -0.10 Shift -0.21

N.B.: I use the off-the-shelf models, thus it is not very meaningful to assess the relative merits.

Face Alignment Sensitivity

• Empirical study #2: real face center shifts

Face Alignment Sensitivity

• Empirical study #3: synthesized face scale changes

Conclusion and Discussion

ü We have proposed a new concept, mirrorability and a corresponding metric, mirror error to measure it. ü We show an important property that mirror error is highly correlated to alignment error, which can be used as a feedback for cascaded face alignment. ü We show that most recent face alignment methods are sensitive to initialisation center shifts, (less sensitive w.r.t scale variation).

Future work

• How about face alignment in videos?

ICCV2015 300W Videos Challenge: http://ibug.doc.ic.ac.uk/resources/300-VW/

• How accurate is it desired for a specific task in face analysis?
• Is mirrorability or equivariance worthy more attentions in computer vision?
• Further improvement in face alignment, data vs. model?

Conclusion and Discussion

[Yang, Zhang and Robinson Arxiv1509.04954] [Yang et al. BMVC2015]

• Further improvement in face alignment, data vs. model?

Reference

• Heng Yang, Ioannis Patras, "Mirror, mirror on the wall, tell me, is the error small? ", CVPR 2015.
• Heng Yang, Ioannis Patras, "Fine-tuning regression forests votes for object alignment in the wild", IEEE Trans. Image Processing, Dec.,

2014.

• Heng Yang et al. "Face alignment assisted by head pose estimation", BMVC 2015.
• SDM : Xiong, X., & De la Torre, F. (2013, June). Supervised descent method and its applications to face alignment. CVPR, 2013.
• GNDPM: Tzimiropoulos, G., & Pantic, M. (2014, June). Gauss-newton deformable part models for face alignment in-the-wild, CVPR,

2014.

• CFAN: Zhang, J., Shan, S., Kan, M., & Chen, X. (2014). Coarse-to-fine auto-encoder networks (CFAN) for real-time face alignment.

ECCV 2014, Springer.

• IFA: Asthana, S. Zafeiriou, S. Cheng, M. Pantic, Incremental Face Alignment in the Wild, CVPR 2014.
• CCNF: Baltrušaitis, Tadas and Robinson, Peter and Morency, Louis-Philippe: "Continuous Conditional Neural Fields for Structured

Regression", ECCV, 2014.

• TCDCN: Zhang, Zhanpeng and Luo, Ping and Loy, Chen Change and Tang, Xiaoou: "Facial Landmark Detection by Deep Multi-task

Learning",ECCV,2014

• TREE:Kazemi, Vahid and Sullivan, Josephine: "One Millisecond Face Alignment with an Ensemble of Regression Trees",CVPR,2014.
• LBF: Ren, Shaoqing and Cao, Xudong and Wei, Yichen and Sun, Jian: "Face Alignment at 3000 FPS via Regressing Local Binary

Features", CVPR,2014.

• GNDPM: Tzimiropoulos, Georgios and Pantic, Maja: "Gauss-newton deformable part models for face alignment in-the-wild", CVPR,2014.
• IFA: Asthana, Akshay and Zafeiriou, Stefanos and Cheng, Shiyang and Pantic, Maja: "Incremental Face Alignment in the Wild", CVPR,

2014.

• Sagonas, Christos and Tzimiropoulos, Georgios and Zafeiriou, Stefanos and Pantic, Maja: "300 Faces in-the-Wild Challenge: The first

facial landmark localization Challenge", ICCV,2013.

• Burgos-Artizzu, Xavier P and Perona, Pietro and Dollar, Piotr: "Robust face landmark estimation under occlusion", ICCV,2013.
• Y. Yang and D. Ramanan. Articulated human detection with flexible mixtures of parts. T-PAMI, 2013.
• F. Wang and Y. Li. Beyond physical connections: Tree models in human pose estimation. CVPR, 2013

• Online demo: http://www.xuanzhixuan.com/
• Homepage: https://sites.google.com/site/yanghengcv/home
• Email: yanghengnudt@gmail.com

Peter Robinson @ Cambridge Ioannis Patras @ QMUL Shaogang Gong @ QMUL Xuming He @ NICTA/ANU

Thanks