Context Constrained Facial Landmark Localization Based on Discontinuous Haar-like Features

1. Context Constrained Facial Landmark Localization Based on Discontinuous Haar-like Features Xiaowei Zhao*, Xiujuan Chai*, Zhiheng Niu**, Cherkeng Heng**,Shiguang Shan* *Institute of Computing Technology, CAS **Panasonic Singapore Laboratories Pte Ltd (PSL)

2. Outline • Background • Motivation • Our Method • Experiments • Summary

3. Task • Localize key facial landmarks • e.g. corners of the eyes, corners ofthe mouth Localization of key facial landmarks

4. Previous Works • Rule-based methods • Skin color-based • R.Hsu et al. 2002 • Integral projection • Z.Zhou et al. 2004

5. Previous Works • Rule-based methods • Deformable template • Fitting some pre-defined geometric shapes to the input image • A.Yuille et al. 1992

6. Previous Works • Rule-based methods • Deformable template • Statistical models • Fitting some Statistical shape/appearance models • Active Shape Models (ASM) • T.Cootes et at. 1995 • Active Appearance Models (AAM) • T.Cootes et al. 1998 • Constrained Local Model • D.Cristinacce et al. 2006 • Many variations…

7. Previous Works Rule-based methods Deformable template Statistical models AdaBoost-based methods Almost the same as face detection AdaBoost + Haar-like

8. Previous Works • Rule-based methods • Deformable template • Statistical models • AdaBoost-based methods • Context-constrained methods • The positions of one landmark can be “determined” from other landmarks. • T.Kozakaya et al. 2010 • M.Valstar et al. 2010

9. Previous Works • Boosted Regression and Graph Models [M.Valstar et al. CVPR 2010] • Explore the relationship between a patch L and the target location T. • SVR is used to estimate the direction of the target (left panel) and the distance to the target (right panel) It is good, but might be a little complicated.

10. Motivation • Problem • How to exploit the context in an easy way, better not to change the AdaBoost framework? • A new type of feature?

11. Outline • Background • Motivation • Our Method • Experiments • Summary

12. Basic Idea • Traditionally, we do like this… Target point

13. Our method • Discontinuous Haar-like Features • Mode #1: subtraction of two rectangles Mode #1

14. Our method • Discontinuous Haar-like Features • Mode #2: subtraction of three rectangles Mode #2

15. Our method • Discontinuous Haar-like Features • Mode #2: subtraction of four rectangles Mode #3

16. Discontinuous Haar-like Features • Principles Behind • Context is modeled naturally in the form of feature co-occurrence

17. Some Implementation Details

18. Some Implementation Details • Exploiting both features • Traditional Haar-like features • Within the candidate window • Proposed discontinuous Haar-like features • Exploiting the context

19. Some Implementation Details • Feature Styles Selection • Mode #2 and #3 are too complicated to control (numerous candidate features)

20. Some Implementation Details • Even for Mode #1, numerous features • Reduce the number of candidates • Blocks are restricted to be square • The size of the square is fixed • Larger sampling steps Sampling step=2 Sampling step=2

21. Some Implementation Details • No need to change the Boosting framework • Automatically select the features/weak classifiers (Look-Up Table by B.Wu et al. 2004) • Feature heritage is used to speed up detection. • Use less different features

22. Outline • Background • Motivation • Our Method • Experiments • Summary

23. Training and Testing Dataset Training datasets CAS-PEAL, PIE, FGRC v1, FG-Aging Totally about 7000 near-frontal images with labels Testing datasets: BioID, Cohn-Kanade FE database

24. Are the New Feature Valid? Discontinuous Haar-like features • The top 4 features selected by using Real AdaBoost (for right mouth corner) 1 2 3 4

25. Are the New Feature Valid? About 1/3 features selected by Boosting are discontinuous Haar-like features! Traditional Haar-like features Discontinuous Haar-like features Percentage of two kinds of features

26. Are the New Feature Valid? • When both features are used as candidates, fewer total features are used (with similar accuracy)! • So, new feature is more discriminative. Using only traditional Haar-like features as candidates Using both feature as candidates Comparison of selected feature number

27. How Block Size affects? In case of 105x105 face size Square size = 10 x 10 works better

28. Comparison with only Haar-like Features Test Results on BioID database Outer corner of left eye Outer corner of right eye

29. Comparison with only Haar-like Features Test Results on BioID database Inner corner of left eye Inner corner of right eye

30. Comparison with only Haar-like Features Test Results on BioID database Left mouth corner Right mouth corner

31. Comparison with Other Methods • Comparison with other methods on BioID *S. Milborrow and F. Nicolls, “Locating facial features with an extended active shape model,” In Proc. ECCV, pp. 504–513, 2008. **K. Kinoshita, Y. Konishi, S. Lao, and M. Kawade, “A fast and robust facial feature detection and 3D head pose estimation based on 3D model fitting,” In Proc. MIRU, 2008.

32. Comparison with Other Methods • Comparison with other methods on Cohn-kanade database *D. Vukadinovic and M. Pantic, “Fully automatic facial feature point detection using gabor feature based boosted classifiers,” In Proc.Systems, Man and Cybernetics, vol. 2, pp. 1692–1698, 2005.. ***S. Milborrow and F. Nicolls, “Locating facial features with an extended active shape model,” In Proc. ECCV, pp. 504–513, 2008.

33. Summarize Our contribution To model context, discontinuous Haar-like features are designed. Its effectiveness is Preliminarily validated. Pros Simple Context leads to robustness to variations No need to change the Boosting part Cons Not effective for occlusion Large pool of candidate features

34. Future Work • Explore Mode #2 & #3 features • More context in each feature • Generalize to other object detection • e.g. face detection (using hair, shoulder as context) • Locate multiple facial landmarks Simultaneously • Co-training • How to obtain occlusion-robustness?

35. Thanks!

36. Boosted Regression & Graph Models SVR regression (0,0) • Regression Prediction • Learning the mapping between the appearance of the area surrounding a point and the positions of these points • Support vector regression is adopted • Haar-like feature is adopted to describe appearance • AdaBoost-based feature selection to reduce the dimension of appearance features • Spatial Relations • Preventing unfeasible facial points combinations • Difference with our method • Only context/feature around facial point is used • MRF is used to model the relationship among facial landmarks MRF points model