An Illumination Invariant Face Recognition System for Access Control using Video

1. An Illumination Invariant Face Recognition System for Access Control using Video Ognjen Arandjelović Roberto Cipolla Funded by Toshiba Corp. and Trinity College, Cambridge

2. Eigenfaces 3D Morphable Models Wavelet methods Face Recognition • Single-shot recognition – a popular area of research since 1970s • Many methods have been developed • Bad performance in presence of: • Illumination variation • Pose variation • Facial expression • Occlusions (glasses, hair etc.)

3. Recognition setup Training stream Novel stream Face Recognition from Video • Face motion helps resolve ambiguities of single shot recognition – implicit 3D • Video information often available (surveillance, authentication etc.)

4. Facial features Face pattern manifold Face region Face Manifolds • Face patterns describe manifolds which are: • Highly nonlinear, and • Noisy, but • Smooth

5. ? Limitations of Previous Work • In this work we address 3 fundamental questions: • How to model nonlinear manifolds of face motion • How to achieve illumination and pose robustness • How to choose the distance measure

6. Unchanging identity, changing illumination Changing identity, unchanging illumination Face Motion Manifolds: Revisited • Motivation: How can we use the prior knowledge on the shape of the manifolds?

7. Pose Clusters • Face motion manifolds are nonlinear, but: • Low-dimensional (c.f. registration for the reduction of the dimensionality), and • Key observation: can be described well using only 3 linear pose clusters Colour-coded pose clusters for 3 manifolds

8. Determining Pose Clusters • Pose clusters are semantic clusters: • K-means and similar algorithms are unsuitable • We are using a simple method based on the motion parallax • Membership decided based on Maximum Likelihood Yaw measure Pupils Distribution for 3 clusters Discrepancy η Image plane

9. Pose Clusters: Example Input manifold and colour-coded pose clusters Sample frames from the 3 pose clusters

10. Illumination compensation • Performed in two stages: • Coarse illumination compensation (exploiting face smoothness) • Fine illumination compensation (exploiting low dimensionality of the face illumination subspace) Input Output

11. Face regions 1 2 3 4 Region-based GIC Gamma Intensity Correction (GIC) Solved by 1D non-linear optimization Canonical image • Region-based GIC (RGIC): faces are (roughly) divided into regions with smoothly varying surface normal Varying Gamma

12. γvalue map Smoothed γmap Mean face Artefacts removed Input face RGIC face Our method Boundary artefacts Region-based GIC: Artefacts • Region-based GIC suffers from artefacts at region boundaries

13. Illumination Subspace • Each input frame corrected for a linear Pose Illumination Subspace component to match the reference distribution of the same pose • Illumination subspace is high-dimensional • Constrained to expected variations by Mahalanobis distance Illumination Subspace Input manifold Reference manifold

14. Illumination Compensation Results Strong side lighting Original/input frames Illumination-corrected frames Reference frames And in face pattern space…

15. Comparing Pose Clusters Reference cluster Reduced spread Novel cluster Cluster centres • “Distribution-based” distances (Kullback-Leibler divergence, Resistor Average Distance etc.) unsuitable • We use the simple Euclidean distance between cluster centres

16. Unified Manifold Similarity • Recognition based based on the likelihood ratio: Manifolds belong to the same person Distances between pose clusters • Learn likelihoods from ground truth training data Likelihood histogram RBF-interpolated likelihood Two-pose interpolated likelihood Likelihood now monotonically decreasing Undefined value regions

17. Face Video Database Revisited • Testing performed under extreme, varying illuminations 10 illumination conditions used (random 5 for training, others for testing)

18. Translation manifold Skew manifold Rotation manifold Registration • Linear operations on images are highly nonlinear in the pattern space • Translation/rotation and weak perspective can be easily corrected for directly from point correspondences • We use the locations of pupils and nostrils to robustly estimate the optimal affine registration parameters

19. Detect features Crop & affine register faces Registration Method Used • Feature localization based on the combination of shape and pattern matching (Fukui et al. 1998)

20. Results • Very high recognition rates attainted (96% average) under extreme variations in illumination • Other methods showed little to no illumination invariance

21. Results, continued • The method was shown to give promising results for authentication uses: • Good separability of inter- and intra- class manifold distances was found • It can provide a secure system with only 0.1% false positive rate and 8% false negative rate Cumulative distributions of inter- and intra- class manifold distances The ROC curve for the proposed method

22. Future Research • Non-constant illumination within a single sequence causes problems • Illumination compensation is still not perfect – pose illumination subspaces have unnecessarily high dimensions • Pose estimation is too primitive – outliers cause problems in estimation of linear subspaces • Complete pose invariance is still not achieved (what if there are no corresponding pose clusters?) For suggestions, questions etc. please contact me at:oa214@cam.ac.uk