1 / 20

A face analysis exemplar : Face detection, landmarking and facial expression recognition.

A face analysis exemplar : Face detection, landmarking and facial expression recognition. Dr . Brais Martinez. Slides can be downloaded from braismartinez.com. Overview. Model-free part-based tracking. Part-based facial landmarking. Face Analysis. PostDoc. PhD. End 2010.

suzuki
Télécharger la présentation

A face analysis exemplar : Face detection, landmarking and facial expression recognition.

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. A face analysis exemplar: Face detection, landmarking and facial expression recognition. Dr.Brais Martinez Slides can be downloaded from braismartinez.com

  2. Overview Model-free part-based tracking Part-based facial landmarking Face Analysis PostDoc PhD End 2010 • Research visits to: • Imperial College London (MajaPantic) • 9/2007-3/2008 • Oregon State University (SinisaTodorovic) • 7/2013-10/2013

  3. Overview Multi-view face detection Facial Landmarking Facial Action Unit Detection [Under Review] IVC. J. Orozco, B. Martinez, M. Pantic, “Empirical analysis of cascade deformable models for multi-view face detection” [IF 2012: 1.96, Q1] 2010 CVPR - M. Valstar, B. Martinez, X. Binefa, M. Pantic, “Facial point detection using boosted regression and graph models” 2013 TPAMI - B. Martinez, M. Valstar, X. Binefa, M. Pantic, “Local evidence aggregation in regression-based facial point detection” [Under Review] CVIU - B. Martinez, M. Pantic, “Facial landmarkingfor in-the-wild images with local inference based on global appearance” 2014 TSMCB - B. Jiang, M. Valstar, B. Martinez, M. Pantic, “A dynamic appearance descriptor approach to facial actions temporal modelling” [Under Review] IJCV - B. Jiang, B. Martinez, M. Valstar, M. Pantic, “Automatic analysis of facial actions: A survey” [Under Review] ICPR - B. Jiang, B. Martinez, M. Valstar, M. Pantic, “Decision level fusion of domain specific regions for facial action recognition” 2014TSMCB - B. Jiang, M. Valstar, B. Martinez, M. Pantic, “A dynamic appearance descriptor approach to facial actions temporal modelling” [Under Review] IJCV - B. Jiang, B. Martinez, M. Valstar, M. Pantic, “Automatic Analysis of Facial Actions: A Survey” [Under Review] ICPR - B. Jiang, B. Martinez, M. Valstar, M. Pantic, “Decision Level Fusion of Domain Specific Regions for Facial Action Recognition” 2010CVPR - M. Valstar, B. Martinez, X. Binefa, M. Pantic, “Facial Point Detection using Boosted Regression and Graph Models” 2013TPAMI - B. Martinez, M. Valstar, X. Binefa, M. Pantic, “Local Evidence Aggregation in Regression-based Facial Point Detection” [Under Review] CVIU - B. Martinez, M. Pantic, “Facial landmarking for in-the-wild images with local inference based on global appearance”

  4. Face detection using cascaded DPM Part-based model • The Deformable Parts Model (DPM): • Object composed of parts • Current state-of-the-art model in object detection • Weakly-supervised • Uses Linear SVM (we used 35k+ training images!) • Very efficient implementations (both training and testing) Convolve filter with gradient im. object loc. parts loc. Penalise deformations

  5. Cascaded DPM Non-frontal poses: Mixture model Speed: cascaded search Full score Root Filter Part Filters Part Locations Score 1 part > No Score 2 parts Scale: Multi-scale sliding window > No

  6. Results: DPM face detection Dataset: AFLW True Positive Rate Proposed Zhu&Ramanan Multiview V&J False Positive Rate • Advantages over Zhu & Ramanan: • Only face bound annotations needed • Better for lower resolution • 5 parts instead of 66 • Cascade detection

  7. Overview Multi-view face detection Facial Landmarking Facial Action Unit Detection [Under Review] IVC. J. Orozco, B. Martinez, M. Pantic, “Empirical Analysis of Cascade Deformable Models for Multi-view Face Detection” 2010CVPR - M. Valstar, B. Martinez, X. Binefa, M. Pantic, “Facial Point Detection using Boosted Regression and Graph Models” [81 citations] 2013TPAMI - B. Martinez, M. Valstar, X. Binefa, M. Pantic, “Local Evidence Aggregation in Regression-based Facial Point Detection” [IF 2012: 4.80, Q1] [Under Review] CVIU - B. Martinez, M. Pantic, “Facial landmarking for in-the-wild images with local inference based on global appearance” [IF 2012: 1.23, Q3] 2014TSMCB - B. Jiang, M. Valstar, B. Martinez, M. Pantic, “A dynamic appearance descriptor approach to facial actions temporal modelling” [Under Review] IJCV - B. Jiang, B. Martinez, M. Valstar, M. Pantic, “Automatic Analysis of Facial Actions: A Survey” [Under Review] ICPR - B. Jiang, B. Martinez, M. Valstar, M. Pantic, “Decision Level Fusion of Domain Specific Regions for Facial Action Recognition” 2014TSMCB - B. Jiang, M. Valstar, B. Martinez, M. Pantic, “A dynamic appearance descriptor approach to facial actions temporal modelling” [Under Review] IJCV - B. Jiang, B. Martinez, M. Valstar, M. Pantic, “Automatic Analysis of Facial Actions: A Survey” [Under Review] ICPR - B. Jiang, B. Martinez, M. Valstar, M. Pantic, “Decision Level Fusion of Domain Specific Regions for Facial Action Recognition” 2010CVPR - M. Valstar, B. Martinez, X. Binefa, M. Pantic, “Facial Point Detection using Boosted Regression and Graph Models” 2013TPAMI - B. Martinez, M. Valstar, X. Binefa, M. Pantic, “Local Evidence Aggregation in Regression-based Facial Point Detection” [Under Review] CVIU - B. Martinez, M. Pantic, “Facial landmarking for in-the-wild images with local inference based on global appearance”

  8. Part-based facial landmarking Classical part-based: Train: 1 classifier per point (e.g. logistic classifier) Construct response maps Test: Construct response map (sliding window over ROI) Maximise response constrained to feasible shape (constrained gradient ascent ) Do regression! CVPR 2010 – Facial Point Detection using Boosted Regression and Graph Models Constrained gradient ascent

  9. Regression for Localisation Current estimate BoRMaN algorithm Face Detection Δx Regression: Δy Obtain prior location (starting point) HOG Eval. regressors (new location hypotheses) = Ground truth Correct hypothesis (shape restrictions) Output • MRF-based shape model • Detect bad estimations • Propose an alternative Multiple Regression Methodologies: Least Squares, SVR, GP, random forests…

  10. Shape model Relations are rotation and scaleindependent Angle α between segments Ratio ρ between segment lengths

  11. Regression-basedlandmarking Major improvements: Established a trend: • Prediction accumulation/voting • Facial landmarking using regression: • 2010: • CVPR • 2012: • CVPR (Microsoft Res.) • CVPR (ETH, Van Gool) • ECCV (Manchester Univ.– Cootes) • 2013: • TPAMI (iBug) • CVPR (CMU) • CVPR (iBug) • ICCV (Microsoft Res.) • ICCV (QMUL) 2013 TPAMI – Martinez, Valstar, Binefa, Pantic Cascaded regression Best performing nowadays! …

  12. Regression: Vote aggregation What if we are too far from the target? What if we have bad predictions? Errors Uniformly distributed do NOT accumulate Errors Gaussiandistributed DO accumulate • Base of the algorithm: • Accumulate predictions, a prediction being a small Gaussian

  13. LEAR algorithm

  14. Overview Multi-view face detection Facial Landmarking Facial Action Unit Detection [Under Review] IVC. J. Orozco, B. Martinez, M. Pantic, “Empirical Analysis of Cascade Deformable Models for Multi-view Face Detection” 2010CVPR - M. Valstar, B. Martinez, X. Binefa, M. Pantic, “Facial Point Detection using Boosted Regression and Graph Models” 2013TPAMI - B. Martinez, M. Valstar, X. Binefa, M. Pantic, “Local Evidence Aggregation in Regression-based Facial Point Detection” [Under Review] CVIU - B. Martinez, M. Pantic, “Facial landmarking for in-the-wild images with local inference based on global appearance” 2014TSMCB - B. Jiang, M. Valstar, B. Martinez, M. Pantic, “A dynamic appearance descriptor approach to facial actions temporal modelling” [IF 2012: 3.24, Q1] [Under Review] IJCV - B. Jiang, B. Martinez, M. Valstar, M. Pantic, “Automatic Analysis of Facial Actions: A Survey” [IF 2012: 3.62, Q1] [Under Review] ICPR - B. Jiang, B. Martinez, M. Valstar, M. Pantic, “Decision Level Fusion of Domain Specific Regions for Facial Action Recognition” 2014TSMCB - B. Jiang, M. Valstar, B. Martinez, M. Pantic, “A dynamic appearance descriptor approach to facial actions temporal modelling” [Under Review] IJCV - B. Jiang, B. Martinez, M. Valstar, M. Pantic, “Automatic Analysis of Facial Actions: A Survey” [Under Review] ICPR - B. Jiang, B. Martinez, M. Valstar, M. Pantic, “Decision Level Fusion of Domain Specific Regions for Facial Action Recognition” 2010CVPR - M. Valstar, B. Martinez, X. Binefa, M. Pantic, “Facial Point Detection using Boosted Regression and Graph Models” 2013TPAMI - B. Martinez, M. Valstar, X. Binefa, M. Pantic, “Local Evidence Aggregation in Regression-based Facial Point Detection” [Under Review] CVIU - B. Martinez, M. Pantic, “Facial landmarking for in-the-wild images with local inference based on global appearance”

  15. Action Unit detection – what is it about? Facial expression recognition Message judgment: Directly decode the meaning of the expression Happiness? • 6 universal expressions: happiness, anger, sadness, fear, surprise, disgust • (constant message to sign relation) • Pre-segmented episodes Pain? • Sign judgment: • Study the physical signals composing the expression • An AU relates to the activation of a facial muscle • “Agnostic” (not concern about “knowing” the message) • Can represent any expression • Reasoning upon needed to understand • Frame-based labelling Facial Action Coding System is the most common sign judgment approach.

  16. Action Unit analysis: what and why • Research problems within the field: • AU detection (per-frame) • AU intensity estimation • AU temporal segment detection • AU correlations (for structured prediction) • Semantics of AUs • What do they allow (that normal facial expression analysis does not): • Pain detection • Deceit detection • Detection of social signals (conflict, agreement/disagreement,…)

  17. How Action Unit detection is done Pre-processing Feature extraction Machine Analysis SVM, ANN, Boosting… Appearance Face detection Facial landmark detection Registration Dynamic T Non-ref. affine Trans. Graph models (label consistency) Geometric [Under Review] IJCV - Jiang, Martinez, Valstar, Pantic, “Automatic Analysis of Facial Actions: A Survey”

  18. TOP features Markov Model over temporal segments Three orthogonal planes (TOP): Extension to spatio-temporal volumes of histogram features Neut Onset Representing the face Allows: analysis of AU temporal segments Apex Offset 2014 TSMC-B - “A dynamic appearance descriptor approach to facial actions temporal modelling”

  19. Publications [Under Review] B. Jiang, B. Martinez, M. Valstar, M. Pantic, “Automatic Analysis of Facial Actions: A Survey”. International Journal of Computer Vision [IF 2012: 3.62, Q1] J. Orozco, B. Martinez, M. Pantic, “Empirical Analysis of Cascade Deformable Models for Multi-view Face Detection”, Image and Vision Computing [IF 2012: 1.96, Q1] B. Martinez, M. Pantic, “Facial landmarking for in-the-wild images with local inference based on global appearance”, Computer Vision and Image Understanding [IF 2012: 1.23, Q3] B. Jiang, B. Martinez, M. Valstar, M. Pantic, “Decision Level Fusion of Domain Specific Regions for Facial Action Recognition”, Int. Conf. on Pattern Recognition, 2014 [Journals] 2014 B. Jiang, M. Valstar, B. Martinez, M. Pantic, “A dynamic appearance descriptor approach to facial actions temporal modelling”, In IEEE Tans. on System Man and Cybernetics – Part B [IF 2012: 3.24, Q1] 2013 B. Martinez, M. Valstar, X. Binefa, M. Pantic, “Local Evidence Aggregation in Regression-based Facial Point Detection”, In IEEE Trans. on Pattern Analysis and Machine Intelligence [IF 2012: 4.80, Q1] 2013 S. Petridis, B. Martinez, M. Pantic, “The MAHNOB Laughter Database”, In Image and Vision Computing Journal[IF 2012: 1.96, Q1] 2011 M. Vivet, B. Martinez and X. Binefa, “DLIG: Direct Local Indirect Global Alignment for Video Mosaicing”, In IEEE Trans. on Circuits and Systems for Video Technology [IF 1.65, Q2] 2008 B. Martinez, X. Binefa, “Piecewise affine kernel tracking for non-planar targets”, In Pattern Recognition [IF: 3.28, Q1] [Conferences] 2010 M. Valstar, B. Martinez, X. Binefa, M. Pantic, “Facial Point Detection using Boosted Regression and Graph Models”, In IEEE Int’l Conf. on Computer Vision and Pattern Recognition [27% acceptance rate, 81 citations] 2010 B. Martinez, X. Binefa, M. Pantic, “Facial Component Detection in Thermal Imagery”, In IEEE Int'l Conf. Computer Vision and Pattern Recognition - Workshops

  20. Thanks!

More Related