1. 2011-2012 MITRE Computer Science ClinicLandmarking and Pose Correction for Face Recognition Background Pose correction pipeline Image warping MITRE Corporation is a federally-funded research-and-development corporation that has developed their own facial recognition system, known as MITRE Matcher. Non-frontal facial images create a significant challenge to the recognition process for both MITRE Matcher and other facial recognition systems, even if the degree of the pose variation is as small as ten or twenty degrees. This project's goal was to research, implement, and evaluate facial-landmarking algorithms and approaches to pose- analysis and pose-correction. Pose-corrected image We use thin-plate splines to smoothly warp from one face set of landmarks to another. By isolating the yaw from ASM, we transform a landmarked face into a neutral, frontal pose. The other ASM-derived vectors enable other transformations. 1.97 0.93 Original off-pose image Landmark map Best landmarks Facial Landmarking ASEF Overview:The original image is cropped to the face and landmarked to determine possible feature locations on the face. The best combination of feature locations is selected using a combination of spatial heuristics and statistical estimation. Using a statistical model of landmark variation with pose, the landmarks are neutrally posed. These landmarks are used to warp the image to a neutral pose. Example yaw warp: off-pose (right) to frontal pose (middle) The original and a forward-facing comparison image and the resulting match scores. For reference, the self-match score is about 6.29 Results and Deliverables • The team is delivering to MITRE: • Code implementing ASEF and UMACE landmarking, Active Shape Models and its component algorithms, and our pose-correction technique, as well as scripts and applications for testing and demonstrating all of these algorithms. • Accuracy results for landmarking and pose-corrected match scores. Because full-image pose-correction can lead to lower match scores, improved face recognition may result from comparing feature-relative patches instead of warped full images. The team's feature extraction routines will form the basis of that process. Training image Synthetic image Exact filter Average filter (ASEF) Landmarking heuristics Modeling Faces with Active Shape Models The Average of Synthetic Exact Filters (ASEF) is a texture-based method for landmarking. To create an ASEF filter we specify a desired synthetic output for each training image. That desired output consists of a Gaussian dot centered at the ground truth landmark position in the training image. This results in a filter that exactly transforms the training image into the synthetic image. We average all of a dataset'sexact filters to get the final ASEF filter, which can then be applied to facial images to locate the landmark. multiple responses An Active Shape Model (ASM) uses a dataset's statistics to capture the possible shapes that objects of a certain class can take. We used face shapes consisting of sets of landmarks (nose tip, mouth corners, etc.) and trained the model to recognize the configuration of the average face in our training data. We also find the most significant parameters that describe the ways a face can vary from the average while still representing a viable face. King of the Hill is a technique for finding the n local maxima in a two-dimensional array. We use it to determine the top three possible locations in our UMACE or ASEF filter responses. UMACE geometric constraints The mean face (green) computed from 300 faces (white), after alignment via the Procrustes algorithm the team implemented. Facial features tend to end up in roughly the same area of each facial crop. We can take advantage of this by constraining the area in which we search for each landmark. Circles showing 5%, 10%, and 25% of inter-ocular distance Accuracy results for the best-match feature in each of six locations Facial image with right-eye region UMACE filter Response of eye region to filter Three standard deviations from the mean along the largest source of variation, roughly corresponding to yaw combining features We also investigated and implemented a feature-detection algorithm involving the use of Unconstrained Minimum Average Correlation Energy (UMACE) filters. We divide the average values of a standard square region around the ground truth eye location of each training image by the average power spectrum for that same region. This gives us a correlation filter that we can apply to a standard eye-containing region to determine possible eye locations within that region. An Active Shape Model (at right) and/or feature-strength heuristics can provide a probability that a particular set of landmarks form a face shape. The current system uses only feature strength, but can support additional metrics in the future. Acknowledgments MITRE Liaisons Joshua Klontz '10 Mark Burge Team Members Elliot Godzich '12 Dylan Marriner '12 Emily Myers-Stanhope '12 Emma Taborsky '12 (PM) Heather Williams '12 Three standard deviations from the mean along the largest source of variation, roughly representing pitch Faculty Advisor Zachary Dodds