Feature Detection and Emotion Recognition

1. Feature Detection and Emotion Recognition Chris Matthews Advisor: Prof. Cotter

2. Motivation #1: Attempt to Answer a Long-Existing Question • Used to definitively characterize what expressions the Mona Lisa is displaying (F.Y.I., she is 83% happy, 9% disgusted, 6% fearful and 2% angry, according to BBC News)

3. Motivation #2: Create “Life-Like” Robots • Create convincingartificial intelligence.

4. Motivation #3: Enhance Society! • Currently being used to teach autistic children to pick out facial subtleties and their corresponding emotions

5. Methodology FEATURE DETECTION • Isolate and crop particular areas of the face EMOTION RECOGNITION Training • Train neural networks for each area • Combine the resultants from each and come out with a definitive result • Alter variables of the networks by trial-and-error until the desired results are achieved Testing • Input new photos into the trained network and check results

6. Feature Detection:SUSAN filtering for Edge Detection • Because no derivatives are implemented in SUSAN, the algorithm excels in noisy images

7. Mouth not fully enclosed Only the pupil of the left eye is enclosed Even if everything was perfectly encapsulated, how would one make sense of the detected objects? Massive Problem: Boolean images don’t necessarily make Computer Vision problems easier!

8. Lesson Learned: Complete Automation is difficult! • New methodology for isolating parts of the face – manual labor. • Draw matrices over the approximate area of interest • Apply filters to detect the actual object of interest • Crop again based on those findings

9. The neural network will perform poorly if there is variance in either the x or y directions, from photo to photo Example: The Uncentered Eye

11. Voila!

12. On to the Emotion Training… • Once the areas have been defined and scaled, they can be used as inputs to neural networks

13. Introduction to Neural Networks: The Perceptron

14. Perceptron Implementation • Initialize weight matrix and bias array to small, random values. • Feed an image through the network • Calculate the error • Readjust the weight matrix and bias array based on the error • Iteratively train the network using a dictionary of photos.

15. Yet another problem! • Each neuron has one weight value for each pixel • Weight matrix is too large to train!

16. Solution: PCA • Principle Component Analysis generates a set of eigenvectors. • Each picture can be reconstructed using a weighted sum of these eigenvectors.

18. Final Architecture • Use a set of adaptive backpropagation networks, training on PCA coefficients. • Use majority rules to determine the emotion.

19. Results • Training with 60 photos yielded 100% accuracy mapping to only two targets: happy and sad • Training with 112 photos yielded 60% accuracy mapping to four targets: angry, fearful, happy, and sad.

20. Future Work • Find larger and more diverse image dictionaries • Improve Feature Detection • Read Psychological Journals and apply their findings into the algorithms

21. Questions?

22. A gross simplification of how SUSAN works • Smallest Univalue Segment Assimilating Nucleus • Edge if n = (½)*pi*r^2 • Corner if n << (½)*pi*r^2