Names and Faces

1. Names and Faces Presented By Krishnan Ramnath Robotics Institute, CMU, Pittsburgh

2. Agenda • Motivation and Goal • Dataset • Algorithm Outline • Results

3. Goal President George W. Bush makes a statement in the Rose Garden while Secretary of Defense Donald Rumsfeld looks on, July 23, 2003. Rumsfeld said the United States would release graphic photographs of the dead sons of Saddam Hussein to prove they were killed by American troops. Photo by Larry Downing/Reuters

4. Motivation • The web has made large multimedia collections readily available - Organise contents • Commercial face recognition systems do not work in practice. - Datasets were captured in labs and lack the complexity of real world faces

5. Airport Face Scanner Failed (Wired News) Facial recognition technology tested at the Palm Beach International Airport had a dismal failure rate, according to preliminary results from a pilot program at the facility. The Palm Beach airport tried Visionics' FaceIt system, which snaps photographs of passersby using a security camera and breaks down their facial features into a numeric code that is matched against the photograph database. The system failed to correctly identify airport employees 53 percent of the time, according to test data that was obtained by the American Civil Liberties Union under Florida's open records law. "The preliminary results at the Palm Beach International Airport confirm that the use of facial recognition technology is simply ineffective and of no value," said Randall Marshall, legal director of the state ACLU chapter.

6. Methodology 1) Collect a new dataset from news photographs - Individual faces are not labeled 2) Developed a technique for automatically associating names with faces Applications: • Organise news pictures by the individuals present • Browse by individual

7. Dataset • 500,000 news pictures, associated captions • Pictures captured “in the wild” • Mainly people: politicians, actors, sports players • Multiple names for one face • Heavy tailed distribution • Multiple faces for one name

8. Dataset France's Amelie Mauresmo has pulled out of the Australian Open (news - web sites) because of a knee injury. The French number one, ranked sixth in the world, has been struggling with a painful knee cap for two months and said on December 18, 2002 she would not be fit in time to resume playing in January. Mauresmo is shown during the Kremlin Cup, Oct. 4. (Grigory Dukor/Reuters) Actress Winona Ryder (news) reacts to remarks by prosecutor Ann Rundle during the sentencing hearing in her felony shoplifting case Friday, Dec. 6, 2002 at the Beverly Hills, Calif., courthouse. At right is Ryder's attorney Mark Geragos. Ryder was sentenced to three years of probation and was ordered to perform 480 hours of community service. (AP Photo/Steve Grayson, POOL)

9. Dataset (cont) Doctor Nikola shows a fork that was removed from an Israeli woman who swallowed it while trying to catch a bug that flew in to her mouth, in Poriah Hospital northern Israel July 10, 2003. Doctors performed emergency surgery and removed the fork. (Reuters) President George W. Bush waves as he leaves the White House for a day trip to North Carolina, July 25, 2002. A White House spokesman said that Bush would be compelled to veto Senate legislation creating a new department of homeland security unless changes are made. (Kevin Lamarque/Reuters)

10. Dataset (cont.) Actor Arnold Schwarzenegger discusses his plans for his campaign with reporters after he announced that he will be a candidate for governor of California in the recall election, August 6, 2003. Schwarzenegger ended weeks of speculation about his candidacy for the October 7 referendum. (Fred Prouser/Reuters Producer and director Bruce Paltrow has died at the age of 58 in Rome, Italy, the U.S. Consulate said on October 3, 2002. Paltrow had suffered from throat cancer for several years, but the cause of his death was not immediately known. He is seen with his daughter actress Gwyneth Paltrow after the Academy Awards in Los Angles in March 21, 1999 file photo. (Fred Prouser/Reuters)

11. Algorithm Outline Proper Name Extraction Face Detection Face Rectification Build Face Representation Assign names to faces by clustering (CVPR) Improve clustering by incorporating a language model (NIPS)

12. Algorithm Outline • Proper Name Extraction • Face Detection • Face Rectification • Build Face Representation • Assign names to faces by clustering (CVPR) • Improve clustering by incorporating a language model (NIPS)

13. Proper Name Extraction President George W. Bush makes a statement in the Rose Garden while Secretary of Defense Donald Rumsfeld looks on, July 23, 2003. Rumsfeld said the United States would release graphic photographs of the dead sons of Saddam Hussein to prove they were killed by American troops. Photo by Larry Downing/Reuters

14. Proper Name Extraction • Determine proper name strings • Captions are very stylized • Use morphological rules • WordNet for parts of speech President George W. Bush makes a statement in the Rose Garden while Secretary of Defense Donald Rumsfeld looks on, July 23, 2003. Rumsfeld said the United States would release graphic photographs of the dead sons of Saddam Hussein to prove they were killed by American troops. Photo by Larry Downing/Reuters Detected Names: President George, Defense Donald Rumsfeld, Donald Rumsfeld, Saddam Hussein.

15. Proper Name Extraction • Extract Names using an open source proper name detector: H. Cunningham, D. Maynard, K. Bontcheva, V. Tablan, “GATE: A Framework and Graphical Development Environment for Robust NLP Tools and Applications” • An Example: • Information Extraction

16. Algorithm Outline • Proper Name Extraction • Face Detection • Face Rectification • Build Face Representation • Assign names to faces by clustering (CVPR) • Improve clustering by incorporating a language model (NIPS)

17. Face Detection • Face Detector • courtesy of Cordelia Schmid (Schneidermann & Kanade) • 45,000 large faces • mainly frontal • Scaled to 86x86 pixels President George W. Bush makes a statement in the Rose Garden while Secretary of Defense Donald Rumsfeld looks on, July 23, 2003. Rumsfeld said the United States would release graphic photographs of the dead sons of Saddam Hussein to prove they were killed by American troops. Photo by Larry Downing/Reuters

18. Face Detector • Schneidermann and Kanade • Representation by subsets of Quantized Wavelet Coefficients

19. Algorithm Outline • Proper Name Extraction • Face Detection • Face Rectification • Build Face Representation • Assign names to faces by clustering (CVPR) • Improve clustering by incorporating a language model (NIPS)

20. Rectify Faces • Feature detectors - trained to recognize nose, corners of eyes and mouth (SVM is used for this) • Determine affine transformation between detected and canonical positions • Project images into canonical pose

21. Image Features Geometric Blur Feature Similarity (A.) Berg & Malik ‘01 x0 x0 Sparse Signal, S Geometric Blur, B Apply a spatially varying blur, and sample values Authors claim significant gain in rectification accuracy using this

22. Rectification • Rectification score helps eliminate errors in face detection

23. Algorithm Outline • Proper Name Extraction • Face Detection • Face Rectification • Build Face Representation • Assign names to faces by clustering (CVPR) • Improve clustering by incorporating a language model (NIPS)

24. Face Representation • Faces initially represented as vectors of pixels • Huge dataset – Need good representation • Dimensionality Reduction • Kernel PCA -- dimension reduction onto nonlinear basis. • Compute kernel function, K(i,j) for each pair of images • Calculate eigenvectors of kernel matrix and project

25. Kernel PCA Use kernel function to compute PCs in high-dimensional feature space • Compute a dot product kernel matrix • They use a Gaussian kernel for this work • Center the matrix by subtracting off avg row, column and adding average element values • Compute Eigen decomposition of kernel matrix • Compute projections of test point onto the normalized eigenvectors of kernel matrix Non-linear Component Analysis as a Kernel Eigenvalue Problem, Bernhard Scholkopf, Alexander Smola, Klaus-Robert Muller

26. Issues and Hacks • Dataset too large – kernel matrix – NxN (N = images) • Approximation to calculate eigenvectors of K: • Nystrom Approximation – • Compute subsets of kernel matrix and use them to approximate whole of kernel matrix • Only compute N*m entries in K • Only compute eigenvectors of mxm matrices

27. Partition kernel matrix as: • Approximate C with Nystrom Method Solve problem for a random subset of pixels and extrapolate to full set of pixels • A in this case was computed from 1000 randomly selected images Spectral Grouping using Nystrom Method, Fowlkes, Serge Belongie, Fan Chung, Jitendra Malik

28. Face Representation (cont.) • We are dealing with face classes • We need to work in the space that is useful for telling classes apart • Efficient clustering in discriminant coordinates • Linear Discriminants Analysis • Discriminants take advantage of class information • Don’t know classes • Build discriminants using one name faces, project all face vectors into this space

29. Face Representation (cont.)

30. Within-Class Scatter Matrix (average scatter of classes ard mean) • Between-Class Scatter Matrix (scatter of condnal means ard overall mean) • LDA computes projection that maximizes the ratio: • By solving the generalised eigenvalue problem: Linear Discriminants Analysis • Consider: and

31. Algorithm Outline • Proper Name Extraction • Face Detection • Face Rectification • Build Face Representation • Assign names to faces by clustering (EM) • Improve clustering by incorporating a language model

32. Name-Face Assignment • Name Assignment = hidden variable problem • Expectation Maximization over name-face correspondences (hidden variables) • Mantra: • E: Given an initial name-face clustering, estimate name-face correspondences • M: Update clusters using estimated correspondences

33. Name-Face Assignment

34. EM • Likelihood of a picture under a given assignment of names-faces: • Complete data log likelihood: P(face|name) Hidden variable Images Correspondences

35. EM vs MM • Maximal Assignment • Instead of expected value, only maximum likelihood is assigned non-zero probability of 1 • EM (soft) vs MM (hard) • Claim: MM better • Reason: EM assigns weight to incorrect assignments, expected value influenced by wrong assignments

36. Algorithm Outline • Proper Name Extraction • Face Detection • Face Rectification • Build Face Representation • Assign names to faces by clustering (EM) • Improve clustering by incorporating a language model

37. Improved Clustering: Context Understanding • Context provides cues as to whether name is pictured or not • Incorporate a language model • Additions to EM: • Find expected name-face (given a face clustering and language model) • Update face clusters and language model given correspondences

38. Language Model P(Pictured | Context) Cues: POS tags before and after name, location in caption, distance to closest {. , ( ) (L)(C)(R)}. Cue Combination: multiple independent cues Yes/No

39. Language Model • Naïve Bayes model vs Maximum Entropy model • Maximum Entropy Model: • Specify set of constraints based on statistics • Choose model consistent with statistics • As uniform as possible • Similar performance (ME slightly better) • Also: Word and Face context : Not much improvement

40. Algorithm Outline • Proper Name Extraction • Face Detection • Face Rectification • Build Face Representation • Assign names to faces by clustering (EM) • Improve clustering by incorporating a language model • Results

41. Results British director Sam Mendes and his partner actress Kate Winslet arrive at the London premiere of 'The Road to Perdition', September 18, 2002. The films stars Tom Hanks as a Chicago hit man who has a separate family life and co-stars Paul Newman and Jude Law. REUTERS/Dan Chung World number one Lleyton Hewitt of Australia hits a return to Nicolas Massu of Chile at the Japan Open tennis championships in Tokyo October 3, 2002. REUTERS/Eriko Sugita

42. Results US President George W. Bush (L) makes remarks while Secretary of State Colin Powell (R) listens before signing the US Leadership Against HIV /AIDS , Tuberculosis and Malaria Act of 2003 at the Department of State in Washington, DC. The five-year plan is designed to help prevent and treat AIDS, especially in more than a dozen African and Caribbean nations(AFP/Luke Frazza) German supermodel Claudia Schiffer gave birth to a baby boy by Caesarian section January 30, 2003, her spokeswoman said. The baby is the first child for both Schiffer, 32, and her husband, British film producer Matthew Vaughn, who was at her side for the birth. Schiffer is seen on the German television show 'Bet It...?!' ('Wetten Dass...?!') in Braunschweig, on January 26, 2002. (Alexandra Winkler/Reuters)

43. The Results of Adding Language before: CEO Summit after: Martha Stewart before: US House after: Andrew Fastow before: Julia Vakulenko after: Jennifer Capriati before: James Bond after: Pierce Brosnan before: Vice President Dick Cheney after: President George W. before: al Qaeda after: Null before: Marcel Avram after: Michael Jackson before: Ric Pipino after: Heidi Klum Before: Image based clustering After: Image + language based clustering

44. With Lang Model • Without Lang Model

45. Caption Labeling IN Pete Sampras IN of the U.S. celebrates his victory over Denmark's OUT Kristian Pless OUT at the OUT U.S. Open OUT at Flushing Meadows August 30, 2002. Sampras won the match 6-3 7- 5 6-4. REUTERS/Kevin Lamarque Germany's IN Chancellor Gerhard Schroeder IN, left, in discussion with France's IN President Jacques Chirac IN on the second day of the EU summit at the European Council headquarters in Brussels, Friday Oct. 25, 2002. EU leaders are to close a deal Friday on finalizing entry talks with 10 candidate countries after a surprise breakthrough agreement on Thursday between France and Germany regarding farm spending.(AP Photo/European Commission/HO) 'The Right Stuff' cast members IN Pamela Reed IN, (L) poses with fellow cast member IN Veronica Cartwright IN at the 20th anniversary of the film in Hollywood, June 9, 2003. The women played wives of astronauts in the film about early United States test pilots and the space program. The film directed by OUT Philip Kaufman OUT, is celebrating its 20th anniversary and is being released on DVD. REUTERS/Fred Prouser

46. Results • Face Labeling: • Caption Labeling:

47. WebInterface

48. Face Recognition • Dataset of 3076 faces (241 individuals) • Clusters were hand cleaned to remove erroneous labels • Half for testing and half for training • PCA (100) + NN = 9.4-15.4% • PCA + LDA (50) = 17-27.4% • Usual PCA+LDA claim : 80-90%!! • Conclusion: Challenging Dataset • Online Dataset • Online Results

49. What’s the story? • Problem Identification : Organising data • Dataset collection : News dataset • Usage of simple methods • Adding Language Information • Contributions • Good Face Recognition Dataset • Organising data on the web (News reels in this case)

50. Acknowledgements • Some of the slides for this presentation were obtained from the authors • Thanks to Ms.Tamara Berg, UC Berkeley for providing slides and links to online resources • Thanks for listening!