Combinatorial Optimization and Computer Vision

1. Combinatorial Optimization andComputer Vision Philip Torr

2. Story • How an attempt to solve one problem lead into many different areas of computer vision and some interesting results.

3. Aim Object Category Model • Given an image, to segment the object Segmentation Cow Image Segmented Cow • Segmentation should (ideally) be • shaped like the object e.g. cow-like • obtained efficiently in an unsupervised manner • able to handle self-occlusion

4. Challenges Shape Variability Appearance Variability Self Occlusion

5. Motivation • Current methods require user intervention • Object and background seed pixels (Boykov and Jolly, ICCV 01) • Bounding Box of object (Rother et al. SIGGRAPH 04) Object Seed Pixels Cow Image

6. Motivation • Current methods require user intervention • Object and background seed pixels (Boykov and Jolly, ICCV 01) • Bounding Box of object (Rother et al. SIGGRAPH 04) Object Seed Pixels Background Seed Pixels Cow Image

7. Motivation • Current methods require user intervention • Object and background seed pixels (Boykov and Jolly, ICCV 01) • Bounding Box of object (Rother et al. SIGGRAPH 04) Segmented Image

8. Motivation • Current methods require user intervention • Object and background seed pixels (Boykov and Jolly, ICCV 01) • Bounding Box of object (Rother et al. SIGGRAPH 04) Object Seed Pixels Background Seed Pixels Cow Image

9. Motivation • Current methods require user intervention • Object and background seed pixels (Boykov and Jolly, ICCV 01) • Bounding Box of object (Rother et al. SIGGRAPH 04) Segmented Image

10. Motivation • Problem • Manually intensive • Segmentation is not guaranteed to be ‘object-like’ Non Object-like Segmentation

11. MRF for Image Segmentation Boykov and Jolly [ICCV 2001] EnergyMRF = Unary likelihood Contrast Term Pair-wise terms (Potts Model) Maximum-a-posteriori (MAP) solution x*= arg min E(x) x Data (D) Unary likelihood Pair-wise Terms MAP Solution

12. GraphCut for Inference Source Foreground Cut Image Background Sink Cut:A collection of edges which separates the Source from the Sink MinCut:The cut with minimum weight (sum of edge weights) Solution:Global optimum (MinCut) in polynomial time

13. Energy Minimization using Graph cuts Graph Construction for Boolean Random Variables EMRF(a1,a2) Source (0) a1 a2 Sink (1)

14. Energy Minimization using Graph cuts EMRF(a1,a2) =2a1 Source (0) 2 t-edges (unary terms) a1 a2 Sink (1)

15. Energy Minimization using Graph cuts EMRF(a1,a2) = 2a1 + 5ā1 Source (0) 2 a1 a2 5 Sink (1)

16. Energy Minimization using Graph cuts EMRF(a1,a2) = 2a1 + 5ā1+ 9a2 + 4ā2 Source (0) 2 9 a1 a2 5 4 Sink (1)

17. Energy Minimization using Graph cuts EMRF(a1,a2) = 2a1 + 5ā1+ 9a2 + 4ā2 +2a1ā2 Source (0) 2 9 a1 a2 2 5 4 n-edges (pair-wise term) Sink (1)

18. Energy Minimization using Graph cuts EMRF(a1,a2) = 2a1 + 5ā1+ 9a2 + 4ā2 + 2a1ā2 +ā1a2 Source (0) 2 9 1 a1 a2 2 5 4 Sink (1)

19. Energy Minimization using Graph cuts EMRF(a1,a2) = 2a1 + 5ā1+ 9a2 + 4ā2 + 2a1ā2 +ā1a2 Source (0) 2 9 1 a1 a2 2 5 4 Sink (1)

20. Energy Minimization using Graph cuts EMRF(a1,a2) = 2a1 + 5ā1+ 9a2 + 4ā2 + 2a1ā2 +ā1a2 Source (0) 2 9 Cost of st-cut = 11 1 a1 a2 a1 = 1 a2 = 1 2 5 4 EMRF(1,1) = 11 Sink (1)

21. Energy Minimization using Graph cuts EMRF(a1,a2) = 2a1 + 5ā1+ 9a2 + 4ā2 + 2a1ā2 +ā1a2 Source (0) 2 9 Cost of st-cut = 8 1 a1 a2 a1 = 1 a2 = 0 2 5 4 EMRF(1,0) = 8 Sink (1)

22. Computing the st-mincut from Max-flow algorithms Source (0) • The Max-flow Problem • Edge capacity and flow balance constraints 2 9 • Notation • Residual capacity • (edge capacity – current flow) 1 a1 a2 2 5 4 • Simple Augmenting Path based Algorithms • Repeatedly find augmenting paths and push flow. • Saturated edges constitute the st-mincut. • [Ford-Fulkerson Theorem] Sink (1)

23. Minimum s-t cuts algorithms • Augmenting paths [Ford & Fulkerson, 1962] • Push-relabel [Goldberg-Tarjan, 1986]

24. “source” “sink” T S A graph with two terminals “Augmenting Paths” • Find a path from S to T along non-saturated edges • Increase flow along this path until some edge saturates

25. “source” “sink” T S A graph with two terminals “Augmenting Paths” • Find a path from S to T along non-saturated edges • Increase flow along this path until some edge saturates • Find next path… • Increase flow…

26. “source” “sink” T S A graph with two terminals  MIN CUT “Augmenting Paths” • Find a path from S to T along non-saturated edges • Increase flow along this path until some edge saturates Iterate until … all paths from S to T have at least one saturated edge MAX FLOW

27. MRF, Graphical Model • Probability for a labellingconsists of • Likelihood Unary potential based on colour of pixel • Prior which favours same labels for neighbours (pairwise potentials) mx m(labels) Prior Ψxy(mx,my) my Unary Potential Φx(D|mx) x y D(pixels) Image Plane

28. Example Cow Image Object Seed Pixels Background Seed Pixels Φx(D|obj) x … x … Φx(D|bkg) Ψxy(mx,my) y … y … … … … … Prior Likelihood Ratio (Colour)

29. Example Cow Image Object Seed Pixels Background Seed Pixels Pair-wise Terms Likelihood Ratio (Colour)

30. Contrast-Dependent MRF • Probability of labelling in addition has • Contrast term which favours boundaries to lie on image edges mx m(labels) my x Contrast Term Φ(D|mx,my) y D(pixels) Image Plane

31. Example Cow Image Object Seed Pixels Background Seed Pixels Φx(D|obj) x … x … Φx(D|bkg) Ψxy(mx,my)+ Φ(D|mx,my) y … y … … … … … Pair-wise Term Likelihood Ratio (Colour)

32. Example Cow Image Object Seed Pixels Background Seed Pixels Prior + Contrast Likelihood Ratio (Colour)

33. Object Graphical Model • Probability of labelling in addition has • Unary potential which depend on distance from Θ (shape parameter) Θ (shape parameter) Unary Potential Φx(mx|Θ) mx m(labels) my Object Category Specific MRF x y D(pixels) Image Plane

34. Example Cow Image Object Seed Pixels Background Seed Pixels ShapePriorΘ Prior + Contrast Distance from Θ

35. Example Cow Image Object Seed Pixels Background Seed Pixels ShapePriorΘ Prior + Contrast Likelihood + Distance from Θ

36. Example Cow Image Object Seed Pixels Background Seed Pixels ShapePriorΘ Prior + Contrast Likelihood + Distance from Θ

37. Thought • We can imagine rather than using user input to define histograms we use object detection.

38. Shape Model • BMVC 2004

39. Yuille, ‘91 • Brunelli & Poggio, ‘93 • Lades, v.d. Malsburg et al. ‘93 • Cootes, Lanitis, Taylor et al. ‘95 • Amit & Geman, ‘95, ‘99 • Perona et al. ‘95, ‘96, ’98, ‘00 Pictorial Structure Fischler & Elschlager, 1973

40. Layered Pictorial Structures (LPS) • Generative model • Composition of parts + spatial layout Layer 2 Spatial Layout (Pairwise Configuration) Layer 1 Parts in Layer 2 can occlude parts in Layer 1

41. Layered Pictorial Structures (LPS) Cow Instance Layer 2 Transformations Θ1 P(Θ1) = 0.9 Layer 1

42. Layered Pictorial Structures (LPS) Cow Instance Layer 2 Transformations Θ2 P(Θ2) = 0.8 Layer 1

43. Layered Pictorial Structures (LPS) Unlikely Instance Layer 2 Transformations Θ3 P(Θ3) = 0.01 Layer 1

44. How to learn LPS • From video via motion segmentation see Kumar Torr and Zisserman ICCV 2005. • Graph cut based method.

45. Examples

46. LPS for Detection • Learning • Learnt automatically using a set of examples • Detection • Matches LPS to image using Loopy Belief Propagation • Localizes object parts

47. Detection • Like a proposal process.

48. Pictorial Structures (PS) Fischler and Eschlager. 1973 PS = 2D Parts + Configuration Aim: Learn pictorial structures in an unsupervised manner Layered Pictorial Structures (LPS) Parts + Configuration + Relative depth • Identify parts • Learn configuration • Learn relative depth of parts

49. P2 (x,y,,) P1 P3 MRF Image Motivation Matching Pictorial Structures - Felzenszwalb et al - 2001 Outline Texture Part likelihood Spatial Prior

50. YES NO 2 1 P2 (x,y,,) P1 P3 MRF Image Motivation Matching Pictorial Structures - Felzenszwalb et al - 2001 • Unary potentials are negative log likelihoods Valid pairwise configuration Potts Model