Graph Cut Selections

1. Graph Cut Selections Stephen J. Guy

2. GrabCut – Interactive Foreground Extraction1 Photomontage

3. Agenda • Background • Segmentation Intro • Graph Cut for Segmentation (Boykov & Jolly ‘01) • Copying • Lazy Snapping (Li et al. ’04) • GrabCut (Rother et al. ’04) • Other • Texture & Video Synthesis (Kwatra et al. ’04)

4. Segmentation • Goal • Splitting Image into similar, meaningful(?) regions • E.g. Clustering, Edge Detection, *Graph Cuts* GrabCut “SuperPixels” Rother et al. 2004 X. Ren and J. Malik. Learning a classification model for segmentation. ICCV 2003.

5. Optimization Formulation - Boykov & Jolly ‘01 • A – Proposed Segmentation • E(A) – Overall Energy • R(A) – Degree to which pixels fits model • B(A) – Degree to which the cuts breaks up similar pixels • - Balance A() and B() • Goal: Find Segmentation, A, which minimizes E(A)

6. Graph Cut Basics • Image As Graph • Each pixel is a node • Create links between each pixel • More nodes/links possible (e.g. foreground node) • Graph Cuts • Assign energy for each link based on similarity • Break the links with least energy, but • Require each pixel is still connected to exactly one foreground or background node • Benefits • No Topological constraints on regions (can be concave, unconnected, etc.) • Fast (polynomial time) solution – Ford-Fulkerson Algorithm

7. Graph Cut Overview

8. Link Weights • Pixel links based on color/intensity similarities • Source/Target links based on histogram models of fore/background

9. Gestalt Example • Effect of • Recall:

10. Lazy Snapping

11. Lazy Snapping Overview • Main Steps: • Mark strokes as foreground & background • Perform Boykov & Jolly style graphcut • *Edit Boundary* • Familiar Formulation • Minimize Gibbs Energy: • E1(X) – Cluster (K-means) known foreground/background colors to find representative colors • E2(X) – Similar to Boykov & Jolly

12. Lazy Snapping – Issue • Too Slow to allow for real-time refinement • Solution? Superpixels!

13. Lazy Snaping – Speedup

14. Boundary Editing • Graph Cut is only step 1 • Allow user to edit boundary directly • Formulate as Graph Cut • Allow user to guide boundary regions w/ brush

15. Lazy Snapping – Video

16. User Study • Compared Lazy Snapping to Photoshop

17. User Study • Results • Easy of Use – Lazy Snapping 20% less mistakes • Speed – Lazy Snapping 60% less time • Accuracy – Lazy Snapping 60% less pixels wrong • Feedback • “Almost Magic” • “Much Easier” • Suggestions • Get rid of 2 step process • Make it easy to switch between graph cut strokes and boundary editing

18. More Results

19. GrabCutInteractive Foreground Extraction using Iterated Graph CutsCarsten RotherVladimir KolmogorovAndrew BlakeMicrosoft Research Cambridge-UK

20. Big Changes • New model of foreground & background pixels: • GMM vs. K-mean colors vs. Intensity Histogram • More automation: • No need to specify foreground • Iterative Graph Cuts

22. GrabCut – Interactive Foreground Extraction3 What GrabCut does Magic Wand(198?) Intelligent ScissorsMortensen and Barrett (1995) GrabCut User Input Result Regions Regions & Boundary Boundary

23. Grab Cut Basics • Assign Selection box or Lasso as background • Assign other pixels as unknown • Learn Model of Fore/Background • Assign Link Energy, GraphCut (Boykov & Jolly) • Use GraphCut results to assign Fore/Background • Goto Step 3 • Allow user to add cleanup strokes

24. Foreground (source) Min Cut Background(sink) Cut: separating source and sink; Energy: collection of edges Min Cut: Global minimal enegry in polynomial time GrabCut – Interactive Foreground Extraction5 Graph CutsBoykov and Jolly (2001) Image

25. GrabCut – Interactive Foreground Extraction6 Iterated Graph Cut ? User Initialisation Graph cuts to infer the segmentation K-means for learning colour distributions

26. GrabCut – Interactive Foreground Extraction7 Iterated Graph Cuts Guaranteed toconverge 1 3 4 2 Result Energy after each Iteration

27. GrabCut – Interactive Foreground Extraction8 Colour Model Gaussian Mixture Model (typically 5-8 components) R R Iterated graph cut Foreground &Background Foreground G Background G Background

28. GrabCut – Interactive Foreground Extraction10 Moderately straightforward examples … GrabCut completes automatically

29. GrabCut – Interactive Foreground Extraction11 Difficult Examples Camouflage & Low Contrast Fine structure No telepathy Initial Rectangle InitialResult

30. Error Rate: 0.72% GrabCut – Interactive Foreground Extraction13 Comparison Boykov and Jolly (2001) GrabCut User Input Result Error Rate: 1.87% Error Rate: 1.81% Error Rate: 1.32% Error Rate: 1.25% Error Rate: 0.72%

31. GrabCut – Interactive Foreground Extraction16 Border Matting Hard Segmentation Automatic Trimap Soft Segmentation

32. GrabCut – Interactive Foreground Extraction17 Natural Image Matting Mean Colour Foreground Mean ColourBackground Solve Ruzon and Tomasi (2000):Alpha estimation in natural images

33. GrabCut – Interactive Foreground Extraction19 Border Matting Foreground Noisy alpha-profile 1 Mix Back-ground 0 Foreground Background Mix Fit a smooth alpha-profile with parameters

34. GrabCut – Interactive Foreground Extraction21 Results

35. GrabCut – Interactive Foreground Extraction22 Summary Intelligent Scissors Mortensen and Barrett (1995) LazySnappingLi et al. (2004) Graph Cuts Boykov and Jolly (2001) Magic Wand (198?) GrabCutRother et al. (2004)

36. Comparison

37. Bonus Paper • Texture Synthesis Revisited

38. Basic Idea • Copy overlapping patches one at a times • Find the seamwhich minimize change in neighboring pixels • Formulate as Graph Cut

39. Interesting Advantage • Extends well into 3D