1. GrabCutInteractive Image(and Stereo) Segmentation Joon Jae LeeKeimyung University

2. Characteristics Improved from graph cut – Use Gaussian Mixture Model (GMM) for clustering in color space – Iterative optimization User interaction greatly reduced compared with other methods (Magic Wand, Intelligent Scissors)

3. GrabCut – Interactive Foreground Extraction1 Photomontage

4. GrabCut – Interactive Foreground Extraction3 Problem Fast & Accurate ?

5. GrabCut – Interactive Foreground Extraction4 What GrabCut does Magic Wand(198?) Intelligent ScissorsMortensen and Barrett (1995) GrabCut User Input Result Regions Regions & Boundary Boundary

6. Gaussian Mixture Model The probability of a pixel vector is represented as:

7. Gaussian Mixture Model Two sets of GMM, one for background, one for unknown region

8. Algorithm For each pixel assign a value α α = 0 for background and α = 1 for foreground Use graph cut to minimize a Gibbs energy: α: background / foreground label θ: GMM parameters k: GMM labeling z: pixel value

9. Iterative minimization Initialize background pixel to α = 0 and unknown region (draw box) to α = 1. Initialize two sets of GMMs. 1. Assign GMM labels to each pixel. (Which set of GMM is determined by α) 2. Graph cut minimize (α optimized, GMM label changed to corresponding set of GMM) 3. Update GMM parameters 4. Repeat from step 1 until convergence

10. Foreground (source) Min Cut Background(sink) Cut: separating source and sink; Energy: collection of edges Min Cut: Global minimal enegry in polynomial time GrabCut – Interactive Foreground Extraction7 Graph Cuts - Boykov and Jolly (2001) Image

11. GrabCut – Interactive Foreground Extraction9 Iterated Graph Cuts Guaranteed toconverge 1 3 4 2 Result Energy after each Iteration

12. GrabCut – Interactive Foreground Extraction10 Colour Model Gaussian Mixture Model (typically 5-8 components) R R Iterated graph cut Foreground &Background Foreground G Background G Background

13. Coherence Model An object is a coherent set of pixels: 25 Error (%) over training set: How do we choose ? 25

14. GrabCut – Interactive Foreground Extraction13 Parameter Learning - Problems A Gaussian MRF is not a realistic texture model syntheticGMRF Gaussian? Real Image Gaussian!

15. GrabCut – Interactive Foreground Extraction14 Moderately simple examples … GrabCut completes automatically

16. GrabCut – Interactive Foreground Extraction15 Difficult Examples Camouflage & Low Contrast Fine structure No telepathy Initial Rectangle InitialResult

17. GrabCut – Interactive Foreground Extraction16 Evaluation – Labelled Database Available online: http://research.microsoft.com/vision/cambridge/segmentation/

18. Error Rate: 0.72% GrabCut – Interactive Foreground Extraction17 Comparison Boykov and Jolly (2001) GrabCut User Input Result Error Rate: 1.87% Error Rate: 1.81% Error Rate: 1.32% Error Rate: 1.25% Error Rate: 0.72%

19. GrabCut – Interactive Foreground Extraction18 Comparison BimapGrabCut Error Rate: 2.13% Input Image Ground Truth Trimap Boykov and Jolly Error Rate: 1.36% Error rate - modestly increase User Interactions - considerable reduced

20. GrabCut – Interactive Foreground Extraction19 Results Parameter Learning

21. GrabCut – Interactive Foreground Extraction20 Comparison Intelligent Scissors Mortensen and Barrett (1995) LazySnappingLi et al. (2004) Graph Cuts Boykov and Jolly (2001) Magic Wand (198?) GrabCutRother et al. (2004)