1 / 48

Image Segmentation: beyond Graph Cuts

Image Segmentation: beyond Graph Cuts. Victor Lempitsky. Image segmentation: the problem. background. object. Prior knowledge (“compactness”) Low-level cues (e.g. e dge cues) High-level knowledge (e.g. “Penguin on a rock”) User input. Why image segmentation?.

arden
Télécharger la présentation

Image Segmentation: beyond Graph Cuts

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Image Segmentation:beyond Graph Cuts Victor Lempitsky

  2. Image segmentation: the problem background object • Prior knowledge (“compactness”) • Low-level cues (e.g. edge cues) • High-level knowledge (e.g. “Penguin on a rock”) • User input

  3. Why image segmentation?

  4. Image segmentation: the story • Since long Ago: rule-based methods, such as thresholdingor region growing (magic wand) • Since 1988 [Kass, Witkin, Terzopoulos]: energy optimization via local curve evolution • Since 2001[Boykov, Jolly]:global energy optimization via graph cuts(st-mincut)

  5. Graph cut segmentation Exponential number of segmentations In polynomial time! [Boykov and Jolly, 2001] backgroundxp=0 S (“object”, xp=1) object xp=1 • “Unary” terms: • Color models • User “brushes” • “Pairwise” terms: • Ising prior • Edge cues Alternative notation: T (“background”, xp=0)

  6. Why go beyond? We can express: Edge cues & Ising prior (via Ppq) Brushes (set Fp or Bp to ∞) Color likelihoods (via Fp and Bp) but Part ILempitsky, Blake, Rother, Image Segmentation by Branch-and-Mincut, ECCV 2008 Part IILempitsky, Kohli, Rother, Image Segmentation with A Bounding Box Prior, ICCV 2009 How to segment a car in the image? How to ensure tightness of the bounding box? Still want non-local and efficient optimization!

  7. Image Segmentation by Branch-and-Mincut Victor LempitskyAndrew Blake Carsten Rother

  8. An example 0 0 s 1 1 image from UIUC car dataset [Hammer 64] [Greig, Porteous, Seheult 89] p q ... ... Standard “graph cut” segmentation energy [Boykov, Jolly 01]: Fp=1, Bp=0 Ppq t Fp=0, Bp=1 [Freedman, Zhang 05], [Ali, Farag, El-Baz 07],...

  9. A harder example image from UIUC car dataset Ppq 0 1 Optimal x min x, ω Optimal ω

  10. Energy optimization ω –global parameter (ω ∊ Ω0) Optimization options: • Choose reasonable ω, solve for x[Freedman, Zhang 05], [Pawan Kumar, Torr, Zisserman’ObjCut 05], [Ali, Farag, El-Baz 07] .... • Alternate between x and ω (EM) [Rother, Kolmogorov, Blake’ GrabCut 04], [Bray, Kohli, Torr’PoseCut 06], [Kim, Zabih 03].... • Optimize continuously [Chan, Vese 01], [Leventon, Grimson, Faugeras 00], [Cremers, Osher, Soatto 06], [Wang, Staib 98]... • Exhaustive search pairwise potentials (costs) ≥ 0 constant unary potentials (costs) ....shape priors, color distribution/intensity priors (Chan-Vese, GrabCut)...

  11. Our approach along x dimension along ω dimension • Extremely large, structured domain • Specific “graph cut” function • Low-dimensional (discretized) domain • Function of the general form Branch-and-bound Mincut Branch-and-Mincut [Gavrila, Philomin 99], [Lampert, Blaschko, Hofman 08], [Cremers, Schmidt, Barthel 08]

  12. Search tree Ω0 Ω0

  13. Bounding the energy: an example Fp=0, Bp=1 min Fp=1, Bp=0 ω1 ω2 Fp=0, Bp=0

  14. The lower bound s p q Ω t Computable with mincut!

  15. Lower bound • Monotonic increase towards leaves • Tightness at leaf nodes:

  16. Lower bound: example precomputed computed at runtime

  17. Branch-and-Bound Standard best-first branch-and-bound search: lowest lower bound B C A Small fraction of nodes is visited additional speed-up from “reusing” maxflow computations [Kohli,Torr 05]

  18. Results: shape prior 30,000,000 shapes Exhaustive search: 30,000,000 mincuts Branch-and-Mincut: 12,000 mincuts Speed-up: 2500 times (30 seconds per 312x272 image)

  19. Results: shape prior Left ventricle epicardium tracking (work in progress) Original sequence No shape prior Our segmentation Shape prior from other sequences 5,200,000 templates ≈20 seconds per frame Speed-up 1150 Data courtesy: Dr Harald Becher, Department of Cardiovascular Medicine, University of Oxford

  20. Result: shape prior Can add feature-based detector here UIUC car dataset

  21. Results: Discrete Chan-Vese functional cb c_p cf Chan-Vese functional [Chan, Vese 01]: ∊ [0;255]x[0;255]: quad-tree clustering Global minima of the discrete Chan-Vese functional: Speed-up 28-58 times

  22. Performance Sample Chan-Vese problem:

  23. Conclusion • ̶ good energy to integrate low-level and high-level knowledge in segmentation. • Branch-and-Mincut framework can find its global optimum efficiently in many cases • Ongoing work: Branch-and-X algorithms Branch-and-bound Mincut Dynamic Programming C++ code at http://research.microsoft.com/~victlem/

  24. Multi-label Augmented MRFs

  25. An experiment 22 points FGNet annotations by David Cristinacce and Kola Babalola BioID dataset: 1520 faces, 800 for training, 720 for testing

  26. Pictorial structure MRF [FelzenszwalbHuttenlocker 05] • Tree-structured MRF • 22 nodes • Label space – all image locations Unaries for eye corner Unaries for the chin

  27. Branch-and-DP [FelzenszwalbHuttenlocker 05] “Branch-and-DP” Ω0 10,000 configurations Scales/rotations/deformations 1 configuration Messages are cheap - O(n) (distance transforms) Messages are still cheap - O(n) (distance transforms + van Herk-Gil-Werman algorithm)

  28. Results Space size 10,000, average speed-up 11.5* (3 minutes for fitting) 10,000 templates (mean error 2.8) 1 template (mean error 4.2)

  29. Image Segmentation with A Bounding Box Prior ICCV 2009 Victor LempitskyPushmeetKohliCarsten RotherToby Sharp

  30. Motivation GrabCutRother, Kolmogorov & Blake ‘04 Magnetic LassoMortensen & Barret ‘95 Interactive graph cutBoykov & Jolly ‘01 * NP hard (global color model)* very user friendly * Globally optimal* user intensive * Globally optimal* user friendly

  31. Graph cut systems s Graph cut segmentation [Boykov&Jolly 01] integrates cues and input via: p q ... ... t GMMRF&GrabCut [Blake et al. 04, Rother et al.05] adds the Gaussian mixture fitting idea + reiteration s.t. the shape is “sufficiently tight”

  32. When it is not right straight away… Solutions: More interaction High-level semantic knowledge Or just look at the user input more attentively “Tightness” constraint would help!

  33. Problem Formulation Graph cut segmentation [Boykov&Jolly 01] integrates cues and input via: ? Minimize subject to “shape is sufficiently tight” s.t. the shape is “sufficiently tight”

  34. Crossing paths Upper margin crossing path Left margin Right margin Middle box crossing path Lower margin

  35. Shape tightness bounding box middle box Definition: the shape is tight if it intersects all crossing paths

  36. Shape tightness Tight: Not tight: Corollary: the shape is tight if and only ifthe shape has a connected component touching all 4 sides of the middle box

  37. Incorporating Shape Tightness bounding box middle box C2 C1 all 4-connected crossing paths

  38. Incorporating Shape Tightness [ ] Trivial to convert to an LP now! Problems: It’s integer (hence non-convex) It has combinatorial number of constraints Relax! Related work: K. Kolev, D. Cremers: Integration of Multiview Stereo and Silhouettes via Convex Functionals on Convex Domains. ECCV 2008

  39. Solving the Linear Relaxation bounding box middle box 1 • Cannot enforce all constraints • Dijkstra can check if all constraints are satisfied • Dijkstra can find the most violated constraint • Can switch the constraints ongradually 0.25 0 See also: S. Nowozin and C. H. Lampert:Global Connectivity Potentials for Random Field Models. CVPR 2009

  40. Solving the Linear Relaxation Start with no constraints Iterate: Pick a set of violated paths Activate respective constraints Rerun linear optimization Until all constraints satisfied

  41. Solving the Linear Relaxation Image (in the bounding box) unary terms LP solution

  42. How to Round? [ ] Previous works e.g. [Kolev&Cremers’08]: just threshold at low enough value Our work: use the problem structure to perform provably better rounding

  43. Pinpointing Algorithm bounding box bounding box middle box middle box LP solution Rounded solution • Use “dynamic graph cut” [Boykov&Jolly’01],[Kohli&Torr’04]

  44. Pinpointing Algorithm pinpointing, E = -0.54 thresholding, E = -0.46 input bounding box LP solution Corollary: pinpointing always gets a lower (or same) energy solution compared to thresholding.

  45. Pinpointing Algorithm Pinpointing unaries, E= -16.5 Pinpointing LP solution, E= -17.0 Unary terms LP solution Pinpointing can be used as a fast, standalone heuristics.

  46. Quantitative results GMMRF [Blake et al. 04]: Fit Gaussian mixtures to get unary terms Optimize the graph cut energy + a bounding box prior fast fast slow slow fast Relative ordering in terms of the obtained energy is the same

  47. Refining color models GrabCut [Rother et al. 04]: iterate Fitting gaussian mixtures Optimizing the graph cut energy + a bounding box prior The error rate goes down from 5.1% to 3.7%

  48. Conclusion • Global constraints are powerful • Approximate optimization is possible Thank you for your attention!

More Related