Graph Cut Algorithms for Binocular Stereo with Occlusions

1. Graph Cut Algorithms for Binocular Stereo with Occlusions Vladimir Kolmogorov, Ramin Zabih

2. Overview: • Traditional Stereo Methods • Energy Minimization via Graph Cuts • Stereo with Occlusions • Voxel Labeling Algorithm • Pixel Labeling Algorithm • Results and Conclusions Graph Cut Algorithms for Binocular Stereo with Occlusions Math Basics for Vision and Graphis 2007

3. Traditional Stereo Methods Traditional Stereo Problem pixel correspondences  labeling (disparity) Graph Cut Algorithms for Binocular Stereo with Occlusions Math Basics for Vision and Graphis 2007

4. Traditional Stereo MethodsDisparity disparity depth  disparity ~ depth ground truth disparity Graph Cut Algorithms for Binocular Stereo with Occlusions Math Basics for Vision and Graphis 2007

5. Traditional Stereo MethodsBinocular Stereo • goal is to compute pixels correspondences • traditional stereo problem  pixel labeling problem • advantage: can be solved by graph cuts • problem is formulated as energy term • new goal: find the minimizing labeling Graph Cut Algorithms for Binocular Stereo with Occlusions Math Basics for Vision and Graphis 2007

6. we assign the label to pixel p when p of image I corresponds to p + in I‘ Traditional Stereo MethodsEnergy Function find labeling that minimizes cost for assigning labels smoothness term Graph Cut Algorithms for Binocular Stereo with Occlusions Math Basics for Vision and Graphis 2007

7. Traditional Stereo MethodsEnergy Function • data cost – gives penalty for different intensities • smoothness term – gives penalty for discontinuities (Potts model) other models: absolute distance quadratic Graph Cut Algorithms for Binocular Stereo with Occlusions Math Basics for Vision and Graphis 2007

8. Energy Minimization via Graph Cuts Max-flow / Min-Cut (Ford and Fulkerson Algorithm, Push-Relabel Method) Graph Cut Algorithms for Binocular Stereo with Occlusions Math Basics for Vision and Graphis 2007

9. α-expansion α-β-swap Initial Labeling Energy Minimization via Graph Cuts • convex V vs. metric / semimetric • α-β-swap move • α-expansion move: assigning label α to an arbitrary set of pixels Graph Cut Algorithms for Binocular Stereo with Occlusions Math Basics for Vision and Graphis 2007

10. Stereo with Occlusions Graph Cut Algorithms for Binocular Stereo with Occlusions Math Basics for Vision and Graphis 2007

11. Stereo with Occlusions • treat input symmetrically • scene elements only visible in single view • physically correct scenes  geometric constraints  occlusions  physically possible labelings • introduce constraints in the problem formulation • graph cuts perform unconstrained energy minimization Graph Cut Algorithms for Binocular Stereo with Occlusions Math Basics for Vision and Graphis 2007

12. Voxel Labeling Algorithm • discrete scene of voxels • voxel v is active when visible from both cameras • uniqueness constraint – 1:1 correspondence of pixels Graph Cut Algorithms for Binocular Stereo with Occlusions Math Basics for Vision and Graphis 2007

13. Voxel Labeling AlgorithmEnergy Function smoothness term (Potts model) matching penalty (only active voxels) occlusion penalty set of occluded pixels Graph Cut Algorithms for Binocular Stereo with Occlusions Math Basics for Vision and Graphis 2007

14. Pixel Labeling AlgorithmEnergy Function like traditional stereo but for both images e.g. Potts model active ? Graph Cut Algorithms for Binocular Stereo with Occlusions Math Basics for Vision and Graphis 2007

15. Minimizing the Energy • convert constrained into unconstrained minimization problem • write as sum over pairs • form of energy function = standard stereo problem • minimization with α-expansion algorithm • modified definition of α-expansion move for voxel labeling (0=valid, else ∞) uniqueness Graph Cut Algorithms for Binocular Stereo with Occlusions Math Basics for Vision and Graphis 2007

16. traditional s.p. voxel labeling pixel labeling Results and Conclusions ground truth Tsukuba ref. image Graph Cut Algorithms for Binocular Stereo with Occlusions Math Basics for Vision and Graphis 2007

17. Results and Conclusions • efficient energy minimization polynominal time instead of exponential time • traditional stereo algorithm is faster • pixel labeling better than voxel labeling: • prohibits ‚holes‘ in the scene • allows to use other effective smoothness terms • algorithms can be extended for multiple cameras Graph Cut Algorithms for Binocular Stereo with Occlusions Math Basics for Vision and Graphis 2007

18. Multi-view Stereo via Volumetric Graph Cuts Graph Cut Algorithms for Binocular Stereo with Occlusions Math Basics for Vision and Graphis 2007

19. Recent Work Graph-cut-based stereo matching using image segmentation with symmetrical treatment of occlusions, 2006 TUW Graph Cut Algorithms for Binocular Stereo with Occlusions Math Basics for Vision and Graphis 2007

20. Questions? Graph Cut Algorithms for Binocular Stereo with Occlusions Math Basics for Vision and Graphis 2007

21. References • M. Bleyer, M. Gelautz, „Graph-cut-based stereo matching using image segmentation with symmetrical treatment of occlusions“, 2007 • Y. Boykov, O. Veksler, R. Zabih, „Fast Approximate Energy Minimization via Graph Cuts“, 2001 • V. Kolmogorov, R. Zabih, „Graph Cut Algorithms for Binocular Stereo with Occlusions“,2005 • V. Kolmogorov, R. Zabih, „What energy functions can be minimized via graph cuts“, 2004 • V. Kolmogorov, R. Zabih, „Generalized multi-camera scene reconstruction using graph cuts“, July 2003 • V. Kolmogorov, R. Zabih, „Multi-camera Scene Reconstruction via Graph Cuts“, 2002 • S. Seits, C. Dyer, „Photorealistic Scene Reconstruction by Voxel Coloring“, 1997 • R.Szeliski, R. Zabih, „An Experimental Comparison of Stereo Algorithms“, 1999 Graph Cut Algorithms for Binocular Stereo with Occlusions Math Basics for Vision and Graphis 2007