POSE–CUT Simultaneous Segmentation and 3D Pose Estimation of Humans using Dynamic Graph Cuts

1. POSE–CUTSimultaneous Segmentation and 3D Pose Estimation of Humans using Dynamic Graph Cuts Mathieu Bray Pushmeet Kohli Philip H.S. Torr Department of Computing Oxford Brookes University

2. Objective Image Segmentation Pose Estimate [Images courtesy: M. Black, L. Sigal]

3. Outline • Image Segmentation Problem • Pose-Specific Segmentation • The Pose Inference Problem • Optimization • Results • Conclusion and Future Work

4. Outline • Image Segmentation Problem • Pose-Specific Segmentation • The Pose Inference Problem • Optimization • Results • Conclusion and Future Work

5. The Image Segmentation Problem Segments Image

6. Problem – MRF Formulation • Notation • Labelling x over the set of pixels • The observed pixel intensity values y (constitute data D) • Energy E (x) = - log Pr(x|D) + constant • Unary term • Likelihood based on colour • Pairwise terms • Prior • Contrast term • Find best labelling x* = arg min E(x)

7. MRF for Image Segmentation xi = {segment1, …, segmentk} for instance {obj, bkg} Pairwise Potential ij(xi, xj) xi x(labels) xj Unary Potential i(D|xi) i j D(pixels) Image Plane

8. Can be solved using graph cuts Unary likelihood Contrast Term Ising Model Maximum a-posteriori (MAP) solution x*= Data (D) Unary likelihood Pair-wise Terms MAP Solution MRF for Image Segmentation

9. MRF for Image Segmentation Need for a human like segmentation Unary likelihood Contrast Term Uniform Prior Maximum-a-posteriori (MAP) solution x*= Data (D) Unary likelihood Pair-wise Terms MAP Solution

10. Outline • Image Segmentation Problem • Pose-Specific Segmentation • The Pose Inference Problem • Optimization • Results • Conclusion and Future Work

11. Shape-Priors and Segmentation OBJ-CUT [Kumar et al., CVPR ’05] • Shape-Prior: Layered Pictorial Structure (LPS) • Learned exemplars for parts of the LPS model • Obtained impressive results = + Spatial Layout (Pairwise Configuration) Layer 1 Layer 2

12. Shape-Priors and Segmentation OBJ-CUT [Kumar et al., CVPR ’05] • Shape-Prior: Layered Pictorial Structure (LPS) • Learned exemplars for parts of the LPS model • Obtained impressive results Unary likelihood colour Shape-Prior Colour + Shape Image

13. Problems in using shape priors • Intra-class variability • Need to learn an enormous exemplar set • Infeasible for complex subjects (Humans) • Multiple Aspects? • Inference of pose parameters

14. Do we really need accurate models? • Interactive Image Segmentation [Boykov & Jolly, ICCV’01] • Rough region cues sufficient • Segmentation boundary can be extracted from edges additional segmentation cues user segmentation cues

15. Do we really need accurate models? • Interactive Image Segmentation • Rough region cues sufficient • Segmentation boundary can be extracted from edges

16. Rough Shape Prior - The Stickman Model • 26 degrees of freedom • Can be rendered extremely efficiently • Over-comes problems of learning a huge exemplar set • Gives accurate segmentation results

17. Pose-specific MRF Formulation  (pose parameters) Unary Potential i(xi|) Pairwise Potential ij(xi, xj) xi x(labels) xj Unary Potential i(D|xi) i j D(pixels) Image Plane

18. Pose-specific MRF Energy to be minimized Pairwise potential Unary term Potts model Shape prior distance transform

19. Pose-specific MRF Energy to be minimized Pairwise potential Unary term Potts model Shape prior + = Colour likelihood colour+ shape MAP Solution Shape Prior Data (D)

20. What is the shape prior? Energy to be minimized Pairwise potential Unary term Potts model Shape prior How to find the value of ө?

21. Outline • Image Segmentation Problem • Pose-Specific Segmentation • The Pose Inference Problem • Optimization • Results • Conclusion and Future Work

22. Formulating the Pose Inference Problem

23. Formulating the Pose Inference Problem

24. Resolving ambiguity using multiple views Pose specific Segmentation Energy

25. Outline • Image Segmentation Problem • Pose-Specific Segmentation • The Pose Inference Problem • Optimization • Results • Conclusion and Future Work

26. Solving the Minimization Problem To solve: Let F(ө) = Minimize F(ө) using Powell Minimization Computational Problem: Each evaluation of F(ө) requires a graph cut to be computed. (computationally expensive!!) BUT.. Solution: Usethe dynamic graph cut algorithm [Kohli&Torr, ICCV 2005]

27. solve SA differences between A and B PB* Simpler problem A and B similar SB Dynamic Graph Cuts PA cheaper operation PB computationally expensive operation

28. solve xa differences between A and B PB* Simpler problem A and B similar Dynamic Graph Cuts 20 msec xb 400 msec

29. Outline • Image Segmentation Problem • Pose-Specific Segmentation • The Pose Inference Problem • Optimization • Results • Conclusion and Future Work

30. Segmentation Results Only Colour Colour + Smoothness Colour + Smoothness + Shape Prior Image [Images courtesy: M. Black, L. Sigal]

31. Segmentation Results - Accuracy

32. Segmentation + Pose inference [Images courtesy: M. Black, L. Sigal]

33. Segmentation + Pose inference [Images courtesy: Vicon]

34. Outline • Image Segmentation Problem • Pose-Specific Segmentation • The Pose Inference Problem • Optimization • Results • Conclusion and Future Work

35. Conclusions • Efficient method for using shape priors for object-specific segmentation • Efficient Inference of pose parameters using dynamic graph cuts • Good segmentation results • Pose inference • Needs further evaluation • Segmentation results could be used for silhouette intersection

36. Future Work • Use dimensionality reduction to reduce the number of pose parameters. • results in less number of pose parameteres to optimize • would speed up inference • Use of features based on texture • Appearance models for individual part of the articulated model (instead of using a single appearance model).

37. Thank You