1. Using Strong Shape Priors for Multiview Reconstruction Yunda Sun Pushmeet Kohli Mathieu Bray Philip HS Torr Department of Computing Oxford Brookes University

2. Objective Images Silhouettes Parametric Model + Pose Estimate Reconstruction [Images Courtesy: M. Black, L. Sigal]

3. Outline • Multi-view Reconstruction • Shape Models as Strong Priors • Object Specific MRF • Pose Estimation • Results

4. Outline • Multi-view Reconstruction • Shape Models as Strong Priors • Object Specific MRF • Pose Estimation • Results

5. Multiview Reconstruction Need for Shape Priors

6. Multiview Reconstruction • No Priors • Silhouette Intersection • Space Carving • Weak Priors • Surface smoothness • Snow et al. CVPR ’00 • Photo consistency and smoothness • Kolmogorov and Zabih [ECCV ’02] • Vogiatzis et al. [CVPR ’05] [Image Courtesy: Vogiatzis et al.]

7. Outline • Multi-view Reconstruction • Shape Models as Strong Priors • Object Specific MRF • Pose Estimation • Results

8. Shape-Priors for Segmentation • OBJ-CUT [Kumar et al., CVPR ’05] • Integrate Shape Priors in a MRF • POSE-CUT [Bray et al., ECCV ’06] • Efficient Inference of Model Parameters

9. Parametric Object Models as Strong Priors • Layered Pictorial Structures • Articulated Models • Deformable Models

10. Outline • Multi-view Reconstruction • Shape Models as Strong Priors • Object Specific MRF • Pose Estimation and Reconstruction • Results

11. Object-Specific MRF

12. Object-Specific MRF Energy Function Shape Prior Unary Likelihood Smoothness Prior x:Voxel label θ: Model Shape

13. Object-Specific MRF Shape Prior : shortest distance of voxel i from the rendered model x:Voxel label θ: Model Shape

14. Object-Specific MRF Smoothness Prior Potts Model x:Voxel label θ: Model Shape

15. Object-Specific MRF Unary Likelihood For a soft constraint we use a large constant K instead of infinity x:Voxel label θ: Model Shape : Visual Hull

16. Object-Specific MRF Energy Function Shape Prior Unary Likelihood Smoothness Prior Can be solved using Graph cuts [Kolmogorov and Zabih, ECCV02 ]

17. Object-Specific MRF Energy Function Shape Prior Unary Likelihood Smoothness Prior How to find the optimal Pose?

18. Outline • Multi-view Reconstruction • Shape Models as Strong Priors • Object Specific MRF • Pose Estimation • Results

19. Inference of Pose Parameters Rotation and Translation of Torso in X axes Rotation of left shoulder in X and Z axes

20. Inference of Pose Parameters Let F(ө) = Minimize F(ө) using Powell Minimization Computational Problem: Each evaluation of F(ө) requires a graph cut to be computed. (computationally expensive!!) BUT.. Solution: Usethe dynamic graph cut algorithm [Kohli&Torr, ICCV 2005]

21. Outline • Multi-view Reconstruction • Shape Models as Strong Priors • Object Specific MRF • Pose Estimation • Results

22. Experiments • Deformable Models • Articulated Models • Reconstruction Results • Human Pose Estimation

23. Deformable Models Visual Hull • Four Cameras • 1.5 x 105 voxels • DOF of Model: 5 Our Reconstruction Shape Model

24. Articulated Models

25. Articulated Models • Four Cameras • 106 voxels • DOF of Model: 26 Camera Setup Shape Model

26. Articulated Models • 500 function evaluations of F(θ) required • Time per evaluation: 0.15 sec • Total time: 75 sec Let F(ө) =

27. Articulated Models Visual Hull Our Reconstruction

28. Pose Estimation Results Visual Hull Reconstruction Pose Estimate

29. Pose Estimation Results • Quantitative Results • 6 uniformly distributed cameras • 12 degree (RMS) error over 21 joint angles

30. Pose Estimation Results • Qualitative Results

31. Pose Estimation Results Video 1, Camera 1

32. Pose Estimation Results Video 1, Camera 2

33. Pose Estimation Results Video 2, Camera 1

34. Pose Estimation Results Video 2, Camera 2

35. Future Work • Use dimensionality reduction to reduce the number of pose parameters. • results in less number of pose parameters to optimize • would speed up inference • High resolution reconstruction by a coarse to fine strategy • Parameter Learning in Object Specific MRF

36. Thank You

37. Object-Specific MRF Energy Function Shape Prior Unary Likelihood Smoothness Prior +