Human Posture Recognition with Convex Programming

1. Human Posture Recognition with Convex Programming Hao Jiang, Ze-Nian Li and Mark S. Drew School of Computing Science Simon Fraser University Burnaby, BC, V5A 1S6

2. Human Posture Recognition • Recognizing human postures is very important in vision and multimedia. • It has many applications in surveillance, human computer interaction, image and video database analysis and retrieval. • At the same time, recognizing human postures is a hard problem. Simon Fraser University

3. The Challenges of Human Posture Recognition • It is hard to recognize human postures because: • Articulated nature of a human body • No segmentation schemes are available for general images or videos. • Strong background clutters. • Large appearance changes because of clothing • Different schemes have been studied. Simon Fraser University

4. Methods for Posture Recognition • Methods having been studied: • Silhouette based method with background subtraction • Multi-camera based methods • Tracking body movement • Chamfer matching based schemes • Shape context based schemes • These methods are not sufficient to address the problem robustly. Simon Fraser University

5. The Proposed Method • We will present a matching based scheme that has the following properties: • Based on a robust convex (linear) programming matching scheme • Work for cases where no background subtraction is available • Able to deal with strong background clutters • Able to deal with large appearance changes Simon Fraser University

6. Matching Distance Transform Canny Edge Detection Distance Transform Template Generation Template Image Feature Point Selection Delaunay Triangulation Matching With LP Target Image Canny Edge Detection Distance Transform Object Recognition result Simon Fraser University

7. Matching as a Labeling Problem Target p’ p fp Target Clutter q fq Target q’ Template Mesh Target Image Simon Fraser University

8. The Labeling Problem • The matching problem can be formulated as the following optimization problem: Matching cost Smoothing term Simon Fraser University

9. Convex Relaxation • The original problem is a hard non-convex problem. We convert it to LP: c’(s,j) |fp-fq| Simon Fraser University

10. Properties of the Relaxation • For convex problems, LP exactly solves the continuous extension of the original problem. • For general non-convex problems, LP solves the problem where each matching surface is replaced by the lower convex hull. • The “cheapest” basis set for each site corresponds to the lower convex hull’s vertices Simon Fraser University

11. The Effect of Covexification For non-convex problems, the relaxation replaces each c(m,j) by its lower convex hull surface: c(0,j) For site 0 Label Label c(i,j) … Convexification c(M-1,j) For site M-1 Label Label : Lower Convex Hull Vertices :Basic Labels Simon Fraser University

12. Searching Scheme of Simplex Method • Using simplex method, there are at most three adjacent non-zero weight basis labels: Searching for one site : Non-zero-weight basis label : Zero-weight basis label : non basis label : Continuous label Simon Fraser University

13. Successive Relaxation Scheme • Single relaxation may miss the global optimum because of convexification effect • An intuitive scheme is to shrink the trust region and reconvexify the data in the smaller region • This scheme is found to be able to greatly improve the matching results Simon Fraser University

14. The Trust Region Shrinking Simon Fraser University

15. Successive Relaxation Scheme (An Example) min C(1,r1)+ C(2,r2)+0.5|r1-r2| Simon Fraser University

16. Shape Recognition • We have to define the goodness of matching • Matching cost (M): Average difference of the template and target image in the ROI. • Deformation (D): Affine transformation compensated pairwise distance changes • Shape Context in the ROI (C). • Finally, we use M + a*D+b*C to quantify the matching Simon Fraser University

17. Random Dots Experiment Noise: 100% Random Disturbance: 5 Noise: 50% Random Disturbance: 5 Noise: 50% Random Disturbance: 10 Noise: 100% Random Disturbance: 10 Simon Fraser University

18. Matching Synthetic Images Results : LP : ICM : BP : GC (a): Template model showing distance transform; (b): Matching result of proposed scheme; (c): Matching result by GC; (d): Matching result by ICM. (e): Matching result by BP. Simon Fraser University

19. Matching Leaves Simon Fraser University

20. Experiment Results An example where traditional methods fail. (a): Template image; (b): Target image; (c): Edge map of template image; (d): Edge map of target image; (e): Template mesh; (f): Matching result of the proposed scheme; (g): ICM matching result; (h): Sliding template search result. Simon Fraser University

21. Gesture Recognition Results Template Top match Second match Simon Fraser University

22. Gesture Recognition Results Simon Fraser University

23. Video Browsing Result Simon Fraser University

24. Video Browsing Result Simon Fraser University

25. Multiple Target Matching Results Simon Fraser University

26. Conclusion and Future Directions • We present a robust matching framework for human posture recognition • The method can be applied to multimedia data retrieval in image or video database, or human computer interaction applications • In future work: • We will add tempera information for behavior recognition Simon Fraser University

27. Future work • The successive reconvexification is in fact very general. It can be used to increase the robustness of many other matching schemes, such as BP and GC • The proposed matching can be used for many other applications, such as tracking, object recongnition, motion estimation etc. Simon Fraser University

28. The End Simon Fraser University