Nalin Pradeep Senthamil Masters Student, ECE Dept.

1. Nalin Pradeep Senthamil Masters Student, ECE Dept. Advisor, Dr Stan Birchfield Committee Members, Dr Adam Hoover, Dr Brian Dean

2. Accurate Tracking of Non-Rigid Objects using Level Sets Clemson University, Clemson, SC USA Accepted in ICCV, 2009

3. Outline • Tracking Overview • Literature • Proposed Approach • Object Fragmentation • Region Growing Mechanism • GMM modeling (feature-spatial) • Level Set Framework • Fragment Motion using Joint-KLT • Results • Conclusion

4. Tracking Overview • Idea: Obtain Trajectories over time to locate object • Three Main Categories • Point Tracking – Kalman, Particle filters • Kernel Tracking – Collins et al (linear RGB), Comaniciu (Mean-Shift) • Contour Tracking – Shah et al, Cremers et al • Applied to Surveillance – Vessel, human, vehicle etc • Why not internet videos ? – 65,000 videos get uploaded in YouTube everyday (rich market)

5. Literature • Linear RGB [Collins et al. 2003] • Ada-boost classifier [Avidan 2005] • Fragments based fixed size [Adam et al. 2006] • Key-point Feature learning [Grabner et al. 2007] • Shape priors [Cremers et al. 2006] • Contour tracking using texture [Shah et al 2005] • Limitations • Ignore secondary cues such as multimodality • Lack in determining accurate object shape • Usually non-contour based techniques drift during occlusion • Often ignore spatial arrangement of pixels

6. Tracker Initialization User clicked ROI around object Object Fragmentation Each object as set of fragments Object Modeling Strength Map Computation Level-Set Formulation Estimate Fragment Motion Algorithm Block Diagram Update made at each frame

7. Object Fragmentation Region Growing Mechanism • Random pixel selected from mask – fragment (f) • Neighboring pixels added to (f) within Γ (std deviation) • Gaussian Model of (f) updated • Each (f) represents a Gaussian ellipsoid • Both Object and background are fragmented

8. Object Modeling (GMM) Joint feature-spatial space,

9. Strength Map +ve for FGND -ve for BKGND

10. Level Set Framework • Level Set is numerical technique for fitting contour • Level Set on 2D image is viewed as 3D function • Contour in level set identified at zero level

11. Level Set for strength map speed contour • In general, Level set evolution defined by • Gradient Descent Iteration Strength Image Contour (zero level set) Strength Image Divergence operator

12. Iterations using “Elmo” strength map Curve can grow inward and outward Figure shows for first frame as example Curve evolves from previous contours in subsequent tracking Level-Set Evolution

13. Fragment Motion • Joint-KLT: Combines algorithms of KLT and HS • Hence, • Used to align coordinate system of object and model fragments • Increases accuracy of strength map data term smoothness term

14. Fragment Motion (contd.) • ‘N’ features tracked in each fragment are averaged • Motion of each fragment gives ‘prior’ information before computing strength map • Drastic motion can be addressed KLT Joint-KLT

15. Results - Videos

16. Shape Matching • Hausdorff metric is mathematical measure to compare two sets of points • Application in Occlusion Handling and Shape recognition ‘a’ and ‘b’ are two point sets

17. Occlusion Handling • Rate of decrease in object size determines occlusion • Contour shapes learnt online is used to hallucinate during occlusion • Best shape is identified using Hausdorff distance metric • Previously learnt subsequent shapes are hallucinated during occlusion

18. Results – Occlusion Videos

19. Results – More Comparison Videos

20. Quantitative Comparison Walk Behind Elmo Doll Girl Circle Average Normalized error obtained against ground-truth of sequences at every 5 frames.

21. Conclusion • Tracking algorithm based on modeling object and background with mixture of Gaussians • Simple and efficient region growing mechanism to achieve fast computation • Embedding “strength map” into Level-Set Framework • Joint KLT introduced in the framework to improve accuracy • Future Work: • Robust shape prior learning and matching • Self-occlusion handling for unknown fragments

22. Alternative Tracking Framework (outline) • Overview • Proposed Approach • Vessel Detection • Saliency Map • Thresholding • Vessel Tracking • Strength Map using Linear RGB • ML Framework for Search • Results

23. Object Detection Using Saliency Map • Saliency: Property of objects standing out relative to their neighbors. • There is a statistical relationship between backgrounds of all natural images similar to pre-attentive search done by human visual system. • Zhang et al (CVPR 2007) observed redundancies in log Fourier spectra of natural images. Hence, any statistical singularities in the spectrum can be treated as anomalies.

24. Smoothing in frequency domain Smoothing in spatial domain Saliency Map Computation • Algorithm • Letbe the image. • Real part of Fourier Spectrum • Phase • Log Spectrum • Spectral Residual • Saliency Map , j=sqrt(-1)

25. Sample Saliency Map detections

26. Object Tracking • Objects detected through saliency used as FGND • Immediate surrounding used as BKGND • Strength Model Computed similar to Collins Linear RGB • 49 features selected from linear combination used to identify strength map • Maximum Likelihood Framework based search used to localize objects in each frame • Region search was identified based on object velocity

27. Object Tracking – Strength Model hist-index probability Small value – 0.01 Variance of L(i) with respect to a distribution a(i) • 49 features of RGB are normalized into 0-255 and discretized into 0-32 histogram bins • For each feature, • Variance Ratio of Log-likelihood is identified that best discriminates object from background

28. Strength Model - Outputs

29. Object Tracking – ML Framework mean Covariance Strength Map Prevent pixel locations farther from object • Objective was to recover tight bound around object • ML Framework is like EM algorithm • Search objective is to maximize the function (Mean, Covariance)

30. Object Tracking – ML Framework Mean and covariance of current estimate • To maximize the function, Mean and Covariance are computed iteratively • E-Step • M-Step Iterated for 2-3 times to get optimal values

31. Conclusion • Algorithm was real time and supported around 25-30 fps in speed • Saliency map based detection was introduced • Concept of “strength map” from adaptive-fragmentation is applied here • Depends only on color (linearRGB), and combination with KLT features would add robustness to the system. Good way to combine is explored.

32. Thank you !