Speaker: Srivishnu ( Kaushik ) Satyavolu Advisor: Dr. Pete Willemsen

1. Real Time Visual Body Feedback & IR Tracking in HMD Based Virtual Environments Using Microsoft Kinects Speaker: Srivishnu ( Kaushik ) Satyavolu Advisor: Dr. Pete Willemsen

2. Analysis of using multiple kinects to implement an inexpensive IR based tracking system, in addition to real time 3D visual body feedback and natural interaction to the user in HMD based Virtual Environments. Thesis Problem

4. Motivation • Absence of real time ( self ) visual body feedback in existing HMD based VEs • Difficulties to study the perceptions / interaction of multiple users simultaneously in HMD based VEs • Full-body Tracking possible only by highly expensive tracking systems & Motion Capture suits

5. Initial Approach Using OpenCV Issues???

6. Microsoft Xbox Kinect

9. Issues • Camera Calibration • RGB Depth Mapping • 3D User segmentation, smoothing & representation • IR Interference between multiple kinects • Relative Orientation between multiple Kinects • Tracking User Positions and/or pose

10. Camera Calibration & RGB Depth Mapping Issues: Intrinsic Camera Calibration Incorrect mapping between RGB and Depth images Why is this interesting?: Accurate RGB to Depth Mapping Solution: OpenCV Camera Calibration using Standard Chessboard Recognition Techniques

14. 3D User Representation Issue: 3D Point Cloud representation Why is this interesting?: Real time Visual body feedback to the user Approaches used: Triangulation, RGB/ IR / Depth Background Subtraction, Basic Filtering Techniques

15. IR Interference Issue: Interference of IR between multiple kinects Why is this interesting?: For many reasons, which you will see shortly Solution: Not as bad as expected Unresolved at the moment

20. Relative Orientation between multiple kinects Issue: Extrinsic Camera Calibration Why is this Interesting?: Relative Orientation of Multiple Kinects Solutions: Manual Orientation Automatic Calibration using OpenCV’s Chessboard Recognition Techniques Others?

21. User Tracking Issues: Tracking User Position, Orientation & Pose across large VR Lab spaces Why is it Interesting?: Real time 3D Natural Interaction by User across VR Lab spaces Solution: IR Based Kinect Tracking System?

22. IR based Kinect Tracking System • What is it for? • Track User Head Position and orientation ( possibly ) in VR Lab Spaces • How is it done? • Tracking an IR marker across a VR Space ( how large??? ) • Why do we need it? • Interference issues ( again?? ) with existing Tracking Sytems like World Viz etc. • Extending OpenNI and Microsoft SDK pose estimation techniques( effects of interference?? )

25. Issues • External Interference ( and again??? ) • Marker visibility • Jitter / Noise

27. Current Approach • Principle of locality • Multiple Kinects ( interference??? ) over the network • Mean Filter • Interpolation???

28. Experiments & Results Experiment #1: Evaluation of noise/jitter of kinect's IR based position tracking over a large space Results: Standard Deviation: 0.001 – 0.003 % of the mean

33. Future Scope & Conclusion • Photorealisitc and real time 3D Self Representation • 3D Visual Body Feedback to the User • Real time 3D Natural Interaction of the User across VR lab spaces • Kinect Based IR tracking System in conjunction with Microsoft / OpenNI pose estimation techniques • Increased perception and level of presence in VEs • Yet to be verified, but it's quite possible • Better understanding of HMD based VEs with multiple users

34. References • Libfreenect • Nicholas Burrus’ work ( RGBDemov0.4) • Oliver Kreylos’ Kinect Viewer • OpenCV • And all the other websites from which I downloaded some of the images used in this presentation

35. THANK YOU

36. Questions?