Active Range Imaging Datasets for Indoor Surveillance

1. Active Range Imaging Datasets for Indoor Surveillance C. Distante, G. Diraco, A. Leone Institute for Microelectronics and Microsystems – CNR, Lecce (Italy)

2. Introduction • Outline of active range vision • □ Range imaging technologies • □ Properties of Time-Of-Flight range sensors • Active range vision Vs Passive vision □ Comparison between TOF camera and stereo vision □ Advantages and drawbacks in surveillance contexts • Datasets for indoor surveillance • Case study • □ TOF sensor-based fall detection • Conclusions

3. In the last years several active range sensors have been presented (Canesta Inc., Mesa Imaging AG, 3DV Systems Ltd, …). The ability to describe scenes in three dimensions opens new scenarios, providing new opportunities in different applications, including visual monitoring (object detection, tracking, recognition, image understanding), security, biometrics, automotive, robotics, medical imaging, … Active range sensors provide depth information allowing to use algorithms much less complex and allows problems to be approached in a new, robust and cost-efficient way. Datasets are presented in order to suggest a common basis for comparative analysis of vision algorithms. • Introduction • Outline of active range vision • Active range vision Vs Passive vision • Datasets for indoor surveillance • Case study • Conclusions

4. Range Imaging (RIM) is the fusion of two different technologies, integrating depth measurement and imaging aspects. It’s a new measurement technique, not yet well-known and investigated. • Introduction • Outline of active range vision • Active range vision Vs Passive vision • Datasets for indoor surveillance • Case study • Conclusions Depth Measurement Techniques Taxonomy Contactless depth measurement Triangulation (stereoscopy, structured light - sub-millimeter resolution) Interferometry (light source scanning - sub-micrometer resolution) Time-Of-Flight(millimeter resolution) Pulse (Direct measure) Continuous wave modulation(Indirect measure, eye-safe)

5. Intensity image Depth image High distance Low distance Principles of phase shift modulation-based TOF sensors • Introduction • Outline of active range vision • Active range vision Vs Passive vision • Datasets for indoor surveillance • Case study • Conclusions The depth estimation is realized by measuring the phase shift φ of the signal round-trip from the device to the target and back.

6. Main features of modulation-based TOF sensors • standard CMOS technology • high frame rate (up to 30 fps) • fairly good spatial resolution (up to QCIF @ 176x144 pixels) • fairly good field of view (up to 80x80 degrees) • aliasing effects (non-ambiguity range up to 30 meters) • low depth measurement error (< 1% in non-ambiguity range) • direct Cartesian coordinate output (x, y, z) for 3D reconstruction • built-in band-pass optics for background light suppression • illumination power less than 1W (LED array, Class 1 for eye-safe) • Introduction • Outline of active range vision • Active range vision Vs Passive vision • Datasets for indoor surveillance • Case study • Conclusions • Two critical parameters affect performances: • modulation frequency (it is a design parameter that mainly affects the non-ambiguity range) • integration time (it could be adjusted and affects depth resolution and frame rate)

7. Comparison of the most important characteristics of TOF cameras and stereo vision systems • Introduction • Outline of active range vision • Active range vision Vs Passive vision • Datasets for indoor surveillance • Case study • Conclusions

8. Advantages in the use of TOF sensors in surveillance contexts • Introduction • Outline of active range vision • Active range vision Vs Passive vision • Datasets for indoor surveillance • Case study • Conclusions

9. Drawbacks in the use of TOF sensors in surveillance contexts • Introduction • Outline of active range vision • Active range vision Vs Passive vision • Datasets for indoor surveillance • Case study • Conclusions

10. Datasets description • Each sequence is acquired at QCIF resolution by a state-of-the-art TOF sensor (MESA SR-3000) and it is composed by 1800 frames captured at variable frame rate (by varying integration time) • Introduction • Outline of active range vision • Active range vision Vs Passive vision • Datasets for indoor surveillance • Case study • Conclusions • Sequences have been acquired in wall/ceiling-mounting configurations at different subject orientations, in presence/absence of occlusions, in order to cover a large amount of events • Extrinsic parameters are available for calibration purpose (camera-floor distance, camera orientation, scene depth) • Sequences present people (one/more persons in the scene) having different postures (stand, sit, lay down, bent, squat) • In the sequences persons have different behaviours (walking, falling down, moving objects, picking objects, limping) • Datasets are at http://siplab.le.imm.cnr.it

11. Datasets description • A generic frame (176x144 pixels) of each sequence presents the following structure: • Raw data • Depth image (2Bytes/pixel – unsigned integer) • Intensity image (2Bytes/pixel – unsigned integer) • FPGA processed data • Depth image with noise reduction (2Bytes/pixel – unsigned integer) • Intensity image with noise reduction (2Bytes/pixel – unsigned integer) • Cartesian x coordinate (4Bytes/pixel – signed float) • Cartesian y coordinate (4Bytes/pixel – signed float) • Cartesian z coordinate (4Bytes/pixel – signed float) • Introduction • Outline of active range vision • Active range vision Vs Passive vision • Datasets for indoor surveillance • Case study • Conclusions For each frame a great amount of information is defined (495KBytes)!

12. 7.5 meters 0.5 meters Aliasing and multi-path effects Noise is due to high-reflective objects (LCD TV) • Introduction • Outline of active range vision • Active range vision Vs Passive vision • Datasets for indoor surveillance • Case study • Conclusions Intensity image Depth image Fluctuations are due to the continuously adjusted emitted light power

13. Introduction • Outline of active range vision • Active range vision Vs Passive vision • Datasets for indoor surveillance • Case study • Conclusions Intensity image Depth image Could the depth information help the tracking in the presence of total occlusions?

14. Introduction • Outline of active range vision • Active range vision Vs Passive vision • Datasets for indoor surveillance • Case study • Conclusions Intensity image Depth image

15. TOF sensor-based fall detection • Introduction • Outline of active range vision • Active range vision Vs Passive vision • Datasets for indoor surveillance • Case study • Conclusions Gaussians mixture for background modelling Bayesian approach for segmentation

16. In indoor surveillance applications, range images provide a better perception of scenes in all illumination conditions, deterring the use of cheap stereo systems that fail in dark or low-textured environments. If critical parameters of TOF sensor are adjusted, reliable, computationally low-cost and real-time segmentation/tracking can be realized by only using depth measure, since intensity images present unwanted fluctuations. Depth information overcomes projective ambiguity, whereas intensity image provides appearance information, so that the joined use of them improves critical steps (object recognition, behavior analysis, …) allowing a better description of moving objects. The suggested datasets provide common basis to investigate vision algorithms; they can be improved by defining ground-truth data to quantify performances. • Introduction • Outline of active range vision • Active range vision Vs Passive vision • Datasets for indoor surveillance • Case study • Conclusions

17. THANK YOU FOR YOUR ATTENTION