Reconstruction with Depth and Color cameras for 3D Autostereoscopic Consumer Displays

1. Reconstruction with Depth and Color cameras for 3D Autostereoscopic Consumer Displays SAIT – INRIA collaboration Period: 15 July 2012 / 15 January 2013 Date: 3-4 December 2012

2. INRIA team • Georgios Evangelidis, postdoc, 100% • Michel Amat, development engineer, 100% • Soraya Arias, senior development engineer, 20% • Jan Cech, posdoc, 20% • Radu Horaud, 10%

3. Past achievements • A method and software for aligning TOF data with a stereoscopic camera pair • Extension to the calibration of several TOF-stereo units • 3D texture-based rendering of the TOF data using the color-image information

4. Publications • One CVPR 2011 paper • A tutorial at ICIP 2011 • One Springer Briefs book just published • The two teams published several other papers

5. Current achievements • Finalization of the calibration & rectification methods/software • TOF to stereo-pair mapping with filtering • TOF + texture in live mode • Disparity map initialization • Stereo correspondence based on seed-growing • Final high-resolution depth map with gap filling • A paper submitted to CVPR’13

6. Improved Calibration • New calibration board • mat sticker glued to a rigid plane • plane attached to a tripod • Refined Calibration algorithm • TOF-Stereo Calibration error: <1.5 pixel • Improved Rectification • Rectification error: <0.25 pixel

7. Given a calibrated TOF-Stereo system • Each TOF point PT defines a correspondence between PL and PR

8. Correspondences (samples) obtained by using the calibration parameters • each correspondence comes from a TOF point • different color -> different depth

9. Correspondences (samples) obtained by using the calibration parameters • each correspondence comes from a TOF point • different color -> different depth

10. TOF-to-Left Mapping • We use the left image as reference

11. TOF-to-Left Mapping is not perfect Resolution mismatch

12. TOF-to-Left Mapping is not perfect Left-to-Tof Occlusions Left-to-Tof Occlusions: the depth decreases from left to right

13. TOF-to-Left Mapping is not perfect Tof-to-Left Occlusions Tof-to-Left Occlusions: the depth increases from left to right

14. Point Cloud filtering • We reject points in left-to-tof occluded area • We keep the minimum-depth points in case of overlap (due to Tof-to-left occlusions)

15. Disparity Map: Initialization • Run Delauney-Triangulation on low-resolution point cloud

16. Disparity Map: Initialization • Run Delauney-Triangulation on low-resolution point cloud… • …and initialize the stereo disparity map It looks good, but it’s noisy and non-accurate!

17. Seed-Growing Idea • Start from points with known disparities (seeds) and propagate the disparity to neighboring points (video?) • Main issues: • What are our seeds? • What is the visiting order of seeds? • How do I propagate the message? • How the stereo and depth data are fused within this framework?

18. Depth-Color Fusion N d • Built on the seed-growing idea • A:Depth data, S: Stereo data, dN : neighbor of d • For each pixel (node), find its disparity value that maximizes the posterior probability (MAP) A S Range-search constraint Penalize the choice wrt to depth information d Penalize the choice wrt to color information Input data Pixel with unknown disparity A represents the initial estimation of d (obtained by the previous interpolation) S represents the color matching cost that corresponds to d Pixel with known disparity

19. Depth-Color Fusion N d A • Bayes rule translates each posterior into a likelihood • If likelihood terms are chosen from the exponential family, the “-log”-ness translates MAP into an energy minimization scheme S Because of the Bayes rule Because of the uniform distribution d We are currently working on these terms! Input data Pixel with unknown disparity Pixel with known disparity

20. Seed-Growing Idea (revisited) • For each pixel, an energy function is defined and we look for its minimizer (disparity) • Main issues: • What are our seeds? • the points from Tof-to-Left mapping after refinement • What is the visiting order of seeds? • First visit reliable seeds (points with low energy value) • How do we propagate the message? • Given the disparity of a seed, bound the disparity-range for its neighbor • How the stereo and depth data are fused within this framework? • Described above

21. Examples White areas: unreliable matches Black areas: Occlusions

22. Examples (with gap filling)

23. Examples (with gap filling)

24. Examples (with gap filling)

25. Examples (with gap filling)

26. Paper Submission • Stereo-Depth Fusion for High-Resolution Disparity Maps. G. Evangelidis, R. Horaud, M. Amat, and S. Lee – submitted to CVPR 2013. • An extended version of the CVPR submission is under preparation and it will be submitted to IEEE TPAMI in January/February 2013.

27. Work during the remaining month • Improve the accuracy of the matching by better exploiting the color/texture information • Currently the software implementation runs in offline-mode: We will provide a live-mode version at approximatively 1-2 frames/second • An updated version will be available at the end of the period (~15 January 2013)

28. Prospects for the next collaboration(1 February 2013 – 31 January 2014) • Finalize the TOF-stereo seed-growing algorithm, in particular improve the performance in non-textured areas • Depth disambiguation using TOF-TOF and TOF-stereo • Combine depth disambiguation with the seed-growing algorithm • Perform full 3D realistic rendering with four TOF-stereo units • Perform continuous 3D reconstruction with a moving TOF-stereo unit