Dealing with Multiscale Depth Changes and Motion in Depth Edge Detection

1. Dealing with Multiscale Depth Changes and Motion in Depth Edge Detection Rogerio Feris 1, Ramesh Raskar2, Matthew Turk 1 1 University of California, Santa Barbara 2 Mitsubishi Electric Research Labs

2. Motivation • Intensity Discontinuities (Edge Methods) - Limited in their ability to reveal scene structure - Shape boundaries are not captured in low-contrast scenes

3. Motivation • Ideally, we want to detect edges according to their physical origin (discontinuities in depth, surface normal, albedo, illumination and motion) • Our work is focused on the detection and modeling of depth discontinuities, also known as depth edges Why this is an important problem?

4. Motivation Canny Edges Depth Edges Input Image • Depth edges correspond to sharp discontinuities in depth. They outline shape boundaries and are directly tied to the 3D scene geometry. • Useful for many computer vision applications such as segmentation, stereo, and recognition.

5. Motivation • Obvious way to detect depth edges: acquire depth map and then find discontinuities • Depth recovery methods are noisy at discontinuities Can we find depth edges without full 3D reconstruction?

6. Depth Edges with MultiFlash Raskar, Tan, Feris, Yu, Turk – ACM SIGGRAPH 2004

11. Shadow-Free Depth Edges Shadow-Free Bottom Flash Top Flash Left Flash Right Flash Ratio images and directions of epipolar traversal Depth Edges

12. Limitations • Outdoor Scenes • Specular Reflections • Transparent or Low albedo surfaces • Thin narrow objects • Lack of Background • Dynamic Scenes

13. Varying Illumination Parameters

14. Multiscale Depth Changes Small Baseline Shadow Missed Large Baseline Shadow Detaches Baseline Tradeoff

15. A MultiBaseline Approach

16. Multiscale Depth Changes Small Baseline Shadow Missed Large Baseline Shadow Detaches Minimum Image

17. Small Baseline Our Final Result Large Baseline

18. Input Image Canny Edges

19. Single Baseline Our MultiBaseline Approach

20. Multiscale Depth Changes Drawbacks: • Slow acquisition time • Detached shadows caused by small baseline flash Solution: Linear Light Sources Linear Light

21. Multiscale Depth Changes Image Capture Ratio Right Flash Ratio Bottom Flash Depth Edges

22. Multiscale Depth Changes Linear Light Drawbacks: • More sensitive to noise • Depth edges lying in shadowed regions

23. NoFlash Flash No Background

24. NoFlash Flash Flash NoFlash

25. Varying Wavelength • Light sources are triggered at the same time! • Colored shadows are explored to detect depth edges

26. Varying Wavelength White Light Colored Light

27. Varying Wavelength Colored Light / White Light Colored Light

28. Varying Wavelength • Depth edges from a single image Learning Shadow Color Transitions Depth Edges with variable Wavelength Canny Edges

30. Variation of illumination parameters (spatial position, number, type, and wavelength of light sources) for robust depth edge detection Conclusions • - A novel multibaseline approach for multiscale depth edge detection • - A novel method to detect depth edges in motion with variable wavelength light sources

31. Limitations • Limited to indoor scenes • Depth edges in motion with one-shot photography only for specific applications, not for general scenes. Future Work • Extend this framework to detect other physical discontinuities (surface normal, albedo, illumination, and motion)

32. Thank you !