Depth from Diffusion

1. Depth from Diffusion Changyin Zhou Oliver Cossairt Shree Nayar Columbia University Supported by ONR

2. Optical Diffuser

3. Optical Diffuser ~ 10 micron Micrograph of a Holographic Diffuser (RPC Photonics) [Gray, 1978] [Chang et al., 2006] [Garcia-Guerrero et al. 2007]

4. Diffuser to preview the image (B&H) Diffusers for illumination (B&H) Diffusers to soften the image Diffusers as Accessories

5. Object Object Diffuser Camera Diffuser Camera Diffusion Encodes Depth The amount of diffusion varies with depth.

6. Geometry of Diffusion: A Pinhole Camera Object P Miss Pinhole Q Sensor

7. θ Geometry of Diffusion: A Pinhole Camera Object P Pinhole Sensor Diffuser

8. A θ B θ Geometry of Diffusion: A Pinhole Camera Object P Pinhole Sensor Diffuser

9. Geometry of Diffusion: A Pinhole Camera Diffusion Law: Object A P θ B θ Pinhole O 2r V Z U Sensor Diffuser Object

10. Geometry of Diffusion: A Pinhole Camera Diffusion Size and Depth: Object A P θ B θ Pinhole O 2r V Z U Sensor Diffuser Object

11. Geometry of Diffusion: A Pinhole Camera Diffuser as a proxy object Diffusion Size and Depth: Z P Pinhole O 2r V U Sensor Diffuser Object

12. Captured Image Diffusion PSF Latent clear image Diffusion Size Diffusion as Convolution: A Pinhole Camera Assume field angle and depth are constant for small image patches, we have:

13. Geometry of Diffusion: A Lens Camera

14. Geometry of Diffusion: A Lens Camera Diffuser as a proxy object Z P Pinhole O 2r V U Sensor Diffuser Object

15. Geometry of Diffusion: A Lens Camera Diffuser as a proxy object The captured image can be further blurred due to defocus. Z P Lens 2r V U Sensor Diffuser Object

16. is the diffusion PSF if a pinhole were used. is the defocus PSF if the diffuser were removed. Diffusion as Convolution: A Lens Camera For a lens camera with a diffuser, we have: The Final PSF Diffusion PSF Defocus PSF

17. 1. Capture Two Images With a diffuser Without a diffuser 2. Estimate Blur Size r 3. Compute Depth Z Depth from Diffusion (DFDiff) Algorithm Same form as in DFD

18. P Depth from Defocus r Z Sensor Focal Plane Lens Diffusion pattern Depth from Diffusion P θ θ r Pinhole Z Diffuser Sensor Aperture pattern Depth from Diffusion vs. Depth from Defocus [Pentland, 1987] [Subbarao, 1988] [Watanabe & Nayar, 1996] [Chaudhuri & Rajagopalan, 1999] [Favaro & Soatto, 2005] [Schechner & Kiryati, 2000]

19. A Diffuser of 21.8o Field of View Depth from Diffusion vs. Depth from Defocus Depth from Diffusion Suppose 22.5x15mm Sensor, 10 um pixel, 100 mm EFL Object Any lens is fine! P Object distance = 1000 mm Depth precision is about 0.1mm.

20. Depth from Diffusion vs. Depth from Defocus Depth from Defocus Suppose 22.5x15mm Sensor, 10 um pixel, 100 mm EFL Object Lens Aperture diameter ? Field of View P Object distance = 1000 mm Depth precision is about 0.1mm.

21. Aperture diameter 800 mm Depth from Diffusion vs. Depth from Defocus Depth from Defocus Suppose 22.5x15mm Sensor, 10 um pixel, 100 mm EFL Object P Object distance =1000 mm Depth precision is about 0.1mm.

22. Depth from Diffusion vs. Depth from Defocus Depth from Diffusion Suppose 22.5x15mm Sensor, 10 um pixel, 100 mm EFL A Diffuser of 11.2o Object Any lens is fine! P Object distance =5000 mm Depth precision is about 1.0mm.

23. Depth from Diffusion vs. Depth from Defocus Depth from Defocus Suppose 22.5x15mm Sensor, 10 um pixel, 100 mm EFL Object Lens Aperture diameter ? P Object distance =5000 mm Depth precision is about 1.0mm.

24. Aperture diameter 2000 mm Depth from Diffusion vs. Depth from Defocus Depth from Defocus Suppose 22.5x15mm Sensor, 10 um pixel, 100 mm EFL Object P Object distance = 5000 mm Depth precision is about 1.0mm.

25. Captured Modeled PSF Measurement: A Pinhole Camera F/22 , Field Angle = 0o Z = 5 mm Z = 2 mm - Canon EOS T1i; EF 50mm F/1.8 Lens; - Luminit Holographic Diffuser (10o Gaussian); - Diffuser distance: U = 1m

26. Captured Modeled PSF Measurement: A Pinhole Camera F/22 , Field Angle = 10o Z = 2 mm Z = 5 mm - Canon EOS T1i; EF 50mm F/1.8 Lens; - Luminit Holographic Diffuser (10o Gaussian); - Diffuser distance: U = 1m

27. Captured Modeled PSF Measurement: A Lens Camera F/1.8 , Field Angle = 10o - Canon EOS T1i; EF 50mm F/1.8 Lens; - Luminit Holographic Diffuser (10o Gaussian); - Diffuser distance: U = 1m

28. Experiments Canon 20D + 50mm Lens Five playing cards, 0.29mm thick each Luminit Diffuser (20o)

29. Experiments Captured WITHOUT a Diffuser Captured WITH a Diffuser

30. Experiments Five playing cards, 0.29mm thick each (mm) Computed Depth Map (~ 0.1 mm precision)

31. Experiments A small sculpture of about 4mm thickness Canon G5 Compact Camera Luminit Diffuser (5o)

32. Experiments Captured WITHOUT a Diffuser Captured WITH a Diffuser

33. Experiments A small sculpture of about 4mm thickness Computed Depth Map A 3D View of Depth Map

34. 450 mm Experiments 650 mm Canon 20D; Gaussian Diffuser (10o)

35. (mm) Experiments Stitched Depth Map (precision)

36. Object Diffuser Camera Summary • Formulated the image formation with optical diffusers • Proposed Depth from Diffusion - Require a diffuser on the object side + High-precision depth estimation + Distant objects + Less sensitive to lens aberrations • Demonstrated high-precision depth estimation

37. Depth from Diffusion Changyin Zhou Oliver Cossairt Shree Nayar Columbia University Supported by ONR