Shape from X

1. CS 636 Computer Vision Shape from X Nathan Jacobs Slides by Lazebnik

2. Cues to 3D • Motion • Shading • Defocus • (moving) Shadows and Specularities • Texture

3. Photometric Stereo Merle Norman Cosmetics, Los Angeles • Readings • R. Woodham, Photometric Method for Determining Surface Orientation from Multiple Images. Optical Engineering 19(1)139-144 (1980). (PDF) Slides by Seitz

4. image intensity of P Diffuse reflection • Simplifying assumptions • I = Re: camera response function f is the identity function: • can always achieve this in practice by solving for f and applying f -1 to each pixel in the image • Ri = 1: light source intensity is 1 • can achieve this by dividing each pixel in the image by Ri

5. Shape from shading • Suppose • You can directly measure angle between normal and light source • Not quite enough information to compute surface shape • But can be if you add some additional info, for example • assume a few of the normals are known (e.g., along silhouette) • constraints on neighboring normals—“integrability” • smoothness • Hard to get it to work well in practice • plus, how many real objects have constant albedo?

6. L3 L2 L1 Photometric stereo N V Can write this as a matrix equation:

7. Solving the equations

8. Least squares solution: Solve for N, kd as before What’s the size of LLT? More than three lights • Get better results by using more lights

9. Computing light source directions • Trick: place a chrome sphere in the scene • the location of the highlight tells you where the light source is

10. V2 V1 N Depth from normals • Get a similar equation for V2 • Each normal gives us two linear constraints on z • compute z values by solving a matrix equation orthographic projection

11. Results… Input (1 of 12) Normals Normals Shadedrendering Texturedrendering

12. Results… from Athos Georghiades http://cvc.yale.edu/people/Athos.html

13. Limitations • Big problems • doesn’t work for shiny things, semi-translucent things • shadows, inter-reflections • Smaller problems • camera and lights have to be distant • calibration requirements • measure light source directions, intensities • camera response function • Newer work addresses some of these issues • Some pointers for further reading: • Zickler, Belhumeur, and Kriegman, "Helmholtz Stereopsis: Exploiting Reciprocity for Surface Reconstruction." IJCV, Vol. 49 No. 2/3, pp 215-227. • Hertzmann & Seitz, “Example-Based Photometric Stereo: Shape Reconstruction with General, Varying BRDFs.” IEEE Trans. PAMI 2005

14. Depth from (De)Focus

15. Many Methods • Try many focal depths, pick the sharpest for each pixel • Treat circle of confusion like a local blur… use a deconvolution method. • (more on that)

16. Image and Depth from a Conventional Camera with a Coded Aperture Anat Levin, Rob Fergus, Frédo Durand, William Freeman MIT CSAIL

17. Output #1: Depth map Single input image:

18. Output #1: Depth map Single input image: Output #2: All-focused image

19. Lens and defocus Image of a point light source Lens’ aperture Camera sensor Lens Point spread function Focal plane

20. Lens and defocus Image of a defocused point light source Lens’ aperture Camera sensor Object Lens Point spread function Focal plane

21. Lens and defocus Image of a defocused point light source Lens’ aperture Camera sensor Object Lens Point spread function Focal plane

22. Lens and defocus Image of a defocused point light source Lens’ aperture Camera sensor Object Lens Point spread function Focal plane

23. Lens and defocus Image of a defocused point light source Lens’ aperture Camera sensor Object Lens Point spread function Focal plane

24. Out of focus In focus Depth and defocus Depth from defocus: Infer depth by analyzing local scale of defocus blur ill posed

25. Challenges • Hard to discriminate a smooth scene from defocus blur • Hard to undo defocus blur ? Out of focus Input Ringing with conventional deblurring algorithm

26. Defocus as local convolution Calibrated blur kernels at different depths Input defocused image

27. Defocus as local convolution Local sub-window Calibrated blur kernels at depth Sharp sub-window Input defocused image Depth k=1: Depth k=2: Depth k=3:

28. Overview Try deconvolving local input windows with different scaled filters: ? Larger scale ? Correct scale ? Smaller scale Somehow: select best scale.

29. ? Larger scale ? Correct scale ? Smaller scale Challenges • Hard to deconvolve even when kernel is known • Hard to deconvolve even when kernel is known Input Ringing with the traditional Richardson-Lucy deconvolution algorithm • Hard to identify correct scale:

30. Deconvolution is ill posed ? =

31. Deconvolution is ill posed Solution 1: = ? Solution 2: = ?

32. Idea 1: Natural images prior What makes images special? Natural Unnatural Image gradient Natural images have sparse gradients put a penalty on gradients

33. Deconvolution with prior Convolution error Derivatives prior 2 _ + ? Low Equal convolution error 2 _ + ? High

34. “spread” gradients “localizes” gradients Richardson-Lucy Gaussian prior Sparse prior Comparing deconvolution algorithms (Non blind) deconvolution code available online: http://groups.csail.mit.edu/graphics/CodedAperture/ Input

35. Comparing deconvolution algorithms (Non blind) deconvolution code available online: http://groups.csail.mit.edu/graphics/CodedAperture/ Input “spread” gradients “localizes” gradients Richardson-Lucy Gaussian prior Sparse prior

36. Recall: Overview Try deconvolving local input windows with different scaled filters: ? Larger scale ? Correct scale ? Smaller scale Somehow: select best scale. Challenge: smaller scale not so different than correct

37. Depth from Diffusion Changyin Zhou Oliver Cossairt Shree Nayar Columbia University Supported by ONR

38. Optical Diffuser

39. Optical Diffuser ~ 10 micron Micrograph of a Holographic Diffuser (RPC Photonics) [Gray, 1978] [Chang et al., 2006] [Garcia-Guerrero et al. 2007]

40. Diffuser to preview the image (B&H) Diffusers for illumination (B&H) Diffusers to soften the image Diffusers as Accessories

41. Object Object Diffuser Camera Diffuser Camera Diffusion Encodes Depth The amount of diffusion varies with depth.

42. Geometry of Diffusion: A Pinhole Camera Object P Miss Pinhole Q Sensor

43. θ Geometry of Diffusion: A Pinhole Camera Object P Pinhole Sensor Diffuser

44. A θ B θ Geometry of Diffusion: A Pinhole Camera Object P Pinhole Sensor Diffuser

45. Geometry of Diffusion: A Pinhole Camera Diffusion Law: Object A P θ B θ Pinhole O 2r V Z U Sensor Diffuser Object

46. Geometry of Diffusion: A Pinhole Camera Diffusion Size and Depth: Object A P θ B θ Pinhole O 2r V Z U Sensor Diffuser Object

47. Geometry of Diffusion: A Pinhole Camera Diffuser as a proxy object Diffusion Size and Depth: Z P Pinhole O 2r V U Sensor Diffuser Object

48. Captured Image Diffusion PSF Latent clear image Diffusion Size Diffusion as Convolution: A Pinhole Camera Assume field angle and depth are constant for small image patches, we have:

49. Experiments Canon 20D + 50mm Lens Five playing cards, 0.29mm thick each Luminit Diffuser (20o)

50. Experiments Captured WITHOUT a Diffuser Captured WITH a Diffuser