3D Vision

1. 3D Vision • Why? • The world is 3D • Not all useful information is readily available in 2D • Why so hard? • “Inverse problem”: one image = many scenes • Complex relationship between objects & pixels • Noise, occlusion, etc. • What can we do? • Use "hints" from our knowledge of the world • Add more information to the problem!

2. Labeling Image Edges • Many edge labels carry 3D information • Occluding blade (>) • Occluding surface to right along arrow • Convex crease (+) • Edge is closer than both surfaces • Concave crease (–) • Edge is further than both surfaces • Limb (>>) • Edge is "horizon" of curving-away surface • Others are about reflectance or illumination changes • Mark (M) • Change due to paint or material boundary • Illumination Boundary (S) • Shadow edge

3. Intrinsic Image Pixel Contents • Depth (range) • Orientation (surface normal) • Illumination • Albedo (reflectance) • Given this information, the picture (intensity values) can be completely reconstructed

4. 3D Cues in Single 2D Images • Occlusion • Occluding objects are closer to the camera • The crossbar of a T-junction belongs to the occluding object • Perspective scaling and foreshortening • If two copies of the same object appear in a picture, the smaller one is further away. • Scaling is parallel to the image plane • Foreshortening is perpendicular to the image plane • Texture Gradient • Since texture is composed of repeated patterns, changes in size and density of texture convey depth cues

5. "Shape-From" Methods • Use a cue (e.g. texture, shading) from a small region of an image • Cues generally give partial surface orientation information • E.g. degree of tilt • Related cues can give "boundary conditions" to start from • Solve for continuous surfaces that satisfy both the general and boundary constraints

6. Example: Shape From Shading Figure 12.2

7. Gradient Space Represents Surface Normal

8. Reflectance Geometry • Three directions are important: normal to the surface, surface to light source, and surface to camera

9. Types of Reflection (Review) • Specular reflection (mirror) • Color depends on light source color • Limited scattering: angle of incidence = angle of reflection • Camera sees light if it’s pointed in the right direction • Nearly all light is reflected • Lambertian reflection (matte) • Color depends on material properties of object • Light evenly scattered throughout half-space • Camera sees light if surface is visible • Amount of light reflected is proportional to angle between surface and light source

10. Shape From Shading • Lambertian surface, known light source direction • Reflective component (highlights) can be subtracted out in preprocessing • Relative brightness of surface patches constrain their directions • Darker patches are more tilted away • A given brightness value represents a circle in gradient space • Boundary pixels indicate surface at 90 degrees from normal (if smooth surface) • Solve an optimization problem: brightness term and smoothness term

11. Shape From Shading (cont’d) • Brightness term • Intensity is a function of reflectance, and reflectance is a function of surface normals (p,q) and light source direction (vx, vy, vz) • Smoothness term • Try to minimize integral of partial derivatives of p and q in x and y direction

12. Photometric Stereo • Extension to multiple "images" • Lambertian surface, several light sources • Each image has one light source, constrains surfaces • Solve an overdetermined linear (matrix) system - like camera calibration with extra points • Implementation • Surround your object with a frame containing inward-pointing lights • Take an image with each light in turn • Use images and known light directions to solve the equations.

13. More Variations • SHINY • Photometric stereo system for highly reflective materials • Used to accurately characterize welds • Accurate color determination (plastic objects) • Separate highlights from matte portion • Determine illumination color from highlight • Determine object color • Create “matte object” for photometric stereo or shape from shading

14. Shape From Texture Figure 12.3

15. Shape from Texture • Transformation of original texture related to surface normal • Solve for affine transformation between original texel and viewed texel • Transformation depends on surface normal & distance • (Assume camera is far enough to avoid worst perspective distortion) • If the original texel is known, transformations can be computed directly • If the original texel is unknown, assume the largest visible texel is directly facing the camera • Use smoothness or shape constraints to eliminate alternatives

16. Shape from Focus • Vary the focal length of the camera (i.e. zoom lens) • Objects at different distances will become clear at different focal lengths • Can use comparisons between pairs of images to get relative distances

17. Choice Depends on Object • Solid, matte object (or matte separated from specular) • Shape from shading, photometric stereo • Highly reflective object • Shape from specular reflection • Regular textured object • Shape from texture, stereo, focus • Irregular textured object • Stereo, focus