Shape From Shading

1. Shape From Shading Mica Arie-Nachimson May 2011

2. What is Shading? • Well… not shadow… • We can’t reconstruct shape from one shadow… Image from www.moolf.com

3. What is Shading? • Variable levels of darkness • Gives a cue for the actual 3D shape • There is a relation between intensity and shape Images from S. Narasimhan, Carnegie Mellon; www.erco.com; www.alesmav.com; H. Wang, University of California

4. Shading Examples • These circles differ only in grayscale intensity • Intensities give a strong “feeling” of scene structure

5. Talk Outline • Introduction and Basics • Notations • Reflectance map • Photometric stereo • Main Approaches • Propagation Solutions • Basic Solution by Horn [85] • Using Fast Marching, Kimmel & Sethian [2001] • Global energy minimization solution using Belief Propagation – only if time permits

6. Introduction and Basics Problem definition Notations Reflectance Map Photometric Stereo

7. What determines scene radiance? • The amount of light that falls on the surface • The fraction of light that is reflected (albedo) • Geometry of light reflection • Shape of surface • Viewpoint n

8. Albedo • A property of the surface • Usually normalized • Can be modeled as a single scalar for the entire surface or dependent on location • Usually assumed to be a known scalar • Is this a ball with uniform albedo or is this a 2D circle with varying albedo? • We’re not going to talk about the albedo

9. Surface Normal Convenient notation for surface orientation • A smooth surface has a tangent plane at every point • We can model the surface using the normal at every point

10. The Shape From Shading Problem • Given a grayscale image • And albedo • And light source direction • Reconstruct scene geometry • Can be modeled by surface normals

11. Lambertian Surface • Appears equally bright from all viewing directions • Reflects all light without absorbing • Matte surface, no “shiny” spots • Brightness of the surface as seen from camera is linearly correlated to the amount of light falling on the surface Today we will discuss only Lambertiansurfaces under point-source illumination n

12. Some Notations:Surface Orientation • A smooth surface has a tangent plane at every point • Mark • Parametrize surface orientation by first partial derivatives of

13. Some Notations:Surface Orientation Surface normal • , • Normalize

14. Reflectance Map • Relationship between surface orientation and brightness Lambertian surface • Image irradiance (brightness) is proportional to

15. Reflectance Map Example iso-brightness contour • Brightness as a function of surface orientation Lambertian surface Illustration from S. Narasimhan, Carnegie Mellon

16. Reflectance Map of a Glossy Surface • Brightness as a function of surface orientation Surface with diffuse and glossy components

17. Reflectance Map Examples • Brightness as a function of surface orientation

18. Graphics with a 3D Feel • Given a 3D surface , lighting and viewing direction, we can compute the gray level of pixel of the surface • Find the gradient of the surface • Use to determine the gray level

19. Shape From Shading? • Can we reconstruct shape from a single image? • Two variables with one equation… So what do we do? Illustration from S. Narasimhan, Carnegie Mellon

20. Shape From Shading! • Use more images • Photometric stereo • Shape from shading • Introduce constraints • Solve locally • Linearize problem Images from R. Basri et al. “Photometric Stereo with General, Unknown Lighting“, IJCV 2006

21. Photometric Stereo • Take several pictures of same object under same viewpoint with different lighting

22. Photometric Stereo • Take several pictures of same object under same viewpoint with different lighting

23. Photometric Stereo • Take several pictures of same object under same viewpoint with different lighting

24. Main Approaches forShape From Shading A Quick Review

25. Main Approaches • Minimization: Solve for a global error function while introducing constraint(s): • Brightness • Smoothness • Intensity gradient • …

26. Main Approaches • Minimization: Solve for a global error function while introducing constraint(s): • Brightness • Smoothness • Intensity gradient • … • Propagation: Grow a solution around an initial known point or boundary • Local: Assume local surface type • Linear: Make problem linear by a linearization of reflectance function

27. Main Approaches • Minimization: Solve for a global error function while introducing constraint(s): • Brightness • Smoothness • Intensity gradient • … • Propagation: Grow a solution around an initial known point or boundary • Local: Assume local surface type • Linear: Make problem linear by a linearization of reflectance function

28. Main Approaches • Minimization: Solve for a global error function while introducing constraint(s): • Brightness • Smoothness • Intensity gradient • … • Propagation: Grow a solution around an initial known point or boundary • Local: Assume local surface type • Linear: Make problem linear by a linearization of reflectance function

29. Basic Propagation Solution Horn [85] Solution by Characteristic Curves

30. Propagating Solution • Suppose that we know the depth of some point on the surface • Extend the solution by taking a small step • But and are unknown, and the image irradiance equation gives only one constraint • If we knew and , can compute changes in using second partial derivatives • and

31. Propagating Solution , what else can we use? • Image irradiance equation

32. Propagating Solution ; • We note that for a small step • For these specific • If you take a small step in the image plane parallel to the direction of the gradient of , then can compute change in

33. Propagating Solution ; • Get ODEs: ; ; ; ; • These equations describe a characteristic curve

34. Propagating Characteristic Curve • Characteristic curve • Need to initialize every curve at some known point • Singular points • Occluding boundaries Image by DejanTodorović

35. Propagating Characteristic Curve • Characteristic curve • Need to initialize every curve at some known point • Singular points • Occluding boundaries • Curves are “grown” independently, very instable Image by DejanTodorović

36. Shape From Shading by Fast Marching “Optimal Algorithm for Shape from Shading and Path Planning”, R. Kimmel and J. A. Sethian [2001]

37. Vertical Light Source Case • Recall reflectance map • Assume light source located near the viewer • =(0,0,1) • This is an Eikonal equation • Can be solved by an numerical algorithm Fast Marching

38. Fast Marching • Expanding Dijkstra’s shortest path algorithm for general surfaces • Represented as triangulated mesh • Flat domains • Many computer vision problems can be set into flat domains • Every pixel is a node, edges between adjacent nodes

39. Dijkstra’s Shortest Paths Connected graphs • Start with the source node 9 6 2 11 14 10 15 9 s 7

40. Dijkstra’s Shortest Paths Connected graphs • Start with the source node • Update all neighbors 9 6 14 2 11 9 14 10 15 9 7 s 7

41. Dijkstra’s Shortest Paths Connected graphs • Start with the source node • Update all neighbors • Go to the closest neighbor • Set its value 9 6 14 2 11 9 14 10 15 9 7 s 7

42. Dijkstra’s Shortest Paths Connected graphs • Start with the source node • Update all neighbors • Go to the closest neighbor • Set its value • Compute its neighbors • Update smaller scores 9 6 14 2 23 11 9 14 10 15 9 7 s 7

43. Dijkstra’s Shortest Paths Connected graphs • Start with the source node • Update all neighbors • Go to the closest neighbor • Set its value • Compute its neighbors • Update smaller scores • Continue with smallest value node • Remember path 9 6 14 2 23 11 9 14 10 15 9 7 s 7

44. Flat Domains: Why Does Dijkstra Fail? • Dijkstra will not find the diagonal path • Need to examine triangles

45. Fast Marching: Problem Definition • Suppose there is a forest fire with multiple sources • Every point that was touched by the fire is burnt and will not be visited by the fire again • Firemen register the time T(x) at which the fire arrives to location x. • The fast marching algorithm simulates this scenario

46. Fast Marching: Problem Definition • Multiple sources • Advancing forward • Advances either at a constant rate or at varying rate • What is the arrival time at every location? Image from G. Rosman, Technion

47. Fast Marching Algorithm • Set and mark the points as done • Set the rest of the points as and mark them as far • All far points adjacent to done points become verify points • Update all verify points using of the done set • The verify with the smallest becomes done • Continue until all points are done Illustrations from selected publications of J. A. Sethian

48. Fast Marching on a Grid Update step T ij ? T i+1,j T i,j-1 i+1,j T i,j-1 i,j+1 T i-1,j ij i,j+1 i-1,j Slide based on slides by R. Kimmel, Technion

49. Fast Marching on a Grid Update step T ij ? T i+1,j T i,j-1 i+1,j T i,j-1 i,j+1 T i-1,j ij i,j+1 i-1,j Slide based on slides by R. Kimmel, Technion

50. Fast Marching on a Grid Update step T ij ? T i+1,j T i,j-1 i+1,j T i,j-1 i,j+1 T i-1,j ij i,j+1 i-1,j Slide based on slides by R. Kimmel, Technion