Computer Vision CS 776 Spring 2014

1. Computer VisionCS 776 Spring 2014 Cameras & Photogrammetry 3 Prof. Alex Berg (Slide credits to many folks on individual slides)

2. Cameras & Photogrammetry 3 Gamma Scientific Adapted from: http://en.wikipedia.org/wiki/File:Commercial_Integrating_Sphere.jpg http://www.autobild.de/artikel/xenonlampen-im-test-2822873.html http://www.math.tu-dresden.de/DMV2000/Impress/PIC003.jpg

3. Stories of light Alex Berg 2012 Jensen & Buhler Siggraph 2002

4. Image formation • What determines the brightness of an image pixel? Light sourceproperties Surface shape and orientation Sensor characteristics Exposure Surface reflectanceproperties Optics Slide by L. Fei-Fei

5. Fundamental radiometric relation • L: Radiance emitted from P toward P’ • E: Irradiance falling on P’ from the lens P α d P’ f z What is the relationship between E and L? Szeliski 2.2.3

6. Fundamental radiometric relation • Image irradiance is linearly related to scene radiance • Irradiance is proportional to the area of the lens and inversely proportional to the squared distance between the lens and the image plane • The irradiance falls off as the angle between the viewing ray and the optical axis increases P α d P’ f z Szeliski 2.2.3

7. The interaction of light and surfaces • What happens when a light ray hits a point on an object? • Some of the light gets absorbed • converted to other forms of energy (e.g., heat) • Some gets transmitted through the object • possibly bent, through “refraction” • Or scattered inside the object (subsurface scattering) • Some gets reflected • possibly in multiple directions at once • Really complicated things can happen • fluorescence • Bidirectional reflectance distribution function (BRDF): how bright a surface appears when viewed from one direction when light falls on it from another Slide by Steve Seitz

8. BRDFs can be incredibly complicated… Slide by Svetlana Lazebnik

9. Questions: how “bright” will surfaces be? what is “brightness”? measuring light interactions between light and surfaces Core idea - think about light arriving at a surface around any point is a hemisphere of directions Simplest problems can be dealt with by reasoning about this hemisphere Radiometry Computer Vision - A Modern Approach Set: Radiometry Slides by D.A. Forsyth

10. Lambert’s wall Computer Vision - A Modern Approach Set: Radiometry Slides by D.A. Forsyth

11. More complex wall Computer Vision - A Modern Approach Set: Radiometry Slides by D.A. Forsyth

12. Principle: two sources that look the same to a receiver must have the same effect on the receiver. Principle: two receivers that look the same to a source must receive the same amount of energy. “look the same” means produce the same input hemisphere (or output hemisphere) Reason: what else can a receiver know about a source but what appears on its input hemisphere? (ditto, swapping receiver and source) Crucial consequence: a big source (resp. receiver), viewed at a glancing angle, must produce (resp. experience) the same effect as a small source (resp. receiver) viewed frontally. Foreshortening Computer Vision - A Modern Approach Set: Radiometry Slides by D.A. Forsyth

13. By analogy with angle (in radians), the solid angle subtended by a region at a point is the area projected on a unit sphere centered at that point The solid angle subtended by a patch area dA is given by Another useful expression: Solid Angle Computer Vision - A Modern Approach Set: Radiometry Slides by D.A. Forsyth

14. Desirable property: in a vacuum, the relevant unit does not go down along a straight line. How do we get a unit with this property? Think about the power transferred from an infinitesimal source to an infinitesimal receiver. We have total power leaving s to r = total power arriving at r from s Also: Power arriving at r is proportional to: solid angle subtended by s at r (because if s looked bigger from r, there’d be more) foreshortened area of r (because a bigger r will collect more power Measuring Light in Free Space Computer Vision - A Modern Approach Set: Radiometry Slides by D.A. Forsyth

15. All this suggests that the light transferred from source to receiver should be measured as: Radiant power per unit foreshortened area per unit solid angle This is radiance Units: watts per square meter per steradian (wm-2sr-1) Usually written as: Crucial property: In a vacuum, radiance leaving p in the direction of q is the same as radiance arriving at q from p which was how we got to the unit Radiance Computer Vision - A Modern Approach Set: Radiometry Slides by D.A. Forsyth

16. Power 1->2, leaving 1: Power 1->2, arriving at 2: But these must be the same, so that the two radiances are equal Radiance is constant along straight lines Computer Vision - A Modern Approach Set: Radiometry Slides by D.A. Forsyth

17. How much light is arriving at a surface? Sensible unit is Irradiance Incident power per unit area not foreshortened This is a function of incoming angle. A surface experiencing radiance L(x,q,f) coming in from dw experiences irradiance Crucial property: Total power arriving at the surface is given by adding irradiance over all incoming angles --- this is why it’s a natural unit Total power is Irradiance Computer Vision - A Modern Approach Set: Radiometry Slides by D.A. Forsyth

18. Many effects when light strikes a surface -- could be: absorbed transmitted skin reflected mirror scattered milk travel along the surface and leave at some other point sweaty skin Assume that surfaces don’t fluoresce e.g. scorpions, washing powder surfaces don’t emit light (i.e. are cool) all the light leaving a point is due to that arriving at that point Light at surfaces Computer Vision - A Modern Approach Set: Radiometry Slides by D.A. Forsyth

19. Assuming that surfaces don’t fluoresce surfaces don’t emit light (i.e. are cool) all the light leaving a point is due to that arriving at that point Can model this situation with the Bidirectional Reflectance Distribution Function (BRDF) the ratio of the radiance in the outgoing direction to the incident irradiance The BRDF Computer Vision - A Modern Approach Set: Radiometry Slides by D.A. Forsyth

20. Units: inverse steradians (sr-1) Symmetric in incoming and outgoing directions - this is the Helmholtz reciprocity principle Radiance leaving a surface in a particular direction: add contributions from every incoming direction BRDF Computer Vision - A Modern Approach Set: Radiometry Slides by D.A. Forsyth

21. In many situations, we do not really need angle coordinates e.g. cotton cloth, where the reflected light is not dependent on angle Appropriate radiometric unit is radiosity total power leaving a point on the surface, per unit area on the surface (Wm-2) note that this is independent of the direction Radiosity from radiance? sum radiance leaving surface over all exit directions, multiplying by a cosine because this is per unit area not per unit foreshortened area Suppressing Angles - Radiosity Computer Vision - A Modern Approach Set: Radiometry Slides by D.A. Forsyth

22. Important relationship: radiosity of a surface whose radiance is independent of angle (e.g. that cotton cloth) Radiosity Computer Vision - A Modern Approach Set: Radiometry Slides by D.A. Forsyth

23. Suppressing the angles in the BRDF • BRDF is a very general notion • some surfaces need it (underside of a CD; tiger eye; etc) • very hard to measure • ,illuminate from one direction, view from another, repeat • very unstable • minor surface damage can change the BRDF • e.g. ridges of oil left by contact with the skin can act as lenses • for many surfaces, light leaving the surface is largely independent of exit angle • surface roughness is one source of this property Computer Vision - A Modern Approach Set: Radiometry Slides by D.A. Forsyth

24. Directional hemispheric reflectance: the fraction of the incident irradiance in a given direction that is reflected by the surface (whatever the direction of reflection) unitless, range is 0-1 Note that DHR varies with incoming direction eg a ridged surface, where left facing ridges are absorbent and right facing ridges reflect. Directional hemispheric reflectance Computer Vision - A Modern Approach Set: Radiometry Slides by D.A. Forsyth

25. For some surfaces, the DHR is independent of illumination direction too cotton cloth, carpets, matte paper, matte paints, etc. For such surfaces, radiance leaving the surface is independent of angle Called Lambertian surfaces (same Lambert) or ideal diffuse surfaces Use radiosity as a unit to describe light leaving the surface DHR is often called diffuse reflectance, or albedo for a Lambertian surface, BRDF is independent of angle, too. Useful fact: Lambertian surfaces and albedo Computer Vision - A Modern Approach Set: Radiometry Slides by D.A. Forsyth

26. Another important class of surfaces is specular, or mirror-like. radiation arriving along a direction leaves along the specular direction reflect about normal some fraction is absorbed, some reflected on real surfaces, energy usually goes into a lobe of directions can write a BRDF, but requires the use of funny functions Specular surfaces Computer Vision - A Modern Approach Set: Radiometry Slides by D.A. Forsyth

27. There are very few cases where the exact shape of the specular lobe matters. Typically: very, very small --- mirror small -- blurry mirror bigger -- see only light sources as “specularities” very big -- faint specularities Phong’s model reflected energy falls off with Phong’s model Computer Vision - A Modern Approach Set: Radiometry Slides by D.A. Forsyth

28. Lambertian + specular • Widespread model • all surfaces are Lambertian plus specular component • Advantages • easy to manipulate • very often quite close true • Disadvantages • some surfaces are not • e.g. underside of CD’s, feathers of many birds, blue spots on many marine crustaceans and fish, most rough surfaces, oil films (skin!), wet surfaces • Generally, very little advantage in modelling behaviour of light at a surface in more detail -- it is quite difficult to understand behaviour of L+S surfaces Computer Vision - A Modern Approach Set: Radiometry Slides by D.A. Forsyth

29. Diffuse reflection • Light is reflected equally in all directions • Dull, matte surfaces like chalk or latex paint • Microfacets scatter incoming light randomly • Effect is that light is reflected equally in all directions • Brightness of the surface depends on the incidence of illumination darker brighter Slide by Svetlana Lazebnik

30. Diffuse reflection: Lambert’s law N S θ B: radiosity (total power leaving the surface per unit area) ρ: albedo (fraction of incident irradiance reflected by the surface) N: unit normal S: source vector (magnitude proportional to intensity of the source) Slide by Svetlana Lazebnik

31. Specular reflection • Radiation arriving along a source direction leaves along the specular direction (source direction reflected about normal) • Some fraction is absorbed, some reflected • On real surfaces, energy usually goes into a lobe of directions • Phong model: reflected energy falls of with • Lambertian + specular model: sum of diffuse and specular term Slide by Svetlana Lazebnik

32. Moving the light source Changing the exponent Specular reflection Slide by Svetlana Lazebnik

33. Fundamental radiometric relation • Application: • S. B. Kang and R. Weiss, Can we calibrate a camera using an image of a flat, textureless Lambertian surface? ECCV 2000. Slide by Svetlana Lazebnik

34. Photometric stereo (shape from shading) • Can we reconstruct the shape of an object based on shading cues? Luca dellaRobbia,Cantoria, 1438 Slide by Svetlana Lazebnik

35. S2 S1 ??? Photometric stereo • Assume: • A Lambertian object • A local shading model (each point on a surface receives light only from sources visible at that point) • A set of known light source directions • A set of pictures of an object, obtained in exactly the same camera/object configuration but using different sources • Orthographic projection • Goal: reconstruct object shape and albedo Sn F&P 2nd ed., sec. 2.2.4

36. Surface model: Monge patch F&P 2nd ed., sec. 2.2.4

37. Image model • Known: source vectors Sjand pixel values Ij(x,y) • Unknown:normal N(x,y) and albedo ρ(x,y) • Assume that the response function of the camera is a linear scaling by a factor of k • Lambert’s law: F&P 2nd ed., sec. 2.2.4

38. (n × 1) (n × 3) (3× 1) known known unknown Least squares problem • Obtain least-squares solution for g(x,y) (which we defined asN(x,y) (x,y)) • Since N(x,y) is the unit normal, (x,y) is given by the magnitude of g(x,y) • Finally, N(x,y) = g(x,y) / (x,y) • For each pixel, set up a linear system: F&P 2nd ed., sec. 2.2.4

39. Example Recovered albedo Recovered normal field F&P 2nd ed., sec. 2.2.4

40. Recall the surface is written as This means the normal has the form: If we write the estimated vector gas Then we obtain values for the partial derivatives of the surface: Recovering a surface from normals F&P 2nd ed., sec. 2.2.4

41. Integrability: for the surface f to exist, the mixed second partial derivatives must be equal: We can now recover the surface height at any point by integration along some path, e.g. Recovering a surface from normals (for robustness, can take integrals over many different paths and average the results) (in practice, they should at least be similar) F&P 2nd ed., sec. 2.2.4

42. Surface recovered by integration F&P 2nd ed., sec. 2.2.4

43. Finding the direction of the light source I(x,y) = N(x,y) ·S(x,y) + A Full 3D case: N S For points on the occluding contour: P. Nillius and J.-O. Eklundh, “Automatic estimation of the projected light source direction,” CVPR 2001 Slide by Svetlana Lazebnik

44. Finding the direction of the light source P. Nillius and J.-O. Eklundh, “Automatic estimation of the projected light source direction,” CVPR 2001 Slide by Svetlana Lazebnik

45. Application: Detecting composite photos Real photo Fake photo M. K. Johnson and H. Farid, Exposing Digital Forgeries by Detecting Inconsistencies in Lighting, ACM Multimedia and Security Workshop, 2005. Slide by Svetlana Lazebnik

46. From light rays to pixel values • Camera response function: the mapping f from irradiance to pixel values • Useful if we want to estimate material properties • Enables us to create high dynamic range images • For more info: P. E. Debevec and J. Malik, Recovering High Dynamic Range Radiance Maps from Photographs,SIGGRAPH 97 Slide by Svetlana Lazebnik

47. Limitations • Orthographic camera model • Simplistic reflectance and lighting model • No shadows • No interreflections • No missing data • Integration is tricky Slide by Svetlana Lazebnik

48. More reading & thought problems (reconstructing from a single image using priors) Shape, Illumination, and Reflectance from Shading Jonathan T. Barron, Jitendra Malik Tech Report, 2013 (implementing this one is the extra credit) Recovering High Dynamic Range Radiance Maps from Photographs. Paul E. Debevec and Jitendra Malik. SIGGRAPH 1997 (people can perceive reflectance) Surface reflectance estimation and natural illumination statistics. R.O. Dror, E.H. Adelson, and A.S. Willsky. Workshop on Statistical and Computational Theories of Vision 2001