Review of 3D vision

1. Review of 3D vision Cmput 428/615

2. Upcoming • Graduate project presentations: Tue Apr 14 15-17:30 • Diego: Computer vision based interface to robots • Kory, Fateme: Video tracking evaluation • Chris, Yifeng: SiftFu reconstruction from stereo • Jay Carriere: Tracking of a 3D moving surface Wed Apr 15 12-14 • Antonio Carlos Furtado: Color mapping for on-line scene reconstruction • Shida He: Detecting, clustering and tracking outlier motion in PTAM • Ahmed: Shape from shading • Sayem, Toukir: Comparison of real time reconstruction methods Apr 21 14:00 – 17 GSB211 Final exam: 4 pages of your notes, calculator

3. 3D object and scene reconstruction • Reconstruct 3D points and cameras = SFM • Reconstruct whole objects = surface • Reconstruct material properties = reflectance SFM Surface estimation Reflectance estimation Cam’s

4. SFM + stereo • dominant planes • plane sweep – homog between 3D pl. and camera pl. • one parameter search – voting for a plane [Zisserman, Werner ECCV02 ] [Bischof et al 3DPVT06 ]

5. SFS + level-set photo consist. Neil Birkbeck • Motivation • method that works for objects with general reflectance • textured and uniform regions shading + texture light variation (multiview PS) • specular materials general BRDF – parametric System Camera calibration: pattern Light calibration: specular white sphere Light variation:rotating table, fixed cam.

6. Image cues Shading [reconstructs normals] shape from shading (SFS) photometric stereo Specular highlights Texture [reconstructs 3D] stereo (relates two views) Silhouette [reconstructs 3D] shape from silhouette [Focus] [ignore, filtered] [parametric BRDF]

7. Projective camera model [Dürer] Projection matrix Camera matrix (internal params) Rotation, translation (ext. params)

8. Multi-view geometry - resection • Projection equation xi=PiX • Resection: • xi,X Pi Given image points and 3D points calculate camera projection matrix.

9. Multi-view geometry - intersection • Projection equation xi=PiX • Intersection: • xi,Pi X Given image points and camera projections in at least 2 views calculate the 3D points (structure)

10. Multi-view geometry - SFM • Projection equation xi=PiX • Structure from motion (SFM) • xi Pi,X • Given image points in at least 2 views calculate the 3D points (structure) and camera projection matrices (motion) • Estimate projective structure • Rectify the reconstruction to metric (autocalibration)

11. Resection using DLT • Camera proj • Use cross prod (as for H) By components: Only 2 linindep Solve resulting Eq. syst

12. Multi-view geometry - intersection • Projection equation xi=PiX • Intersection: • xi,Pi X Given image points and camera projections in at least 2 views calculate the 3D points (structure)

13. Intersection: Linear triangulation Alternative way of linear intersection: • Formulate a set of linear equations explicitly solving for l’s See our VR2003 tutorial p. 26

14. Multi-view geometry - SFM • Projection equation xi=PiX • Structure from motion (SFM) • xi Pi,X • Given image points in at least 2 views calculate the 3D points (structure) and camera projection matrices (motion) • Estimate projective structure • Rectify the reconstruction to metric (autocalibration)

15. SFM for linear camera approximation [Tomasi &Kanade ’92] • Affine camera • Projection • n points, m views: measurement matrix M 2x3 matrix; t 2D vector W: Rank 3 Assuming isotropic zero-mean Gaussian noise, factorization achieves ML affine reconstruction.

16. Object/Scene reconstruction Discrete • Voxel carving • Graph cut techniques Continuous • Variational and level set techniques • Mesh-based reconstruction

17. Disparity/Depth map 3D point Reference image Reference image plane Object (surface) : Normals : Method: find f that best agrees with the input images (minimize the cost functional integrated over the surface) Regularization: smoothness on Visibility: ? (mesh defined on image plane + Zbuffering)

18. Depth w.r. base mesh Object (surface) : , d – displacement direction (displacement map) Normals : local (per triangle) transform to global CS Method: Regularization: smoothness on (local / global) Visibility: ? (fine mesh to connect points on each plane + Zbuffering) How to deal with boundaries ?

19. Mesh Object (surface) : mesh vertices Normals :interpolated Method: move vertices along interpolated normals based on photo-consistency of neighboring triangles. Regularization: smooth normals Visibility: Zbuffering Topologiocal changes ?

20. Reconstruction as labeling Graph cuts Photo-consistency Smoothness Ex: disparity mapvoxels pixels voxels disparities occupancy Discrete formulation: • surface representation • labels Find a set of labels that minimize Notes: • NP hard • Can be solved using MRF energy minimization methods graph cuts, dynamic programming, belief propagation, simulated annealing …

21. Example of 2D geometric graph [Paris, Sillion, Quan: A surface reconstruction method using global graph cut optimization IJCV 05] Disparity map: pixel label = disparity source D4(d1) D1(d1) ’ ’ ’ d1 D1(d2) D2(d2) D4(d2) D3(d2) d2 f1=d4 f2=d3 f3=d3 f4=d4 D4(d3) d3 D4(d4) ’ ’ ’ d4     sink Surface Graph

22. Graph Cuts as Hypersurfaces in 3D labels t “cut” y s p x cut • Graph fully embedded in the working geometric space • Feasible cut = separated hypersurface in the embedding continuous manifold

23. Results Convex smoothing – global solution [Paris, Sillion, Quan IJCV 05, ACCV 04]

24. Measuring LightRadiance • Foreshortening and Solid angle • Measuring light : radiance • Light at surface : interaction between light and surface • irradiance = light arriving at surface • BRDF • outgoing radiance • Special cases and simplifications : Lambertain, specular, parametric and non-parametric models Incoming Outgoing

25. BRDF properties BRDF = Bi-directional reflectance distribution function Measures, for a given wavelength, the fraction of incoming irradiance from a direction i in the outgoing directiono [Nicodemus 70] • Properties : • Non-negative • Helmholtz reciprocity • Linear • Total energy leaving a surface less than total energy arriving at surface

26. BRDF properties isotropic (3DOF) = anisotropic (4 DOF) [Hertzmann&Seitz CVPR03]

27. Lambertian BRDF Diffuse reflectance acts like a low pass filter on the incident illumination. • Emitted radiance constant in all directions • Models – perfect diffuse surfaces : clay, mate paper, … • BRDF = constant = albedo • One light source = dot product normal and light direction light dir normal albedo

28. Specular reflection • Smooth specular surfaces • Mirror like surfaces • Light reflected along specular direction • Some part absorbed • Rough specular surfaces • Lobe of directions around the specular direction • Microfacets • Lobe • Very small – mirror • Small – blurry mirror • Bigger – see only light sources • Very big – fait specularities

29. Phong model Specular dir Symmetric V shaped microfacets

30. Geometry from shading • Photometric Stereo • Several images, different lights • Unknown Lambertian BRDF • Known lights • Unknown lights Shape from Shading One image Known light direction Known BRDF (unit albedo) Ill-posed : additional constraints (intagrability …) [Silver 80, Woodman 81] [Horn] Reconstruct normals Integrate surface Shading reveals 3D shape geometry

31. Photometric stereo One image, one light direction n images, n light directions Recover Albedo = magnitude Normal = normalized [Birkbeck] Given: n>=3 images with different known light dir. (infinite light) Assume: Lambertain object orthograhic camera ignore shadows, interreflections

32. Geometry and reflectance modeling Neil Birkbeck • Motivation • method that works for objects with general reflectance • textured and uniform regions shading + texture light variation (multiview PS) • specular materials general BRDF – parametric System Camera calibration: pattern Light calibration: specular white sphere Light variation:rotating table, fixed cam.

33. Surface discretization Surface S= triangles, move vertices Cost function Normals interpolated Albedo implied by the shape (closed form solution) (knowing light dir+color) Visibility/shadows Z buffering

34. Cost functional 2. Non-Lambertian reflectance BRDF = Phong In practice : filter specular highlights fit full reflectance only at the end Cost functional Per-point cost function Visibility+sampling reflectance camera proj. image light 1. Lambertial reflectance ambient color BRDFalbedo light color light dir.

35. Surface optimization Surface Evolution flow (Euler-Lagrange equations) Regularizer mean curvature H Gradient finite differences Initial shape = visual hull Mesh handles topological changes Multi-resolution(image pyramid)

36. Results : shape + reflectance

37. Summary: representations • Image centered • Depth/disparity map Object centered Implicit (level sets) Mesh Voxels time 3D point Image plane

38. Discrete vs. continuous Gradient descent method VS. Global minimization tool (restricted class of energies) [Boykov CVPRTut 2005]

39. Radiometry - summary

40. MSc Best thesis Vision David Lovi http://webdocs.cs.ualberta.ca/~dlovi/thesis/ • Likes: Baking bread, hanging out with friends, waking up > 12pm • Thesis: Incremental Free-Space Carving for Real-Time 3D Reconstruction Cameras Freespace Uncarved/ occupied Conflicts (a) Initial (b) New point event (d) New camera event In 3D: Triang  Tetrahedra (c) Retriangulation & carving

41. Best PhD Thesis, Vision Neil Birkbeckhttp://webdocs.cs.ualberta.ca/~birkbeck/index.php?target=thesis • Neil Birkbeck: Likes Bicycling, Skateboarding • Thesis: Image-based Capture and Modeling of Dynamic Human Motion and Appearance

42. You could be an award winner! Good luck in studies, work and life! Neil Birkbeck, top skateboarder