Image Primitives and Correspondence

1. Image Primitives and Correspondence Stefano Soatto added with slides from Univ. of Maryland and R.Bajcsy, UCB Computer Science Department University of California at Los Angeles

2. Image Primitives and Correspondence Given an image point in left image, what is the (corresponding) point in the right image, which is the projection of the same 3-D point Siggraph 04

3. Image primitives and Features • The desirable properties of features are: • Invariance with respect to Grays scale/color • With respect to location (translation and rotation) • With respect to scale • Robustness • Easy to compute • Local features vs. global features Siggraph 04

4. Feature analysis • Points sensitive to illumination variation but fast to compute • Neighborhood features : gradient based (edge detectors) measuring contrast ,robust to illumination variation except for highlights Fast computation ,it can be done in parallel. The complimentary feature to gradient is region based. The advantage of this feature is it can encompass larger regions that are homogeneous and save processing time. Siggraph 04

5. Profiles of image intensity edges Siggraph 04

6. The gradient direction is given by: • how does this relate to the direction of the edge? • The edge strength is given by the gradient magnitude Image gradient • The gradient of an image: • The gradient points in the direction of most rapid change in intensity Siggraph 04

7. The discrete gradient • How can we differentiate a digital image f[x,y]? • Option 1: reconstruct a continuous image, then take gradient • Option 2: take discrete derivative (finite difference) • How would you implement this as a cross-correlation? Siggraph 04

8. Effects of noise • Consider a single row or column of the image • Plotting intensity as a function of position gives a signal • Where is the edge? Siggraph 04

9. Look for peaks in Solution: smooth first Siggraph 04 • Where is the edge?

10. 2D edge detection filters • is the Laplacian operator: Laplacian of Gaussian Gaussian derivative of Gaussian Siggraph 04

11. Effect of  (Gaussian kernel size) original Canny with Canny with • The choice of depends on desired behavior • large detects large scale edges • small detects fine features Siggraph 04

12. Scale • Smoothing • Eliminates noise edges. • Makes edges smoother. • Removes fine detail. Siggraph 04 (Forsyth & Ponce)

13. Corner detection Corners contain more edges than lines. • A point on a line is hard to match. Siggraph 04

14. Corners contain more edges than lines. • A corner is easier Siggraph 04

15. Edge Detectors Tend to Fail at Corners Siggraph 04

16. Finding Corners • Intuition: • Right at corner, gradient is ill defined. • Near corner, gradient has two different values. Siggraph 04

17. Formula for Finding Corners We look at matrix: Gradient with respect to x, times gradient with respect to y Sum over a small region, the hypothetical corner WHY THIS? Siggraph 04 Matrix is symmetric

18. First, consider case where: • What is region like if: • l1 = 0? • l2 = 0? • l1 = 0 and l2 = 0? • l1 > 0 and l2 > 0? Siggraph 04

19. General Case: From Linear Algebra, it follows that because C is symmetric: With R a rotation matrix. So every case is like one on last slide. Siggraph 04

20. So, to detect corners • Filter image. • Compute magnitude of the gradient everywhere. • We construct C in a window. • Use Linear Algebra to find l1 and l2. • If they are both big, we have a corner. Siggraph 04

21. Matching - Correspondence Lambertian assumption Rigid body motion Correspondence Siggraph 04

22. Local Deformation Models • Translational model • Affine model • Transformation of the intensity values and occlusions Siggraph 04

23. Motion Field (MF) • The MF assigns a velocity vector to each pixel in the image. • These velocities are INDUCED by the RELATIVE MOTION btw the camera and the 3D scene • The MF can be thought as the projectionof the 3D velocities on the image plane. Siggraph 04

24. Motion Field and Optical Flow Field • Motion field: projection of 3D motion vectors on image plane • Optical flow field: apparent motion of brightness patterns • We equate motion field with optical flow field Siggraph 04

25. 2 Cases Where this Assumption Clearly is not Valid (a) A smooth sphere is rotating under constant illumination. Thus the optical flow field is zero, but the motion field is not. (b) A fixed sphere is illuminated by a moving source—the shading of the image changes. Thus the motion field is zero, but the optical flow field is not. (a) (b) Siggraph 04

26. Brightness Constancy Equation • Let P be a moving point in 3D: • At time t, P has coords (X(t),Y(t),Z(t)) • Let p=(x(t),y(t)) be the coords. of its image at time t. • Let E(x(t),y(t),t) be the brightness at p at time t. • Brightness Constancy Assumption: • As P moves over time, E(x(t),y(t),t) remains constant. Siggraph 04

27. Brightness Constraint Equation short: Siggraph 04

28. Brightness Constancy Equation Taking derivative wrt time: Siggraph 04

29. Brightness Constancy Equation Let (Frame spatial gradient) (optical flow) (derivative across frames) and Siggraph 04

30. Brightness Constancy Equation Becomes: vy r E -Et/|r E| vx The OF is CONSTRAINED to be on a line ! Siggraph 04

31. Interpretation Values of (u, v) satisfying the constraint equation lie on a straight line in velocity space. A local measurement only provides this constraint line (aperture problem). Siggraph 04

32. Aperture Problem • Normal flow Siggraph 04

33. Recall the corner detector The matrix for corner detection: is singular (not invertible) when det(ATA) = 0 But det(ATA) = Õli = 0 -> one or both e.v. are 0 Aperture Problem ! One e.v. = 0 -> no corner, just an edge Two e.v. = 0 -> no corner, homogeneous region Siggraph 04

34. Optical Flow • Integrate over image patch • Solve Siggraph 04

35. Optical Flow, Feature Tracking Conceptually: rank(G) = 0 blank wall problem rank(G) = 1 aperture problem rank(G) = 2 enough texture – good feature candidates In reality: choice of threshold is involved Siggraph 04

36. Optical Flow • Previous method - assumption locally constant flow • Alternative regularization techniques (locally smooth flow fields, • integration along contours) • Qualitative properties of the motion fields Siggraph 04

37. Feature Tracking Siggraph 04

38. 3D Reconstruction - Preview Siggraph 04

39. Harris Corner Detector - Example Siggraph 04

40. Wide Baseline Matching Siggraph 04

41. Region based Similarity Metric • Sum of squared differences • Normalize cross-correlation • Sum of absolute differences Siggraph 04

42. Edge Detection original image gradient magnitude Canny edge detector • Compute image derivatives • if gradient magnitude >  and the value is a local maximum along gradient • direction – pixel is an edge candidate Siggraph 04

43. y   x Line fitting Non-max suppressed gradient magnitude • Edge detection, non-maximum suppression • (traditionally Hough Transform – issues of resolution, threshold • selection and search for peaks in Hough space) • Connected components on edge pixels with similar orientation • - group pixels with common orientation Siggraph 04

44. Line Fitting second moment matrix associated with each connected component • Line fitting Lines determined from eigenvalues and eigenvectors of A • Candidate line segments - associated line quality Siggraph 04

45. Take home messages • Correspondence is easy/difficult/impossible depending on the imaging constraints • Correspondence and reconstruction are tightly coupled problems, can be solved simultaneously [Jin et al., CVPR 2004] • For most scenes simple descriptors suffice to establish a few (50-500) corresponding points/lines • From now on just geometry Siggraph 04