CSCE 643 Computer Vision: Extractions of Image Features

1. CSCE 643 Computer Vision:Extractions of Image Features Jinxiang Chai

2. Good Image Features • What are we looking for? • Strong features • Invariant to changes (affine and perspective, occlusion, illumination, etc.)

3. Feature Extraction Why do we need to detect features? - Features correspond to important points in both the world and image spaces - Object detection/recognition - Solve the problem of correspondence • Locate an object in multiple images (i.e. in video) • Track the path of the object, infer 3D structures, object and camera movement

4. Outline Image Features - Corner detection - SIFT extraction

5. What are Corners? Point features

6. What are Corners? Point features Where two edges come together Where the image gradient has significant components in the x and y direction We will establish corners from the gradient rather than the edge images

7. Basic Ideas What are gradients along x and y directions?

8. Basic Ideas What are gradients along x and y directions?

9. Basic Ideas What are gradients along x and y directions? How to measure corners based on the gradient images?

10. Basic Ideas What are gradients along x and y directions? How to measure corners based on the gradient images? How to measure corners based on the gradient images? - two major axes in the local window!

11. How to Find Two Major Axes? • Principal component analysis (PCA)

12. How to Find Two Major Axes? • Principal component analysis (PCA) The length of two major axes is dependent on the ration of eigen values (λ1/λ2 ).

13. Corner Detection Algorithm 1. Compute the image gradients 2. Define a neighborhood size as an area of interest around each pixel 3x3 neighborhood

14. Corner Detection Algorithm (cont’d) • For each image pixel (i,j), construct the following matrix from it and its neighborhood values e.g. Similar to covariance matrix (Ix,Iy)T!

15. Corner Detection Algorithm (cont’d) • For each matrix C(i,j), determine the 2 eigenvalues λ(i.j)=[λ1, λ2]. Simple case: • This means dominant gradient direction aligns with x or y axis. • If either λ1 or λ2 is close to zero, then this is not a corner.

16. Corner Detection Algorithm (cont’d) • For each matrix C(i,j), determine the 2 eigenvalues λ(i.j)=[λ1, λ2]. Simple case: Interior Region Edge Corner Isolated pixels Large λ1 and small λ2 Large λ1 and large λ2 small λ1 and small λ2 λ1, λ2=0

17. Corner Detection Algorithm (cont’d) • For each matrix C(i,j), determine the 2 eigenvalues λ(i.j)=[λ1, λ2]. General case: • This is just a rotated version of the one on last slide • If either λ1 or λ2 is close to zero, then this is not a corner. • invariant to 2D rotation

18. Eigen-values and Corner • λ1 is large • λ2 is large

19. Eigen-values and Corner • λ1 is large • λ2 is small

20. Eigen-values and Corner • λ1 is small • λ2 is small

21. Corner Detection Algorithm (cont’d) • For each matrix C(i,j), determine the 2 eigenvalues λ(i.j)=[λ1, λ2]. 5. If both λ1 and λ2 are big, we have a corner (Harris also checks the ratio of λs is not too high) ISSUE: The corners obtained will be a function of the threshold !

22. Image Gradients

23. Image Gradients Closeup of image orientation at each pixel

24. The Orientation Field Corners are detected where both λ1 and λ2 are big

25. The Orientation Field Corners are detected where both λ1 and λ2 are big

26. Corner Detection Sample Results Threshold=25,000 Threshold=10,000 Threshold=5,000

27. Outline Image Features - Corner detection - SIFT extraction

28. Scale Invariant Feature Transform (SIFT) • Choosing features that are invariant to image scaling and rotation • Also, partially invariant to changes in illumination and 3D camera viewpoint

29. Motivation for SIFT • Earlier Methods • Harris corner detector • Sensitive to changes in image scale • Finds locations in image with large gradients in two directions • No method was fully affine invariant • Although the SIFT approach is not fully invariant it allows for considerable affine change • SIFT also allows for changes in 3D viewpoint

30. Invariance • Illumination • Scale • Rotation • Affine

31. Readings • Object recognition from local scale-invariant features [pdf link], ICCV 09 • David G. Lowe, "Distinctive image features from scale-invariant keypoints,"International Journal of Computer Vision, 60, 2 (2004), pp. 91-110

32. SIFT Algorithm Overview • Scale-space extrema detection • Keypoint localization • Orientation Assignment • Generation of keypoint descriptors.

33. Scale Space • Different scales are appropriate for describing different objects in the image, and we may not know the correct scale/size ahead of time.

34. Scale space (Cont.) • Looking for features (locations) that are stable (invariant) across all possible scale changes • use a continuous function of scale (scale space) • Which scale-space kernel will we use? • The Gaussian Function

35. Scale-Space of Image • variable-scale Gaussian • input image

36. Scale-Space of Image • variable-scale Gaussian • input image • To detect stable keypoint locations, find the scale-space extrema in difference-of-Gaussian function

37. Scale-Space of Image • variable-scale Gaussian • input image • To detect stable keypoint locations, find the scale-space extrema in difference-of-Gaussian function

38. Scale-Space of Image • variable-scale Gaussian • input image • To detect stable keypoint locations, find the scale-space extrema in difference-of-Gaussian function Look familiar?

39. Scale-Space of Image • variable-scale Gaussian • input image • To detect stable keypoint locations, find the scale-space extrema in difference-of-Gaussian function Look familiar? -bandpass filter!

40. Difference of Gaussian • A = Convolve image with vertical and horizontal 1D Gaussians, σ=sqrt(2) • B = Convolve A with vertical and horizontal 1D Gaussians, σ=sqrt(2) • DOG (Difference of Gaussian) = A – B • So how to deal with different scales?

41. Difference of Gaussian • A = Convolve image with vertical and horizontal 1D Gaussians, σ=sqrt(2) • B = Convolve A with vertical and horizontal 1D Gaussians, σ=sqrt(2) • DOG (Difference of Gaussian) = A – B • Downsample B with bilinear interpolation with pixel spacing of 1.5 (linear combination of 4 adjacent pixels)

42. B1 A1 Difference of Gaussian Pyramid A3-B3 Blur B3 DOG3 A3 Downsample A2-B2 B2 Blur DOG2 A2 Input Image Downsample A1-B1 Blur DOG1 Blur

43. Other issues • Initial smoothing ignores highest spatial frequencies of images

44. Other issues • Initial smoothing ignores highest spatial frequencies of images - expand the input image by a factor of 2, using bilinear interpolation, prior to building the pyramid

45. Other issues • Initial smoothing ignores highest spatial frequencies of images - expand the input image by a factor of 2, using bilinear interpolation, prior to building the pyramid • How to do downsampling with bilinear interpolations?

46. Bilinear Filter Weighted sum of four neighboring pixels x u y v

47. Bilinear Filter y Sampling at S(x,y): (i,j) (i,j+1) u x v (i+1,j+1) (i+1,j) S(x,y) = a*b*S(i,j) + a*(1-b)*S(i+1,j) + (1-a)*b*S(i,j+1) + (1-a)*(1-b)*S(i+1,j+1)

48. Bilinear Filter y Sampling at S(x,y): (i,j) (i,j+1) u x v (i+1,j+1) (i+1,j) S(x,y) = a*b*S(i,j) + a*(1-b)*S(i+1,j) + (1-a)*b*S(i,j+1) + (1-a)*(1-b)*S(i+1,j+1) To optimize the above, do the following Si = S(i,j) + a*(S(i,j+1)-S(i)) Sj = S(i+1,j) + a*(S(i+1,j+1)-S(i+1,j)) S(x,y) = Si+b*(Sj-Si)

49. Bilinear Filter y (i,j) (i,j+1) x (i+1,j+1) (i+1,j)

50. Pyramid Example A3 DOG3 B3 A2 B2 DOG3 A1 B1 DOG1