Matching

1. Matching • Find a smaller image in a larger image • Applications • Find object / pattern of interest in a larger picture • Identify moving objects between a pair of pictures • Pre-process for multi-image recognition, e.g. stereo • Building panoramas from multiple images • You can’t just subtract! • Noise • Occlusion • Distortion

2. A simple matching algorithm • For each position / location / orientation of the pattern • Apply matching criteria • Local maxima of the criteria exceeding a preset threshold are considered matches

3. Match Criteria: Digital Difference • Take absolute value of pixel differences across the mask • “Convolution” with mask * -1 (match causes 0) • Not very robust to noise & slight variations • Best is 0, so algorithms would minimize rather than maximize • Very dark or very light masks will give larger mismatch values!

4. Match Criteria (Correlations) • All are inverted (and 1 is added to avoid divide by 0) • C1 - use largest distance in mask • C2 - use sum of distances in mask • C3 - use sum of squared distances in mask

5. Correlation: Weighted Summed Square Difference • Create a weighted mask. w(i,j) is a weight depending on the position in the mask. Multiply each weight by the square difference, then add them up. Sample weights (Gaussian) Source: www.cse.unr.edu/~bebis/CS791E/Notes/AreaProcess.pdf

6. Matching Strategies • Brute force (try all possibilities) • Hierarchical (coarse-to-fine) • Hierarchical (sub-patterns; graph match) • Optimized operation sequence • Short-circuit processing at mismatches for efficiency • Tracking (if possible)

7. Image Features for Matching • Interest points • Edges • Line segments • Curves

8. Phases of Image Feature Matching • Feature extraction (image region) • Feature description (invariant descriptor) • Feature matching (likely candidates in other images) • Feature tracking (likely candidates in appropriate region of next image)

9. Interest Points • Points that are likely to represent the same ‘location’ regardless of image • Points that are ‘easy’ to find and match • Example: (good and bad features)

10. Goals for Point Features • Useful features tend to persist through multiple scales • Texture elements ‘blur out’ and are no longer interesting above a certain feature • Corners tend to remain corner-like (and localized) over a wide variety of scales • Useful features tend to be invariant to rotation or small changes in camera angle • Avoid operators that are sensitive to rotation • Useful features tend to have a stable location • Avoid features in ‘smooth’ areas of the image • Avoid features along (single) edges

11. Good Localization of Features • Goal: unambiguous feature • Few spurious matches • Match is obviously strongest at the correct point • Result: look for ‘corners’ rather than featureless areas or edges (aperture problem) Figure 4.4

12. Evaluating Features for Localization • Autocorrelation • Measure correlation between feature patch and nearby locations in the same image • Good features will show a clear peak at the appropriate location • Ambiguous features will show ridges or no peak at all

13. Autocorrelation Example (Fig. 4.5)

14. Derivation of Autocorrelation Method • Equations directly from Chapter 4

15. Finding Features by Autocorrelation • Compute autocorrelation matrix A at each pixel • Compute the directional differences (derivatives) Ix and Iy • This is done by convolution with appropriate horizontal and vertical masks (derivatives of Gaussians) • For each point, compute 3 values: • A[0][0] = Ix*Ix • A[0][1] = A[1][0] = Ix*Iy, • A[1][1] = Iy*Iy, i.e. a 3-band image • Blur each of the above images using a bigger Gaussian than step 1 (this is applying the w function) • Compute interest measure based on eigenvalues of the A matrix

16. Finding Features by Autocorrelation (cont) • A has 2 eigenvalues, indicating direction of fastest & slowest change • Features are at local maxima of (smallest eigenvalue) that is above a threshold • See equations 4.10 and 4.11 for alternative measurements • Option: eliminate features that are too close to better ones (Non-maximum suppression)

17. Multiple Scale Image • Blur with bigger filters to get higher scales (less information) • Box filter = averaging • Gaussian filter is more flexible (arbitrary scales, though approximated) • Represent by ‘radius’ σ • Image is now a ‘pyramid’ since each blurred image is technically smaller scale • Higher (more blurred) images can be downsampled • Fewer pixels -> more efficient processing • Point coordinates in multi-scale image: (x, y, σ)

18. Matching Higher-level features • Numerical feature vectors (e.g. SIFT vectors) • Elementary geometric properties (boundaries) • Boundary length (perimeter) • Curvature: perimeter/ count of “direction change” pixels • Chord distribution (lengths & angles of all chords) • Complete boundary representations • Polygonal approximation (recursive line-split algorithm) • Contour partitioning (curvature primal sketch) • B-Splines

19. Components of A Matching System • Match Strategy • Which correspondence should be considered for further processing? • Answer partially depends on application • Stereo – we know where to look • Stitching / tracking – expect many matches in overlap region • Recognition of models in cluttered image – most ‘matches’ will be false! • Basic assumptions • Descriptor is a vector of numbers • Match based on Euclidean distance (vector magnitude) • Efficient Data Structures & Algorithms for Matching

20. Match Strategy: When do they match? • Exact (numerical) match (NOT PRACTICAL) • Match within tolerance, i.e. threshold feature distance • Choose the best match (nearest neighbor) as long as it’s “not too bad” • Choose the nearest neighbor only when it is sufficiently nearer than the 2nd nearest neighbor (NNDR) Figure 4.24

21. Evaluating Matching:False Positives and False Negatives Figure 4.22 TPR (recall) = TP / (TP + FN) or TP / P FPR = FP / (FP + TN) or FP / N PPV (precision) = TP / (TP + TN) or TP / P’ ACC = TP + TN / (TP + FP + TN + FN)

22. Verification • Given model and transformation • Compute locations of model features in the image (e.g. corners or edges) • For each feature, determine whether sufficient evidence (e.g. "edge pixels") supports the line • If enough features are verified by evidence, accept the transformation

23. Pose Estimation (6.1) • Estimate camera’s relative position & orientation to a scene • Least squares formulation • Given: set of matches (x, x’) and a transformation f(x; p) • Determine how well (poorly) f(x; p) = x’ for all x,x’ pairs • Note: p are parameters of transform • Optimization problem: find p so that the following function is minimized • (Take the derivative of the sum and set it to 0, then solve for p) – see optimization.ppt

24. Local Feature Focus Method • In model, identify focus features • These are expected to be easy to identify in the image • For each focus feature, also include nearby features • Matching • Find a focus feature • Find as many nearby features as possible • Determine correspondences (focus & nearby features) • Compute a transformation • Verify the transformation with all features (next slide)

25. RANSAC (RANdom SAmple Consensus) Method • Start with a small (random) set of independent matches (hypothetical, putative) • Determine a transformation that fits this small set of matches • [ Remove ‘outliers’ and refine the transformation ] • [ Find more matches that are consistent with the original set ] • If the result isn’t good enough, choose another set and try again…

26. Pose Clustering • Select the minimal number of control points necessary to determine a pose, based on transformation • E.g. 2 points to find translation + rotation + scale • For each set of points, consider all possible matches to the model • For each correspondence, "vote" for the appropriate transformation (like Hough transform) • Transformations with the most votes "win" • Verify winning transformations in the image

27. Problems with Pose Clustering • Too many correspondences • Imagine 10 points from image, 5 points in model • If all are considered, we have 45 * 10 = 450 correspondences to consider! • In general N image points, M model points yields (N choose 2)*(M choose 2), or (N*(N-1)*M*(M-1))/4 correspondences to consider! • Can we limit the pairs we consider? • Accidental peaks • Just like the regular Hough transform, some peaks can be "conspiracies of coincidences" • Therefore, we must verify all "reasonably large" peaks

28. Improving Pose Clustering • Classify features and use types to match • T-junction vs. L-junction • Round vs. square hole • Use higher-level features • Ribbons instead of line segments • "Constellations of points" instead of points • Junctions instead of points or edges • Use non-geometric information • Color • Texture

29. Cross Ratio: Invariant of Projection • Consider four rays “cut” by two lines • I = (A-C)(B-D) / (A-D)(B-C)

30. Geometric Hashing • What if there are many models to match? • Separate the problem • Offline preprocessing (of models) • Create a hash table of (model, feature set) pairs, where the model-feature set has a given geometry • E.g. 3-point sets hashed by 3rd point's coordinate in terms fo the other two • Online recognition (of image) and pose determination • Find a feature set (e.g. 3 points) • Vote for the appropriate (model, feature set) pairs • Verify high-voting models

31. Hallucination • Whenever you have a model, you have the ability to find it, even in a data set consisting of random points (!) • Subset of points can pass point verification test, even if they don't correspond to the model • Occlusion can make 2 overlapping objects appear like another one • Solution: careful and complete verification (e.g. verify by line segments, not just points)

32. Structural Matching • Recast the problem as "consistent labeling" • A consistent labeling is an assignment of labels to parts that satisfies: • If Pi and Pj are related parts, then their labels f(Pi), f(Pj) are related in the same way • Example: if two segments are connected at a vertex in the model, then the respective matching segments in the image must also be connected at a vertex

33. Interpretation Tree (empty) A=c A=a A=b B=b B=c B=a B=c B=a B=b Each branch is a choice of feature-label match Cut off branch (and all children) if a constraint is violated

34. Constraints on Correspondences (review) • Unary constraints are direct measurements • Round hole vs. square hole • Big hole vs. small hole (relative to some other measurable distance) • Red dot vs. green dot • Binary constraints are measurements between 2 features • Distance between 2 points (relative…) • Angle between segments defined by 3 points • Higher order constraints might measure relationships among 3 or more features

35. Searching the Interpretation Tree • Depth-first search (recursive backtracking) • Straightforward, but could be time-consuming • Heuristic (e.g. best-first) search • Requires good guesses as to which branch to expand next • (Specifics are covered in Artificial Intelligence) • Parallel Relaxation • Each node gets all labels • Every constraint removes inconsistent labels • (Review neural net slides for details)

36. Cross Ratio Examples • Two images of one object makes 2 matching cross ratios! • Dual of cross ratio: four lines from a point instead of four points on a line • Any five non-collinear but coplanar points yield two cross-ratios (from sets of 4 lines)

37. Using Invariants for Recognition • Measure the invariant in one image (or on the object) • Find all possible instances of the invariant (e.g. all sets of 4 collinear points) in the (other) image • If any instance of the invariant matches the measured one, then you (might) have found the object • Research question: to what extent are invariants useful in noisy images?