Object Recognition using Local Affine Frames on Maximally Stable Extremal Regions

1. Object Recognition using Local Affine Frames on Maximally Stable Extremal Regions Stepan Obdrzalek Jirı Matas

2. Proposed Algorithm Identify affine-covariant regions of interest MSER detector Construct local affine frames (LAFs) Invariant to geometry and photometrics Normalize LAF geometry and color Generate descriptors of patches Discrete cosine transformation Recognition & Localization Establish tentative correspondences Find a globally consistent subset Infer presence and location of object

3. Requirement for Region Detectors Consistent Discriminative Invariant (actually: covariant) Appearance is consistent with the transformation scaling, rotation, shearing Fixed shape is insufficient Shape must be covariant to object position (Sticky)

4. Popular Affine Covariant Detectors Harris-Affine Hessian-Affine Edge Intensity Extrema Salient Regions MSER

5. Harris-affine & Hessian-affine Detect interest points Identify corners in image using Harris corner detector Determine the “characteristic” scale Maximization of Laplacian-of-Gaussians Determine an elliptical region for each point Second moment matrix

6. Edge based detector Edges are stable across view, scale, illumination Detect interest points Identify corners in image using Harris corner detector Identify edges using canny Combine to form a parallelogram Determine the “characteristic” scale Parallelograms where textures hit an extremum

7. Intensity based detector Detect interest points Identify local extremum in intensity Analyze rays projecting radially Determine the “characteristic” scale Best-fit ellipse that passes through ray-points with large intensity shifts

8. Salient region detector Based on PDF of intensity values computed over elliptical region Detect interest points Measure the pixel entropy within elliptical regions Select regions with high “complexity” Determine the “characteristic” scale Optimal scale is determined by the identified region

9. Maximally Stable Extremal Region (MSER) Connected component of thresholded image Efficient to implement O(number pixels) Detect interest points All pixels inside the MSER have higher or lower intensities than in the surrounding regions Regions are selected to be stable over intensity range Determine the “characteristic” scale Optimal scale is automatic to MSER algorithm

10. Runtime comparison

15. Local Affine Frame (LAF) from Features Comparing transformed image regions can be simplified by constructing a viewpoint invariant coordinate system that is feature-based Coordinates are based on local features Coordinates “stick” to features Features must describe 6 degrees of freedom Simple points and ellipses are not sufficient MSER regions are sufficient Assumptions Local planarity Perspective camera

16. Local Affine Frame (LAF) from Features

17. Local Affine Frame (LAF) from Features 2D affine transformation has 6 degrees of freedom 6 independent constraints must be found Correspondence of 3 non-collinear points Constraints are derived from detected primitives

18. Local Affine Frame (LAF) from Features Region shape constructions Center of gravity 2 constraints: resolves translation 2x2 covariance matrix ∑(ii) 3 constraints: Together with COG, fixes affine up to unknown rotation Concavities 4 constraints: line and point tangent to line Don’t require detection of whole region Curvature inflection points From concave to convex Straight line segments of boundary

19. Local Affine Frame (LAF) from Features Intensity Constructions: pixels inside a region Orientations of gradients Rotation Direction of dominant texture periodicity Rotaion Extrema of RGB or any scalar function 2 constraints

20. Local Affine Frame (LAF) from Features Topology of regions: Mutual configuration of regions Nested regions Neighboring regions Holes Incident regions

21. LAF Construction Construction of primitives covering 6 degrees of freedom

22. Geometric Normalization Translate between canonical / image frame Origin = (0,0)T, Basis Vectors = (1,0)T, (0,1)T Measurement Region (MR) Image region used to determine local correspondences (-2,3) x (-2,3)

23. Photometric Normalization Translate between canonical / image frame Reflections and shadows are ignored Illumination, gain, aperture, etc. is modeled by affine transformations of color channels Transformation between two patches I and I’ is: Requires 6 additional normalization parameters Intensities are affinely transformed to have zero mean unit variance

24. Normalization of Local Representation Translate between canonical / image frame 12 normalization parameters stored with the descriptor Coverage

25. Descriptors Desirable properties Distinguish between large number of regions Maximize ratio of similarities between match & mismatch Robust or invariant to localization errors & transformations Efficient on memory and speed Discrete Cosine Transformation (JPEG compression) Algorithms require O(n lg n) Hardware implementations Robust to misalignment Same discrimination as SIFT

26. Matching detected frames with query frames Comparison Compute similarities between all detected and query frames Matching Select most likely matches Verification Consistency check that incorporates geometric constraints

27. Comparison Determine the probability that a transformation can take place Based on training experience If probability is below a threshold, ∞ similarity Otherwise, determined by descriptor similarity

28. Matching Nearest Match Most common For each detected frame, find closest query frame Mutually Nearest Match For symmetric matching (e.g. stereo) For each detected, find closest query For each query, find closest detected Match if (close query = close detected) or (diff < threshold) All (or N most) similar Repetitive structures (many ambiguous correspondences) Keep all correspondences, resolution left to verification High number of false correspondences

29. Verification All matches should be consistent with same model 3D models would only be effective if visible parts of the image are very large (building interiors) Sufficient to model as planar surfaces If 2 tentative correspondences are part of the same plane Similar geometric transformation Similar photometric transformation Set of all correspondences is decomposed into subsets of consistent correspondences Each subset represents a single plane in the scene Small sets are rejected

30. Experimental Validation: COIL-100 100 objects 72 images each object 5º pose intervals Controlled lighting

31. Experimental Validation: ZuBuD 201 buildings 5 pictures each

32. Experimental Validation: FOCUS Product logos Logos occupy small image portion 360 color images

33. Conclusion Object recognition based on local measurements Affine invariance achieved by expressing local appearance in terms of affine covariant coordinates Promising results Problems Speed is the primary issue All query compared to all database Speed improved using hashing, cost may be accuracy Planar surface assumption Rigid objects Shadow, etc.