Feature Detection Giacomo Boracchi 6/12/2007 boracchi@elet.polimi.it
Feature matching vs. tracking • Image-to-image correspondences are determined by some salient point. • These are also the basis of passive triangulation-based 3D reconstruction • Feature Matching: Extract features independently and then match by comparing descriptors • Feature Tracking: Extract features in first images and then try to find same feature back in next view What is a good feature?
Feature Properties • Well-defined: i.e. neighboring points should all be different • Stable across views: same scene point should be extracted as feature for neighboring viewpoints
Interest Point Detection Low Level Inspired • Gradient Based (ex Harris, Hessian) • Phase Based (Kovesi) • Entropy Based (Zisserman)
Comparing image regions: Pixel by Pixel • Dissimilarity Measure SSD the Sum of Squared Distances I´(x,y) I(x,y)
Harris – Moravec • Compute the Sum of Square Distances between the image values on the green square at different position “flat” region:no change in all directions “edge”:no change along the edge direction “corner”:significant change in all directions
2 ( ) P ( ) ( ( ) ( ) ) E I I ¡ ¡ ¡ x y w u v u v u x v y = ; ; ; ; u v Moreavec (80) • Response on sliding windows – Moravec (81) • w represent the window (green in the previous slides) • Corner, even small changes are “big” • where is a threshold • Look for local maxima in min{E} above some threshold
µ ¶ 2 ( ( ) ) P ( ) ( ( ) ( ) ) r E E I I 0 0 0 x ¡ ¡ ¡ x y w u v u v u x v y ¡ ¢ ¡ ¢ 1 = = ( ) ( ) E E r E H 0 0 + + ; ; ; ; ; ¼ u x y x y x y v ; ; 2 y Moravec Drawbacks – Solutions • The response is isotropic as only a finite set of displacements (x,y) is considerer • therefore, the same corner rotated may yield different responses. • Solution: Analytical Formulation of E(x,y), expand it in Taylor series • Now E(0,0) = 0 and one can prove that also the term vanishes • H denotes the Hessian Matrix but when it is expressed as a function of I we call it M
µ ¶ x ¡ ¢ ( ) E M ¼ x y x y ; y Moravec Drawbacks – Solutions • Now we can approximatebeing the derivative computed with Sobel or Previtt filters
2 “Edge” 2 >> 1 “Corner”1 and 2 are large,1 ~ 2;E increases in all directions “Flat” region “Edge” 1 >> 2 1 Moravec Drawbacks – Solutions • Considering only the minimum of E is not a great deal, may give too ready responses • Solution consider the SVDof M and pretend that theminimum of the eigenvalues of M is big
Moravec Drawbacks – Solutions • The response may be noisy • Solution: take w as Gaussian distributed Weights
Harris – Stevens (88) • To avoid eigenvalue decomposition 2 “Edge” 2 >> 1 “Corner”1 and 2 are large,1 ~ 2;E increases in all directions “Flat” region “Edge” 1 >> 2 1 Alternatively
Feature point extraction Find points that differ as much as possible from all neighboring points homogeneous edge corner
Feature point extraction homogeneous edge corner Find points for which the following is maximum i.e. maximize smallest eigenvalue of M
Harris corner detector as Feature Selector • Select the local maxima of Harris Measure as Features to perform Matching • Only use local maxima, subpixel accuracy through second order surface fitting
Harris – Stevens (88) : Feature extraction • Pro • Fast • Rotation invariant • Shortcomings • No scale invariant • No affine transform invariant
FeatureTracking Giacomo Boracchi 6/12/ 2007
Comparing image regions: Pixel by Pixel • Dissimilarity Measure SSD the Sum of Squared Distances I´(x,y) I(x,y)
Comparing image regions: Pixel by Pixel • Similarity measures I´(x,y) I(x,y)
Normalized Cross Correlation • Gives a measure between [-1,1] • While the SSD is only positive and not normalized • Independent on Image Intensity • Allows fast implementation using running sum to compute local averages
Tracking Problems • Occlusions • Feature Deformations due to motions • Affine • Perspective • Light Changes (not for NCC based measures) • Noise • Blur • Other Artifacts (e.g. Jpeg compression)
( ( ) ) ( ( ) ) ( ) » » d » l ± » I I t t t + + + x y x y ¿ x a n ´ y ´ ´ x y ¿ ¿ e ´ = = = = ; ; ; ; ; ; ; ; ; KLT – Shi and Tomasi 94 • Take into account also for affine changes in the image • Consider a Video I(x,y,t) it’s the value of the pixel x,y in the image at time t • Then how to relate the same intensity in two different images • And extending such requirement to a neighbor we get feature matching • And, of course
[ ] d d d = x y ; KLT – Shi and Tomasi 94: Feature Tracking • Being the displacement represented by an affine motion field • Consider then only two frames I,J • Affine motion model • Translation only
KLT – Shi and Tomasi 94: Then what Tracking actually is • Given two images I and J we are interested in determiningthe six parameters (4 affine + 2 translational motion) • Then for consequent frame a pure translation model is commonly assumed (i.e. D= Identity)
d d f f k k b b d d T Z T Z 6 2 t t t t z a e z v v e e c c o o r r o o u u n n n n o o w w n n s s e a c c a a n n e e c c o o m m p p u u e e = = , , , , KLT – Shi and Tomasi 94 : Dissimilarity measure • Goal : minimize dissimilarity (SSD based tracking) • A and d can be determined in a closed form (approximate solutions) • An exact solution can be obtained by iterative procedure • And also for the translation only case
KLT – Shi and Tomasi 94: Which are Good features? • ”The right features are exactly those that make the tracker work best” • How the tracker work? • Inverting 2 matrices Z and T • Then Good features are those having the matrices Z and T well conditioned (then again eigenvalues analysis) • Thus, no a priori assumption about feature goodness • Affine model also considered
KLT – Shi and Tomasi 94 : Performances • Adjacent frames – pure displacement Z matrix • Cumulative changes – Affine Transform T matrix
KLT – Shi and Tomasi 94 : Performances • Camera moving forward 2mm per frame • Affine alignment between first and last frame • Stop tracking features with too large errors
KLT – Shi and Tomasi 94 : Performances • Dissimilarity computed using pure displacement. • Dissimilarity computed using affine transforms
A Project: Rotational Blur Removal Giacomo Boracchi
Rotational Blur • In each pixel the blur smears are varying • Only in trivial case of rotation axis orthogonal to image plane this can be considered circumferences
Rotation Axis Estimation: the local blur analysis
Rotation Axis Estimation: the voting procedure
2 ¼ P C ´ ¼ ¼ Blurring paths • blurring path the set of pixels which are intersected by a viewing ray after a rotation of around the axis a C a V P V a circumferences Conic sections
Blur Removal: Scheme and Issues • Projection of the image on the plane orthogonal to rotation axis • In such a way the blurring paths become circumferences • The mapping (an homography) should be extremely accurate • Cartesian to Polar coordinate transform so that the blur becomes shift invariant, i.e. uniform, all the pixels blurred at the same way • Blur Inversion via Regularized Wiener Filtering (Classical Procedure) • Polar to Cartesian coordinate transform • Denoising in Cartesian domain in order to exploit image structures • Using LASIP adaptive filtering algorithms (codes) • Noise Modeling via Monte Carlo approach
License Plate Recognition • Objective: automatically detect and recognize license plate from video sequences on a dsp-equipped camera. YZH 4025
License Plate Recognition – Available Projects • New Starting Project • Probably Strict Deadlines • ONLY THE BRAVES! • License Plate Detection Module • Develop a module that detect the license plate in image according to color and shape • License Plate Recognition Module • Given the license plate image, develop a module that recognize characters (e.g. using a neural network) and provide the license number • Both projects require good C-programming skills • Coordinators: Caglioti, Gasparini, Taddei
Computer Vision Group Team • Vincenzo Caglioti • Giacomo Boracchi, Simone Gasparini, Alessandro Giusti, Pierluigi Taddei
