1 / 108

Algorithms & Applications in Computer Vision

Algorithms & Applications in Computer Vision. Lihi Zelnik -Manor Lecture 9: Optical Flow. Last Time: Alignment. Homographies Rotational Panoramas RANSAC Global alignment Warping Blending. Today: Motion and Flow. Motion estimation Patch-based motion (optic flow)

griffith-parry
Télécharger la présentation

Algorithms & Applications in Computer Vision

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Algorithms & Applications in Computer Vision LihiZelnik-Manor Lecture 9: Optical Flow

  2. Last Time: Alignment • Homographies • Rotational Panoramas • RANSAC • Global alignment • Warping • Blending

  3. Today: Motion and Flow • Motion estimation • Patch-based motion (optic flow) • Regularization and line processes • Parametric (global) motion • Layered motion models

  4. Optical Flow Where did each pixel in image 1 go to in image 2

  5. Optical Flow Pierre Kornprobst's Demo

  6. Introduction • Given a video sequence with camera/objects moving we can better understand the scene if we find the motions of the camera/objects.

  7. Scene Interpretation • How is the camera moving? • How many moving objects are there? • Which directions are they moving in? • How fast are they moving? • Can we recognize their type of motion (e.g. walking, running, etc.)?

  8. Applications • Recover camera ego-motion. Result by MobilEye (www.mobileye.com)

  9. Applications • Motion segmentation Result by: L.Zelnik-Manor, M.Machline, M.Irani “Multi-body Segmentation: Revisiting Motion Consistency” To appear, IJCV

  10. Applications • Structure from Motion Input Reconstructed shape Result by: L. Zhang, B. Curless, A. Hertzmann, S.M. Seitz “Shape and motion under varying illumination: Unifying structure from motion, photometric stereo, and multi-view stereo” ICCV’03

  11. Examples of Motion fields Closer objects appear to move faster!! Forward motion Rotation Horizontal translation

  12. CCD 3D motion vector 2D optical flow vector Motion Field & Optical Flow Field • Motion Field = Real world 3D motion • Optical Flow Field = Projection of the motion field onto the 2d image

  13. The screen is stationary yet displays motion Homogeneous objects generate zero optical flow. Fixed sphere. Changing light source. Non-rigid texture motion When does it break?

  14. The Optical Flow Field Still, in many cases it does work…. • Goal:Find for each pixel a velocity vector which says: • How quickly is the pixel moving across the image • In which direction it is moving

  15. How do we actually do that?

  16. Estimating Optical Flow • Assume the image intensity is constant Time = t Time = t+dt

  17. Brightness Constancy Equation First order Taylor Expansion Simplify notations: Gradient and flow are orthogonal Divide by dt and denote: Problem I: One equation, two unknowns

  18. Time t Time t+dt Time t+dt Problem II: “The Aperture Problem” • For points on a line of fixed intensity we can only recover the normal flow ? Where did the blue point move to? We need additional constraints

  19. Use Local Information Sometimes enlarging the aperture can help

  20. Local smoothnessLucas Kanade (1984) Assume constant (u,v) in small neighborhood

  21. Goal: Minimize 2x2 2x1 2x1 Lucas Kanade (1984) Method: Least-Squares

  22. How does Lucas-Kanade behave? We want this matrix to be invertible. i.e., no zero eigenvalues

  23. How does Lucas-Kanade behave? • Edge  becomes singular

  24. How does Lucas-Kanade behave? • Homogeneous   0 eigenvalues

  25. How does Lucas-Kanade behave? • Textured regions  two high eigenvalues

  26. How does Lucas-Kanade behave? • Edge  becomes singular • Homogeneous regions  low gradients • High texture 

  27. Iterative Refinement • Estimate velocity at each pixel using one iteration of Lucas and Kanade estimation • Warp one image toward the other using the estimated flow field (easier said than done) • Refine estimate by repeating the process

  28. Optical Flow: Iterative Estimation estimate update Initial guess: Estimate: x x0

  29. x x0 Optical Flow: Iterative Estimation estimate update Initial guess: Estimate:

  30. x x0 Optical Flow: Iterative Estimation estimate update Initial guess: Estimate: Initial guess: Estimate:

  31. Optical Flow: Iterative Estimation x x0

  32. Optical Flow: Iterative Estimation • Some Implementation Issues: • Warping is not easy (ensure that errors in warping are smaller than the estimate refinement) • Warp one image, take derivatives of the other so you don’t need to re-compute the gradient after each iteration. • Often useful to low-pass filter the images before motion estimation (for better derivative estimation, and linear approximations to image intensity)

  33. Correlation based methods Regularization based methods Use multi-scale estimation Other break-downs • Brightness constancy is not satisfied • A point does not move like its neighbors • what is the ideal window size? • The motion is not small (Taylor expansion doesn’t hold) • Aliasing

  34. actual shift estimated shift Optical Flow: Aliasing Temporal aliasing causes ambiguities in optical flow because images can have many pixels with the same intensity. I.e., how do we know which ‘correspondence’ is correct? nearest match is correct (no aliasing) nearest match is incorrect (aliasing) To overcome aliasing: coarse-to-fine estimation.

  35. u=1.25 pixels u=2.5 pixels u=5 pixels u=10 pixels image It-1 image I Multi-Scale Flow Estimation image It image It+1 Gaussian pyramid of image It Gaussian pyramid of image It+1

  36. warp & upsample run Lucas-Kanade . . . image It-1 image I Multi-Scale Flow Estimation run Lucas-Kanade image It image It+1 Gaussian pyramid of image It Gaussian pyramid of image It+1

  37. Correlation based methods Regularization based methods Use multi-scale estimation Other break-downs • Brightness constancy is not satisfied • A point does not move like its neighbors • what is the ideal window size? • The motion is not small (Taylor expansion doesn’t hold) • Aliasing

  38. Spatial Coherence Assumption * Neighboring points in the scene typically belong to the same surface and hence typically have similar motions. * Since they also project to nearby points in the image, we expect spatial coherence in image flow. Black

  39. Formalize this Idea Noisy 1D signal: u x Noisy measurements p(x) Black

  40. Regularization Find the “best fitting” smoothed function v(x) u q(x) x Noisy measurements p(x) Black

  41. Membrane model Find the “best fitting” smoothed function v(x) u q(x) x Black

  42. Membrane model Find the “best fitting” smoothed function q(x) u q(x) x Black

  43. Membrane model Find the “best fitting” smoothed function q(x) u q(x) Black

  44. Spatial smoothness assumption Faithful to the data Regularization u q(x) x Minimize: Black

  45. Bayesian Interpretation Black

  46. Discontinuities u q(x) x What about this discontinuity? What is happening here? What can we do? Black

  47. velocity space u2 + + u1 Robust Estimation Noise distributions are often non-Gaussian, having much heavier tails. Noise samples from the tails are called outliers. • Sources of outliers (multiple motions): • specularities / highlights • jpeg artifacts / interlacing / motion blur • multiple motions (occlusion boundaries, transparency) Black

  48. Occlusion occlusion disocclusion shear Multiple motions within a finite region. Black

  49. Coherent Motion Possibly Gaussian. Black

  50. Multiple Motions Definitely not Gaussian. Black

More Related