1 / 93

Visual Motion Estimation Problems & Techniques

Visual Motion Estimation Problems & Techniques. Princeton University COS 429 Lecture Feb. 12, 2004. Harpreet S. Sawhney hsawhney@sarnoff.com. Outline. Visual motion in the Real World The visual motion estimation problem Problem formulation: Estimation through model-based alignment

Télécharger la présentation

Visual Motion Estimation Problems & Techniques

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript


  1. Visual Motion EstimationProblems & Techniques Princeton University COS 429 Lecture Feb. 12, 2004 Harpreet S. Sawhney hsawhney@sarnoff.com

  2. Outline • Visual motion in the Real World • The visual motion estimation problem • Problem formulation: Estimation through model-based alignment • Coarse-to-fine direct estimation of model parameters • Progressive complexity and robust model estimation • Multi-modal alignment • Direct estimation of parallax/depth/optical flow • Glimpses of some applications

  3. Types of Visual Motionin theReal World

  4. Simple Camera Motion : Pan & Tilt Camera Does Not Change Location

  5. Apparent Motion : Pan & Tilt Camera Moves a Lot

  6. Independent Object Motion Objects are the Focus Camera is more or less steady

  7. Independent Object MotionwithCamera Pan Most common scenario for capturing performances

  8. General Camera Motion Large changes in camera location & orientation

  9. Visual Motion due to Environmental Effects Every pixel may have its own motion

  10. The Works! General Camera & Object Motions

  11. Why is Analysis and EstimationofVisual Motion Important?

  12. Foreground Vs. Background Visual Motion Estimationas a means of extractingInformation Content in Dynamic Imagery...extract information behind pixel data...

  13. Foreground Vs. Background Extended Scene Geometry Information Content in Dynamic Imagery...extract information behind pixel data...

  14. Temporal Persistence Foreground Vs. Background Extended Scene Geometry Segment,Track,Fingerprint Moving Objects Layers with 2D/3D Scene Models Layers & Mosaics Direct method based Layered, Motion, Structure & Appearance Analysis provides Compact Representation for Manipulation & Recognition of Scene Content Information Content in Dynamic Imagery...extract information behind pixel data...

  15. An ExampleA Panning Camera • Pin-hole camera model • Pure rotation of the camera • Multiple images related through a 2D projective transformation: also called a homography • In the special case for camera pan, with small frame-to-frame rotation, and small field of view, the frames are related through a pure image translation

  16. Pin-hole Camera Model Y y Z f

  17. Camera Rotation (Pan) Y’ y’ Z’ f

  18. Camera Rotation (Pan) Y’’ f Z’’ y’’

  19. Image Motion due to Rotationsdoes not depend on the depth / structure of the scene Verify the same for a 3D scene and 2D camera

  20. Pin-hole Camera Model Y y Z f

  21. Camera Translation (Ty) Y y X X X X Z f

  22. Translational Displacement Image Motion due to Translation is a function of the depth of the scene

  23. Sample Displacement Fields Render scenes with various motions and plot the displacement fields

  24. Motion Field vs. Optical Flow Y wy Motion Field : 2D projections of 3D displacement vectors due to camera and/or object motion Ty wx X p’ P’ p Tx P wz Z Optical Flow : Image displacement field that measures the apparent motion of brightness patterns Tz

  25. Motion Field vs. Optical Flow Lambertian ball rotating in 3D Motion Field ? Optical Flow ? Courtesy : Michael Black @ Brown.edu Image: http://www.evl.uic.edu/aej/488/

  26. Motion Field vs. Optical Flow Stationary Lambertian ball with a moving point light source Motion Field ? Optical Flow ? Courtesy : Michael Black @ Brown.edu Image : http://www.evl.uic.edu/aej/488/

  27. A Hierarchy of ModelsTaxonomy by Bergen,Anandan et al.’92 • Parametric motion models • 2D translation, affine, projective, 3D pose[Bergen, Anandan, et.al.’92] • Piecewise parametric motion models • 2D parametric motion/structure layers [Wang&Adelson’93, Ayer&Sawhney’95] • Quasi-parametric • 3D R, T & depth per pixel. [Hanna&Okumoto’91] • Plane+parallax [Kumar et.al.’94, Sawhney’94] • Piecewise quasi-parametric motion models • 2D parametric layers + parallax per layer [Baker et al.’98] • Non-parametric • Optic flow: 2D vector per pixel [Lucas&Kanade’81,Bergen,Anandan et.al.’92]

  28. Sparse/Discrete Correspondences&Dense Motion Estimation

  29. Discrete MethodsFeature Correlation&RANSAC

  30. Visual Motion through Discrete Correspondences Images may be separated by time, space, sensor types In general, discrete correspondences are related through a transformation

  31. Discrete MethodsFeature Correlation&RANSAC

  32. Discrete Correspondences • Select corner-like points • Match patches using Normalized Correlation • Establish further matches using motion model

  33. Direct Methods for Visual Motion EstimationEmploy Models of Motionand Estimate Visual MotionthroughImage Alignment

  34. Characterizing Direct MethodsThe What • Visual interpretation/modeling involves spatio-temporal image representations directly • Not explicitly represented discrete features like corners, edges and lines etc. • Spatio-temporal images are represented as outputs of symmetric or oriented filters. • The output representations are typically dense, that is every pixel is explained, • Optical flow, depth maps. • Model parameters are also computed.

  35. Direct Methods : The How Alignment of spatio-temporal images is a means of obtaining : Dense Representations, Parametric Models

  36. Direct Method based Alignment

  37. Images separated by time, space, sensor types Formulation of Direct Model-based Image Alignment[Bergen,Anandan et al.’92] Model image transformation as : Brightness Constancy

  38. Reference Coordinate System Formulation of Direct Model-based Image Alignment Model image transformation as : Images separated by time, space, sensor types

  39. Generalized pixel Displacement Formulation of Direct Model-based Image Alignment Model image transformation as : Images separated by time, space, sensor types Reference Coordinate System

  40. Model Parameters Formulation of Direct Model-based Image Alignment Model image transformation as : Images separated by time, space, sensor types Reference Coordinate System Generalized pixel Displacement

  41. Formulation of Direct Model-based Image Alignment Model image transformation as : Images separated by time, space, sensor types Reference Coordinate System Generalized pixel Displacement Model Parameters

  42. Filtered Image Representations (to account for Illumination changes, Multi-modalities) Formulation of Direct Model-based Image Alignment Compute the unknown parameters and correspondences while aligning images using optimization : What all can be varied ?

  43. Model Parameters Filtered Image Representations (to account for Illumination changes, Multi-modalities) Formulation of Direct Model-based Image Alignment Compute the unknown parameters and correspondences while aligning images using optimization : What all can be varied ?

  44. Measuring mismatches (SSD, Correlations) Model Parameters Filtered Image Representations (to account for Illumination changes, Multi-modalities) Formulation of Direct Model-based Image Alignment Compute the unknown parameters and correspondences while aligning images using optimization : What all can be varied ?

  45. Optimization Function Measuring mismatches (SSD, Correlations) Model Parameters Filtered Image Representations (to account for Illumination changes, Multi-modalities) Formulation of Direct Model-based Image Alignment Compute the unknown parameters and correspondences while aligning images using optimization : What all can be varied ?

  46. Formulation of Direct Model-based Image Alignment Compute the unknown parameters and correspondences while aligning images using optimization : Optimization Function Measuring mismatches (SSD, Correlations) Model Parameters Filtered Image Representations (to account for Illumination changes, Multi-modalities) What all can be varied ?

  47. A Hierarchy of ModelsTaxonomy by Bergen,Anandan et al.’92 • Parametric motion models • 2D translation, affine, projective, 3D pose[Bergen, Anandan, et.al.’92] • Piecewise parametric motion models • 2D parametric motion/structure layers [Wang&Adelson’93, Ayer&Sawhney’95] • Quasi-parametric • 3D R, T & depth per pixel. [Hanna&Okumoto’91] • Plane+parallax [Kumar et.al.’94, Sawhney’94] • Piecewise quasi-parametric motion models • 2D parametric layers + parallax per layer [Baker et al.’98] • Non-parametric • Optic flow: 2D vector per pixel [Lucas&Kanade’81,Bergen,Anandan et.al.’92]

  48. Plan : This Part • First present the generic normal equations. • Then specialize these for a projective transformation. • Sidebar into backward image warping. • SSD and M-estimators.

  49. An Iterative Solution of Model Parameters[Black&Anandan’94 Sawhney’95] • Given a solution at the mth iteration, find by solving : is a weight associated with each measurement.

More Related