1. Visual Motion EstimationProblems & Techniques Princeton University COS 429 Lecture Feb. 12, 2004 Harpreet S. Sawhney hsawhney@sarnoff.com

2. Outline • Visual motion in the Real World • The visual motion estimation problem • Problem formulation: Estimation through model-based alignment • Coarse-to-fine direct estimation of model parameters • Progressive complexity and robust model estimation • Multi-modal alignment • Direct estimation of parallax/depth/optical flow • Glimpses of some applications

3. Types of Visual Motionin theReal World

4. Simple Camera Motion : Pan & Tilt Camera Does Not Change Location

5. Apparent Motion : Pan & Tilt Camera Moves a Lot

6. Independent Object Motion Objects are the Focus Camera is more or less steady

7. Independent Object MotionwithCamera Pan Most common scenario for capturing performances

8. General Camera Motion Large changes in camera location & orientation

9. Visual Motion due to Environmental Effects Every pixel may have its own motion

10. The Works! General Camera & Object Motions

11. Why is Analysis and EstimationofVisual Motion Important?

12. Foreground Vs. Background Visual Motion Estimationas a means of extractingInformation Content in Dynamic Imagery...extract information behind pixel data...

13. Foreground Vs. Background Extended Scene Geometry Information Content in Dynamic Imagery...extract information behind pixel data...

14. Temporal Persistence Foreground Vs. Background Extended Scene Geometry Segment,Track,Fingerprint Moving Objects Layers with 2D/3D Scene Models Layers & Mosaics Direct method based Layered, Motion, Structure & Appearance Analysis provides Compact Representation for Manipulation & Recognition of Scene Content Information Content in Dynamic Imagery...extract information behind pixel data...

15. An ExampleA Panning Camera • Pin-hole camera model • Pure rotation of the camera • Multiple images related through a 2D projective transformation: also called a homography • In the special case for camera pan, with small frame-to-frame rotation, and small field of view, the frames are related through a pure image translation

16. Pin-hole Camera Model Y y Z f

17. Camera Rotation (Pan) Y’ y’ Z’ f

18. Camera Rotation (Pan) Y’’ f Z’’ y’’

19. Image Motion due to Rotationsdoes not depend on the depth / structure of the scene Verify the same for a 3D scene and 2D camera

20. Pin-hole Camera Model Y y Z f

21. Camera Translation (Ty) Y y X X X X Z f

22. Translational Displacement Image Motion due to Translation is a function of the depth of the scene

24. Sample Displacement Fields Render scenes with various motions and plot the displacement fields

25. Motion Field vs. Optical Flow Y wy Motion Field : 2D projections of 3D displacement vectors due to camera and/or object motion Ty wx X p’ P’ p Tx P wz Z Optical Flow : Image displacement field that measures the apparent motion of brightness patterns Tz

26. Motion Field vs. Optical Flow Lambertian ball rotating in 3D Motion Field ? Optical Flow ? Courtesy : Michael Black @ Brown.edu Image: http://www.evl.uic.edu/aej/488/

27. Motion Field vs. Optical Flow Stationary Lambertian ball with a moving point light source Motion Field ? Optical Flow ? Courtesy : Michael Black @ Brown.edu Image : http://www.evl.uic.edu/aej/488/

28. A Hierarchy of ModelsTaxonomy by Bergen,Anandan et al.’92 • Parametric motion models • 2D translation, affine, projective, 3D pose[Bergen, Anandan, et.al.’92] • Piecewise parametric motion models • 2D parametric motion/structure layers [Wang&Adelson’93, Ayer&Sawhney’95] • Quasi-parametric • 3D R, T & depth per pixel. [Hanna&Okumoto’91] • Plane+parallax [Kumar et.al.’94, Sawhney’94] • Piecewise quasi-parametric motion models • 2D parametric layers + parallax per layer [Baker et al.’98] • Non-parametric • Optic flow: 2D vector per pixel [Lucas&Kanade’81,Bergen,Anandan et.al.’92]

29. Sparse/Discrete Correspondences&Dense Motion Estimation

30. Discrete MethodsFeature Correlation&RANSAC

31. Visual Motion through Discrete Correspondences Images may be separated by time, space, sensor types In general, discrete correspondences are related through a transformation

32. Discrete MethodsFeature Correlation&RANSAC

33. Discrete Correspondences • Select corner-like points • Match patches using Normalized Correlation • Establish further matches using motion model

34. Direct Methods for Visual Motion EstimationEmploy Models of Motionand Estimate Visual MotionthroughImage Alignment

35. Characterizing Direct MethodsThe What • Visual interpretation/modeling involves spatio-temporal image representations directly • Not explicitly represented discrete features like corners, edges and lines etc. • Spatio-temporal images are represented as outputs of symmetric or oriented filters. • The output representations are typically dense, that is every pixel is explained, • Optical flow, depth maps. • Model parameters are also computed.

36. Direct Methods : The How Alignment of spatio-temporal images is a means of obtaining : Dense Representations, Parametric Models

37. Direct Method based Alignment

38. Images separated by time, space, sensor types Formulation of Direct Model-based Image Alignment[Bergen,Anandan et al.’92] Model image transformation as : Brightness Constancy

39. Reference Coordinate System Formulation of Direct Model-based Image Alignment Model image transformation as : Images separated by time, space, sensor types

40. Generalized pixel Displacement Formulation of Direct Model-based Image Alignment Model image transformation as : Images separated by time, space, sensor types Reference Coordinate System

41. Model Parameters Formulation of Direct Model-based Image Alignment Model image transformation as : Images separated by time, space, sensor types Reference Coordinate System Generalized pixel Displacement

42. Formulation of Direct Model-based Image Alignment Model image transformation as : Images separated by time, space, sensor types Reference Coordinate System Generalized pixel Displacement Model Parameters

43. Filtered Image Representations (to account for Illumination changes, Multi-modalities) Formulation of Direct Model-based Image Alignment Compute the unknown parameters and correspondences while aligning images using optimization : What all can be varied ?

44. Model Parameters Filtered Image Representations (to account for Illumination changes, Multi-modalities) Formulation of Direct Model-based Image Alignment Compute the unknown parameters and correspondences while aligning images using optimization : What all can be varied ?

45. Measuring mismatches (SSD, Correlations) Model Parameters Filtered Image Representations (to account for Illumination changes, Multi-modalities) Formulation of Direct Model-based Image Alignment Compute the unknown parameters and correspondences while aligning images using optimization : What all can be varied ?

46. Optimization Function Measuring mismatches (SSD, Correlations) Model Parameters Filtered Image Representations (to account for Illumination changes, Multi-modalities) Formulation of Direct Model-based Image Alignment Compute the unknown parameters and correspondences while aligning images using optimization : What all can be varied ?

47. Formulation of Direct Model-based Image Alignment Compute the unknown parameters and correspondences while aligning images using optimization : Optimization Function Measuring mismatches (SSD, Correlations) Model Parameters Filtered Image Representations (to account for Illumination changes, Multi-modalities) What all can be varied ?

48. A Hierarchy of ModelsTaxonomy by Bergen,Anandan et al.’92 • Parametric motion models • 2D translation, affine, projective, 3D pose[Bergen, Anandan, et.al.’92] • Piecewise parametric motion models • 2D parametric motion/structure layers [Wang&Adelson’93, Ayer&Sawhney’95] • Quasi-parametric • 3D R, T & depth per pixel. [Hanna&Okumoto’91] • Plane+parallax [Kumar et.al.’94, Sawhney’94] • Piecewise quasi-parametric motion models • 2D parametric layers + parallax per layer [Baker et al.’98] • Non-parametric • Optic flow: 2D vector per pixel [Lucas&Kanade’81,Bergen,Anandan et.al.’92]

49. Plan : This Part • First present the generic normal equations. • Then specialize these for a projective transformation. • Sidebar into backward image warping. • SSD and M-estimators.

50. An Iterative Solution of Model Parameters[Black&Anandan’94 Sawhney’95] • Given a solution at the mth iteration, find by solving : is a weight associated with each measurement.