Dense Motion Estimation

1. CS 636 Computer Vision Dense Motion Estimation Nathan Jacobs

2. overview • any issues or questions? • review from last time • dense motion estimation • relationship to feature-based • today: simple motion models • next time: more complex motion models

3. Motion and perceptual organization • Sometimes, motion is the only cue Lazebnik

4. Motion and perceptual organization • Even “impoverished” motion data can evoke a strong percept G. Johansson, “Visual Perception of Biological Motion and a Model For Its Analysis", Perception and Psychophysics 14, 201-211, 1973. Lazebnik

5. Motion and perceptual organization • Even “impoverished” motion data can evoke a strong percept G. Johansson, “Visual Perception of Biological Motion and a Model For Its Analysis", Perception and Psychophysics 14, 201-211, 1973. Lazebnik

6. Motion Estimation • Today’s Readings • Szeliski Chapters 7.1, 7.2, 7.4 • Human motion estimation • http://www.sandlotscience.com/Ambiguous/Barberpole_Illusion.htm • http://www.sandlotscience.com/Distortions/Breathing_Square.htm Most slides by Steve Seitz

7. Why estimate motion? • Lots of uses • Track object behavior • Correct for camera jitter (stabilization) • Align images (mosaics) • 3D shape reconstruction • Special effects 1998 Oscar for Visual Effects http://www.youtube.com/watch?v=au0ibcWUxPI

8. review from last time… • Discrete/Feature-based Motion Estimation • Least-squares • Robust Estimators • RANSAC, Hough Transform • Calibration

9. What is the difference? feature-based dense (simple translational motion)

10. feature-based vs. dense feature-based dense (simple translational motion) extract features extract features minimize residual minimize residual optimal motion estimate optimal motion estimate

11. Motion estimation • Input: sequence of images • Output: point correspondence • Feature correspondence: “Feature Tracking” • we’ve seen this already (e.g., SIFT) • can modify this to be more accurate/efficient if the images are in sequence (e.g., video) • Pixel (dense) correspondence: “Optical Flow” • today’s lecture

12. Motion field and parallax X(t+dt) V X(t) • X(t) is a moving 3D point • Velocity of scene point: V = dX/dt • x(t) = (x(t),y(t)) is the projection of X in the image • Apparent velocity v in the image: given by components vx = dx/dt and vy = dy/dt • These components are known as the motion field of the image v x(t+dt) x(t) Lazebnik

13. Motion field and parallax X(t+dt) To find image velocity v, differentiate x=(x,y) with respect to t (using quotient rule): V X(t) v x(t+dt) x(t) Image motion is a function of both the 3D motion (V) and thedepth of the 3D point (Z) Lazebnik

14. Motion field and parallax • Pure translation: V is constant everywhere Lazebnik

15. Motion field and parallax • Pure translation: V is constant everywhere • The length of the motion vectors is inversely proportional to the depth Z • if Vzis nonzero: • Every motion vector points toward (or away from) the vanishing point of the translation direction Lazebnik

16. Motion field and parallax • Pure translation: V is constant everywhere • The length of the motion vectors is inversely proportional to the depth Z • Vzis nonzero: • Every motion vector points toward (or away from) the vanishing point of the translation direction • Vz is zero: • Motion is parallel to the image plane, all the motion vectors are parallel Lazebnik

17. Optical flow • Definition: optical flow is the apparent motion of brightness patterns in the image • Ideally, optical flow would be the same as the motion field • Have to be careful: apparent motion can be caused by lighting changes without any actual motion • Think of a uniform rotating sphere under fixed lighting vs. a stationary sphere under moving illumination Lazebnik

18. Key assumptions • color constancy: a point in H looks the same in I • For grayscale images, this is brightness constancy • small motion: points do not move very far • This is called the optical flow problem Problem definition: optical flow • How to estimate pixel motion from image H to image I? • Solve pixel correspondence problem • given a pixel in H, look for nearby pixels of the same color in I

19. Optical flow constraints (grayscale images) • Let’s look at these constraints more closely • brightness constancy: Q: what’s the equation? • small motion: (u and v are less than 1 pixel) • suppose we take the Taylor series expansion of I:

20. Optical flow equation • Combining these two equations

21. Optical flow equation • Combining these two equations • In the limit as u and v go to zero, this becomes exact

22. The brightness constancy constraint • How many equations and unknowns per pixel? • One equation, two unknowns • Intuitively, what does this constraint mean? • The component of the flow perpendicular to the gradient (i.e., parallel to the edge) is unknown gradient (u,v) • If (u, v) satisfies the equation, so does (u+u’, v+v’) if (u’,v’) (u+u’,v+v’) edge

23. Aperture problem

24. Aperture problem

25. The barber pole illusion http://en.wikipedia.org/wiki/Barberpole_illusion

26. The barber pole illusion http://en.wikipedia.org/wiki/Barberpole_illusion

27. The barber pole illusion http://en.wikipedia.org/wiki/Barberpole_illusion

28. Solving the aperture problem • Basic idea: assume motion field is smooth • Horn & Schunk: add smoothness term • Lucas & Kanade: assume locally constant motion • pretend the pixel’s neighbors have the same (u,v) • Many other methods exist. Here’s an overview: • S. Baker, M. Black, J. P. Lewis, S. Roth, D. Scharstein, and R. Szeliski. A database and evaluation methodology for optical flow. In Proc. ICCV, 2007 • http://vision.middlebury.edu/flow/

29. Lucas-Kanade flow • How to get more equations for a pixel? • Basic idea: impose additional constraints • most common is to assume that the flow field is smooth locally • one method: pretend the pixel’s neighbors have the same (u,v) • If we use a 5x5 window, that gives us 25 equations per pixel!

30. RGB version • How to get more equations for a pixel? • Basic idea: impose additional constraints • most common is to assume that the flow field is smooth locally • one method: pretend the pixel’s neighbors have the same (u,v) • If we use a 5x5 window, that gives us 25*3 equations per pixel!

31. Solution: solve least squares problem • minimum least squares solution given by solution (in d) of: • The summations are over all pixels in the K x K window • This technique was first proposed by Lucas & Kanade (1981) Lucas-Kanade flow • Prob: we have more equations than unknowns

32. Conditions for solvability • Optimal (u, v) satisfies Lucas-Kanade equation • When is this solvable? • ATA should be invertible • ATA should not be too small due to noise • eigenvalues l1 and l2 of ATA should not be too small • ATA should be well-conditioned • l1/ l2 should not be too large (l1 = larger eigenvalue) • Does this look familiar? • ATA is the Harris matrix

33. Observation • This is a two image problem BUT • Can measure sensitivity by just looking at one of the images! • This tells us which pixels are easy to track, which are hard • very useful for feature tracking... Szeliski

34. Errors in Lucas-Kanade What are the potential causes of errors in this procedure? • Suppose ATA is easily invertible • Suppose there is not much noise in the image • When our assumptions are violated • Brightness constancy is not satisfied • The motion is not small • A point does not move like its neighbors • window size is too large • what is the ideal window size?

35. Improving accuracy • Recall our small motion assumption • This is not exact • To do better, we need to add higher order terms back in: This is a polynomial root finding problem • Can solve using Newton’s method • Lucas-Kanade method does one iteration of Newton’s method • Better results are obtained via more iterations

36. Iterative Refinement • Iterative Lucas-Kanade Algorithm • Estimate velocity at each pixel by solving Lucas-Kanade equations • Warp H towards I using the estimated flow field • - use image warping techniques • Repeat until convergence

37. Revisiting the small motion assumption • Is this motion small enough? • Probably not—it’s much larger than one pixel (2nd order terms dominate) • How might we solve this problem?

38. Optical flow result

39. Reduce the resolution!

40. u=1.25 pixels u=2.5 pixels u=5 pixels u=10 pixels image H image I image H image I Gaussian pyramid of image H Gaussian pyramid of image I Coarse-to-fine optical flow estimation

41. warp & upsample run iterative L-K . . . image J image I image H image I Gaussian pyramid of image H Gaussian pyramid of image I Coarse-to-fine optical flow estimation run iterative L-K

42. Error metrics quadratic truncated quadratic lorentzian Robust methods • L-K minimizes a sum-of-squares error metric • least squares techniques overly sensitive to outliers

43. first image quadratic flow lorentzian flow detected outliers Robust optical flow • Robust Horn & Schunk • Robust Lucas-Kanade • Reference • Black, M. J. and Anandan, P., A framework for the robust estimation of optical flow, Fourth International Conf. on Computer Vision (ICCV), 1993, pp. 231-236 http://www.cs.washington.edu/education/courses/576/03sp/readings/black93.pdf

44. Benchmarking optical flow algorithms • Middlebury flow page • http://vision.middlebury.edu/flow/

45. Discussion: features vs. flow? • Features are better for: • Flow is better for:

46. Advanced topics • Particles: combining features and flow • Peter Sand et al. • http://rvsn.csail.mit.edu/pv/ • State-of-the-art feature tracking/SLAM • Georg Klein et al. • http://www.robots.ox.ac.uk/~gk/

47. State-of-the-art Optical Flow in Matlab • Median filtering with spatially varying weights • Edge consistency • Robust error http://www.cs.brown.edu/~black/code.html http://people.csail.mit.edu/celiu/OpticalFlow/

48. for next time • parametric motion models • layered motion models