Motion Detail Preserving Optical Flow Estimation

1. Motion Detail Preserving Optical Flow Estimation Tzu ming Su Advisor：S.J.Wang L. Xu, J. Jia, and Y. Matsushita. Motion detail preserving optical flow estimation. In CVPR, 2010.

2. Outline • Previous Work • Optical flow • Conventional optical flow estimation

3. CCD 3D motion vector 2D optical flow vector Motion Field • Definition：anideal representation of 3D motion as it is projected onto a camera image.

4. Motion field • Applications： • Video enhancement：stabilization, denoising, super resolution • 3D reconstruction：structure from motion (SFM) • Video segmentation • Tracking/recognition • Advanced video editing (label propagation)

5. Motion field estimation • Optical flow Recover image motion at each pixel from spatio-temporal image brightness variations • Feature-tracking Extract visual features (corners, textured areas) and “track” them over multiple frames

6. Optical flow • Definition：the apparent motion of brightness patterns in the images • Map flow vector to color • Magnitude: saturation • Orientation: hue

7. Optical flow • Key assumptions • Brightness constancy • Small motion • Spatial coherence Remark：Brightness constancy is often violated • Use gradient constancy for addition, both of them are called data constraint

8. Ix v It Brightness consistency • 1-D case

9. Brightness consistency • 1-D case I ( x , t ) v ? p

10. Temporal derivative Spatial derivative Brightness consistency • 1-D case v I ( x , t ) p

11. Brightness consistency • 1-D case • 2-D case One equation, two velocity (u,v) unknowns… v u

12. Aperture Problem

13. Aperture Problem

14. Time t Time t+dt Aperture Problem • We know the movement parallel to the direction of gradient , but not the movement orthogonal to the gradient • We need additional constraints ?

15. Conventional estimation • Use data consistency & additional constraint to estimate optical flow • Horn-Schunck • Minimize energy function with smoothness term • Lucas-Kanade • Minimize least square error function with local region coherence

16. Horn-Schunck estimation • Imposing spatial smoothness to the flow field • Adjacent pixels should move together as much as possible • Horn & Schunckequation

17. Horn-Schunck estimation • Use 2D Euler Lagrange • Can be iteratively solved

18. Coarse to fine estimation • Optical flow is assumed to be small motions , but in fact most motions are not • Solved by coarse to fine resolution

19. run iteratively . . . image It-1 image I Coarse to fine estimation run iteratively

20. Outline • Previous Work • Contributions • Extended Flow Initialization • Selective data term • Efficient optimization solver • Experimental result • Conclusion

21. Outline • Previous Work • Contributions • Extended Flow Initialization • Selective data term • Efficient optimization solver • Experimental result • Conclusion

22. Muti-scale problem • Conventional coarse to fine estimation can’t deal with large displacement. • With different motion scales between foreground & background , even small motions can be miss detected.

23. Muti-scaleproblem

24. Ground truth Ground truth Estimate Estimate Estimate … Ground truth

25. Muti-scale problem • Large discrepancy between initial values and optimal motion vectors • Solution： Improve flow initialization to reduce the reliance on the initializationfromcoarser levels

26. Selection Fusion Sparse feature matching Dense nearest-neighbor patch matching

27. Extended Flow Initialization • Sparse feature matching for each level

28. Extended Flow Initialization • Identify missing motion vectors

29. Extended Flow Initialization • Identify missing motion vectors

30. Extended Flow Initialization …

31. Extended Flow Initialization Fuse …

32. Selection Fusion Sparse feature matching Dense nearest-neighbor patch matching

33. Outline • Previous Work • Contributions • Extended Flow Initialization • Selective data term • Efficient optimization solver • Experimental result • Conclusion

34. constraints • Brightness consistency • Gradient consistency • Average

35. constraints • Pixels moving out of shadow • Color constancy is violated : ground truth motion of p1 • Gradient constancy holds • Average:

36. constraints • Pixels undergoing rotational motion • Color constancy holds : ground truth motion of p2 • Gradient constancy is violated • Average:

37. Selective data term • Selectively combine the constraints where

38. Selective data term selective

39. Outline • Previous Work • Contributions • Extended Flow Initialization • Selective data term • Efficient optimization solver • Experimental result • Conclusion

40. Discrete-optimization • Minimizing energy including discrete α & continuous u： • Try to separate α & u • For α • Probability of a particular state of MRF system

41. Discrete-optimization • Partition function • Sum over all possible values of α

42. Discrete-optimization • Optimal condition (Euler-Lagrange equations) • It decomposes to • Minimization • Update α • Compute flow field

43. continuous-optimization • Energy function • Variable splitting

44. continuous-optimization • Fix u , estimate w,p • Fix w,p , estimate u • The Euler-Lagrange equation Is linear.

45. Outline • Previous Work • Contributions • Extended Flow Initialization • Selective data term • Efficient optimization solver • Experimental result • Conclusion

46. selective data term Difference Selective Averaging

47. Experimental results

48. Results from Different Steps Coarse-to-fine Extended coarse-to-fine

50. Large Displacement Overlaid Input