Créer une présentation
Télécharger la présentation

Télécharger la présentation
## Recent progress in optical flow

- - - - - - - - - - - - - - - - - - - - - - - - - - - E N D - - - - - - - - - - - - - - - - - - - - - - - - - - -

**Recent progress in optical flow**progress Presented by: Darya Frolova and Denis Simakov**Optical Flow is not in favor**Very popular slide: Often not using Optical Flow is stated as one of the main advantages of a method Optical Flow methods have a reputation of either unreliable or slow Recent works claim: Optical Flow can be computed fastand accurately**more methods**many methods Optical Flow Research: Timeline Horn&Schunck Lucas&Kanade 1981 1992 1998 now Benchmark:Galvin et.al. Benchmark:Barron et.al. Seminal papers no significant improvement, but a lot of useful ingredients were developed**In This Lecture**We will describe : • Ingredients for an accurate and robust optical flow • How people combine these ingredients • Fast algorithms Papers: • Combining the advantages of local and global optic flow methods (“Lucas/Kanade meets Horn/Schunck”) A. Bruhn, J. Weickert, C. Schnörr, 2002 - 2005 • High accuracy optical flow estimation based on a theory for warping T. Brox, A. Bruhn, N. Papenberg, J. Weickert, 2004 - 2005 • Real-Time Optic Flow Computation with Variational Methods A. Bruhn, J. Weickert, C. Feddern, T. Kohlberger, C. Schnörr, 2003 - 2005 • Towards ultimate motion estimation: Combining highest accuracy with real-time performanceA. Bruhn, J. Weickert, 2005 • Bilateral filtering-based optical flow estimation with occlusion detection J.Xiao, H.Cheng, H.Sawhney, C.Rao, M.Isnardi, 2006**Definitions**The optical flow is a velocity field in the image which transforms one image into the next image in a sequence [ Horn&Schunck ] + = frame #2 frame #1 flow field The motion field … is the projection into the image of three-dimensional motion vectors [ Horn&Schunck]**flow (2)**flow (1): true motion Ambiguity of optical flow Frame 1**Applications**optical flow • video compression • 3D reconstruction • segmentation • object detection • activity detection • key frame extraction • interpolation in time motion field We are usually interested in actual motion**Outline**• Ingredients for an accurate and robust optical flow • Local image constraints on motion • Robust statistics • Spatial coherence • How people combine these ingredients • Fast algorithms**Brightness Constancy**u frame t+1 v frame t**Complex dependence on**Linearized brightness constancy Deviation from brightness constancy (we want to minimize it) Linearize:**Linearized brightness constancy**Let us square the difference: J – “motion tensor”, or “structure tensor”**Averaged linearized constraint**J is a function of x, y (a matrix for every point) Combine over small neighborhoods (more robust to noise): = J ***Method of Lucas&Kanade**• Solve independently for each point [ Lucas&Kanade 1981] linear system Can be solved for every point where matrix is not degenerate**Lukas&Kanade - Results**Rubik cube Hamburg taxi flow field flow field**Brightness is not always constant**Rotating cylinder Brightness constancy does not always hold Gradient constancy holds intensity intensity derivative position position**Local constraints - Summary**We have seen linearized • brightness constancy averaged linearized averaged linearized • gradient constancy**Local constraints work poorly**Optical flow direction using only local constraints input video color encodes direction as marked on the boundary**Where local constraints fail**Uniform regions Motion is not observable in the image (locally)**Where local constraints fail**“Aperture problem” We can estimate only one flow component (normal)**Where local constraints fail**Occlusions We have not seen where some points moved Occluded regions are marked in red**Obtaining support from neighbors**• Two main problems with local constraints: • information about motion is missing in some points => need spatial coherency • constraints do not hold everywhere => need methods to combine them robustly good missing wrong**Robust combination of partially reliable data**or: How to hold elections**L2:**L1: xi → xi + ∆ Influence of xi on E: equal for all xi proportional to Outliers influence the most Majority decides Toy example Find “best” representative for the set of numbers xi**Oligarchy**Democracy Elections and robust statistics many ordinary people a very rich man wealth Votes proportional to the wealth One vote per person like in L1 norm minimization like in L2 norm minimization**usual: L2**robust regularized robust: L1 ε • easy to analyze and minimize • sensitive to outliers • robust in presence of outliers • non-smooth: hard to analyze • smooth: easy to analyze • robust in presence of outliers Combination of two flow constraints [A. Bruhn, J. Weickert, 2005] Towards ultimate motion estimation: Combining highest accuracy with real-time performance**Obtaining support from neighbors**• Two main problems with local constraints: • information about motion is missing in some points => need spatial coherency • constraints do not hold everywhere => need methods to combine them robustly good missing wrong**- flow in the x direction**• flow in the y direction • gradient Homogeneous propagation This constraint is not correct on motion boundaries => over-smoothing of the resulting flow [Horn&Schunck 1981]**Robustness to flow discontinuities**ε (also known as isotropic flow-driven regularization) [T. Brox, A. Bruhn, N. Papenberg, J. Weickert, 2004] High accuracy optical flow estimation based on a theory for warping**Selective flow filtering**• We want to propagate information • without crossing image and flow discontinuities • from “good” points only (not occluded) Solution: use “bilateral” filter in space, intensity, flow; taking into account occlusions [J.Xiao, H.Cheng, H.Sawhney, C.Rao, M.Isnardi, 2006] Bilateral filtering-based optical flow estimation with occlusion detection**I**I I I x x x Bilateral filter Unilateral (usual) Bilateral x Preserves discontinuities! [C. Tomasi, R. Manduchi, 1998] Bilateral filtering for gray and color images.**occluded regions**Using of bilateral filter - Example cyan rectangle moves to the right and occludes background region marked by red [J.Xiao, H.Cheng, H.Sawhney, C.Rao, M.Isnardi, 2006] Bilateral filtering-based optical flow estimation with occlusion detection**Learning of spatial coherence**• Come to the next lecture …**Spatial coherence: Summary**Homogeneous propagation - oversmoothing Robust statistics with homogeneous propagation - preserves flow discontinuities Bilateral filtering - combines information from regions with similar flow and similar intensities Handles occlusions**Two more useful ingredients**in brief – one slide each**2D vs. 3D**Several frames allow more accurate optical flow estimation 2 frames: Several frames:**MultiscaleOptical Flow**Linearization: valid only for small flow pyramid for frame 1 pyramid for frame 2 frame 1warped ? + upsample + (other names: “warping”, “coarse-to-fine”, “multiresolution”)**Methods**How to make tasty soup with these ingredients: several recipes**Outline**• Ingredients for an accurate and robust optical flow • How people combine these ingredients • Lukas & Kanade meet Horn & Schunck • The more ingredients – the better • Bilateral filtering and occlusions • Fast algorithms**Combining ingredients**• Spatial coherency • Homogeneous • Flow-driven • Bilateral filtering + occlusions • Local constraints • Brightness constancy • Image gradient constancy Energy = ∫ϕ(Data) + ∫ϕ(“Smoothness”) Combined using robust statistics Computed coarse-to-fine Use several frames**Combining Local and Global**Remember: Lucas&Kanade Horn&Schunk Basic “Combining local and global” [A. Bruhn, J. Weickert, C. Schnörr, 2002]**Sensitivity to noise – quantitative results**frame t+1 Error measure: angle between true and computed flow in (x,y,t) space frame t ground truth flow**The more ingredients - the better**brightness constancy spatial coherence gradient constancy [Bruhn, Weickert, 2005]Towards ultimate motion estimation: Combining highest accuracy with real-time performance**Quantitative results**Angular error Method Yosemite sequence with clouds Average error decreases, but standard deviation is still high….**Influence of each ingredient**For Yosemite sequence with clouds**Handling occlusions**bilateral filtering of flow:preserve intensity and flow discontinuities; model occlusions [J.Xiao, H.Cheng, H.Sawhney, C.Rao, M.Isnardi, ECCV 2006] Bilateral filtering-based optical flow estimation with occlusion detection