optical flow n.
Skip this Video
Loading SlideShow in 5 Seconds..
Optical flow PowerPoint Presentation
Download Presentation
Optical flow

Optical flow

195 Views Download Presentation
Download Presentation

Optical flow

- - - - - - - - - - - - - - - - - - - - - - - - - - - E N D - - - - - - - - - - - - - - - - - - - - - - - - - - -
Presentation Transcript

  1. Optical flow Or where do the pixels move? Alon Gat

  2. Problem Definition Given: two or more frames of an image sequence Wanted: Displacement field between two consecutive frames optical flow

  3. Visualize Optical Flow Vector Plot: Subsample vector field and use arrows for visualization Color Plot: Visualize direction as color and magnitude as brightness

  4. What is optical flow good for? Extraction of Motion Information • • robot navigation/driver assistance • • surveillance/tracking • • action recognition Processing of Image Sequences • • video compression • • ego motion compensation Related Correspondence Problems • • stereo reconstruction • • structure-from-motion • • medical image registration

  5. How to estimate pixel motion from two images? • Find pixel correspondences • Given a pixel in img1, look for nearby pixels of the same color in img2 • Key assumptions • color constancy: a point in img1 looks “the same” in img2 • For grayscale images, this is brightness constancy • small motion: points do not move very far

  6. Brightness Constancy Assumption Optical Flow: the vector field Displacement: • Assume brightness of patch remains same in both images:

  7. Brightness Constancy Assumption • The LinearizedBrightness Constancy Assumption Idea: If u and v are small and I is sufficiently smooth, one may linearize this constancy assumption via a first-order Taylor expansion around the point Known Unknown

  8. Image derivative

  9. Horn & Schunck Algorithm (MIT 1981) Smoothness Constraint :meaning neighbor pixels in the picture has similar velocities. In other words, nearby pixels moves together

  10. Horn & Schunck Algorithm We seek the set that minimize: Smoothness term Data term brightness constancy • data term - penalizes deviations from constancy assumptions • smoothness term - penalizes dev. from smoothness of the solution • regularization parameter α - determines the degree of smoothness Output – the optical flow!

  11. Smoothing Idea: In order to reduce the influence of noise and outliers, we convolve I0 with a Gaussian of mean μ = 0 and standard deviation Gaussian

  12. Horn & Schunck Algorithm Euler-Lagrange equations According to the calculus of variations, a minimizer of E must fulfill the Euler-Lagrange equations Which are highly non linear system of equations… 

  13. Calculus of Variation So we linearize again! Euler-Lagrange equations Or

  14. Horn & Schunck Algorithm flow derivatives here discredited via linear system of equations

  15. Horn & Schunck Algorithm

  16. Horn & Schunck Algorithm Update Rule:

  17. Examples

  18. Output

  19. Problems with the method. • Hard to find boundaries. • Two approximations. Less accurate.

  20. So what are we trying to do? • Instead of approximating the brightness constancy to the 1stTylor expansion, we’ve add one order. • Second and more important, instead of solving E-L equations (which needed to be linearized) we wrote the Functional as n*m equations and minimized it with regular minimization methods (Gradient decent, Quasi Newton, and others)

  21. 1. Mathemactica for symbolic calculations. Where P are the Image derivatives, and u, v is the optical flow Mathematica Symbolic Toolbox for MATLAB--Version 2.0 (

  22. 2. Matlab for numeric calculations • Quasi Newton IterationThe problem was that calculation time of inverse of non sparse matrix was long.And then multiplying two non sparse matrix… • Gradient Decent.Non of the above problems but linear convergence rate.And convergence to local min.

  23. 3. Leaving Mathematica • In order to get things going faster (moving loooooooong string from matlab to mathematica takes awhile), we found the functional matrix’s constancy and calculate it in matlab. for i=2:imageSizeN-1 for j=2:imageSizeM-1 gradC(i,j)=gradC(i,j)+alpha*(2*(-c(i-1,j)+c(i,j))+2*(-c(i,j-1)+c(i,j))-2*alpha*(-c(i,j)+c(i,j+1)) - 2*alpha*(-c(i,j)+c(i+1,j))+4*exp((-1+c(i,j)^2+s(i,j)^2)^2)*c(i,j)*(-1+c(i,j)^2+s(i,j)^2) +2*(Ix(i,j)*m(i,j)+Ixz(i,j)*m(i,j)+2*Ixx(i,j)*c(i,j)*m(i,j)^2+Ixy(i,j)*m(i,j)^2*s(i,j))*(Iz(i,j)+Izz(i,j)+Ix(i,j)*c(i,j)*m(i,j)+Ixz(i,j)*c(i,j)*m(i,j)+Ixx(i,j)*c(i,j)^2*m(i,j)^2+Iy(i,j)*m(i,j)*s(i,j) +Iyz(i,j)*m(i,j)*s(i,j)+Ixy(i,j)*c(i,j)*m(i,j)^2*s(i,j)+Iyy(i,j)*m(i,j)^2*s(i,j)^2)); gradS(i,j)=gradS(i,j)+4*exp((-1+c(i,j)^2+s(i,j)^2)^2)*s(i,j)*(-1+c(i,j)^2+s(i,j)^2)+2*(Iy(i,j)*m(i,j)+Iyz(i,j)*m(i,j)+Ixy(i,j)*c(i,j)*m(i,j)^2+2*Iyy(i,j)*m(i,j)^2*s(i,j))*(Iz(i,j)+Izz(i,j) +Ix(i,j)*c(i,j)*m(i,j)+Ixz(i,j)*c(i,j)*m(i,j)+Ixx(i,j)*c(i,j)^2*m(i,j)^2+Iy(i,j)*m(i,j)*s(i,j)+Iyz(i,j)*m(i,j)*s(i,j)+Ixy(i,j)*c(i,j)*m(i,j)^2*s(i,j)+Iyy(i,j)*m(i,j)^2*s(i,j)^2) +alpha*(2*(-s(i-1,j)+s(i,j))+2*(-s(i,j-1)+s(i,j))-2*alpha*(-s(i,j)+s(i,j+1))-2*alpha*(-s(i,j)+s(i+1,j))); gradM(i,j)=gradM(i,j)+beta*(2*(-m(i-1,j)+m(i,j))+2*(-m(i,j-1)+m(i,j)))-2*beta*(-m(i,j)+m(i,j+1))-2*beta*(-m(i,j)+m(i+1,j))+2*(Ix(i,j)*c(i,j)+Ixz(i,j)*c(i,j) +2*(Ixx(i,j)*c(i,j)^2*m(i,j)+Iy(i,j)*s(i,j)+Iyz(i,j)*s(i,j)+2*Ixy(i,j)*c(i,j)*m(i,j)*s(i,j)+2*Iyy(i,j)*m(i,j)*s(i,j)^2)*(Iz(i,j)+Izz(i,j)+Ix(i,j)*c(i,j)*m(i,j)+Ixz(i,j)*c(i,j)*m(i,j) +Ixx(i,j)*c(i,j)^2*m(i,j)^2+Iy(i,j)*m(i,j)*s(i,j)+Iyz(i,j)*m(i,j)*s(i,j)+Ixy(i,j)*c(i,j)*m(i,j)^2*s(i,j)+Iyy(i,j)*m(i,j)^2*s(i,j)^2)); end end end

  24. Gradient Decent

  25. So what is there to optimize? • All methods need couple of unknown parameters which need to be selected by an educated guess. (Condor to the rescue) • All the image derivatives and gradients are calculated in a linear manner.

  26. First results Takes about 45min(highend algorithms take around 20sec…)

  27. Questions? The Ground Truth.