Optimisation in Image & Video Processing Closed Form Solutions

1. Optimisation in Image & Video ProcessingClosed Form Solutions Dr. David Corrigan

2. Segmentation • So far we have been looking at algorithms for solving segmentation sets whose labels form discrete sets • But what about problems involving labels with continuous labels? • Denoising • Deblurring • Alpha Matting • Motion Estimation

3. Closed Form Solutions • Many of these problems can be reduced to solving a massive set of equations. • In this handout we will look at • How these systems of equations are solved. • How closed form solutions to labelling problems arise in image processing. • Alpha Matting Algorithms including one based on a closed-form solution

4. Key References • Szeliski, Richard. Computer vision: algorithms and applications. Springer, 2010. (Sections 3.7 and 8.4 of book) • Horn, Berthold K., and Brian G. Schunck. "Determining optical flow." 1981 Technical Symposium East. International Society for Optics and Photonics, 1981. • Pérez, Patrick, Michel Gangnet, and Andrew Blake. "Poisson image editing."ACM Transactions on Graphics (TOG). Vol. 22. No. 3. ACM, 2003. • Levin, A., Lischinski, D., & Weiss, Y. (2008). A closed-form solution to natural image matting. Pattern Analysis and Machine Intelligence, IEEE Transactions on, 30(2), 228-242. • Chuang, Y. Y., Curless, B., Salesin, D. H., & Szeliski, R. (2001). A bayesian approach to digital matting. In Computer Vision and Pattern Recognition, 2001. CVPR 2001. Proceedings of the 2001 IEEE Computer Society Conference on(Vol. 2, pp. II-264). IEEE. • Wang, J., & Cohen, M. F. (2008). Image and video matting: a survey (Vol. 3, No. 2). Now Publishers Inc.

5. Solving Sparse Systems of Equations • Direct methods too intensive • Iterative approaches • Gauss-Siedel • Jacobi • Conjugate Gradient • Usually involving some form of multiresolution (ie. V-cycle Multuigrid, W-cycle Multigrid, Full Multigrid etc.) • http://www.math.ust.hk/~mawang/teaching/math532/mgtut.pdf • The backslash operator in Matlab

6. Inverse Filtering • We looked at how a convolution can be implemented as a matrix multiplication in 4c8 0

7. Variational Methods • Refers to a minimisation technique where energy functions are defined in continuous rather than discrete space. • Energy functions are of the form • If is a convex function of then it minimum can be found by solving a PDE. • Discretisation of the PDE results in a large system of equations.

8. Optical Flow (Motion Estimation) • Popular approach in Computer Vision for Motion Estimation – Horn and Schunck 1981 • Variational framework that yields a full resolution motion field . • Remember for motion estimation we had • In the continuous domain that constraint can be expressed at time as

9. Optical Flow – The data term Expanding the right hand side using a Taylor series we get Subtracting from both sides and dividing by as

10. Optical Flow – The data term • In the optical flow framework So Hence, we can define the data term as However for any point we have 2 unkowns and We need an extra constraint!!

11. The Smoothness Term • Need to introduce a regularisation term that enforces spatial smoothness of the motion field • Equivalent statement in the continuous domain is that the derivative of the motion field have low absolute values

12. The Total Energy and

13. Euler-Legrange Equations • If is convex and differentiable the function is minimised wrt to and when the following equations hold • The EL equations vary depending on the number of dependent variables (x and y), functions (u and v) and order of derivatives (1st order) – See Wikipedia

14. Optical Flow • For , we get • These are 2 elliptical Partial Differential Equations Convert into discrete form and yielding a system of equations

15. Discretisation • Use Numerical approximations for derivatives (many ways of doing this) • Derivatives of can be calculated in the same manner. • The Horn & Schunck paper used different operators to calculate the derivative terms

16. A System of equations • Subbing for the derivatives we get • If there are pixels we will have equations in unknowns. • The solution results in the optical flow field for the current frame.

17. Limitations of the Variational Framework • The EL formula finds the turning points of the . • must be convex infor the estimated turning point to be guaranteed to be the global minimum of . ( has to be >0 for this reason) • A sum of convex functions is also convex. True for integration too. • This means both the data and smoothness terms must be convex (in Graph Cuts the only constraints were on the smoothness term) • A technique called Graduated Non-Convexity can be used when the cost function is not convex (but has only one turning point which is a minimum).

18. Poisson Image Editing (SIGGRAPH ‘03) All images shown here are taken from the original paper

19. Poisson Image Editing • To fill the hole we want to make the of the image gradient of the edited image () to be as close as possible to the some vector field () such that at the hole boundary where is the original destination image Typically is the gradient of the source image ie.

20. Poisson Image Editing • To fill the hole we want to make the of the image gradient of the image () to be as close as possible to the some vector field () such that at the hole boundary where is the image to be edited Applying the EL equations yields

21. Poisson Image Editing • After discretisation this yields (for pixels not on the first and last rows of the image) Notes • The neighbourhood is the standard 4 connected neighbourhood. • For points in no neighbours on the boundary there will be no on the right hand side. • relates to the values of and and is dependent on the particular application envisaged. • If , we will have equations in unknowns.

22. Seamless Cloning (ie. and implements )

23. Mixed Seamless Cloning

24. Mixed Seamless Cloning

25. Texture Flattening

26. Recolouring (turning background to greyscale) • Let be the greyscale version of the source and let be the source in colour.

27. Alpha Matting

28. Matting • The problem with binary segmentation

29. The Matting Equation • Matting process is governed by the following equation • This system of equations is undetermined • Unlike the binary segmentation estimation of and is not trivial. • For greyscale matting 1equation and 3 unknowns per pixel • For colour 3 equations and 7 unknowns per pixel • This is solved by assuming and are locally smooth • Local Colour Models Capture Smoothness.

30. Potential Simplifications • Chroma-Keying

31. Potential Simplifications • Chroma-Keying

32. Potential Simplifications • Tri-Maps

33. Bayesian Matting (CVPR 2001) Notes: • The values of , and are assumed to be independent. Hence • No distribution for is specified. • The distributions for are chosen to be spatial smoothness priors. For example, • Optimisation is iterative. Solve for and then

34. Likelihood Model If and are known then the likelihood is quadratic in . Differentiate wrt. and set to 0.

35. Foreground and Background Priors • Both priors based on a Gaussian Distribution of the local colour distribution • Means and covariances given by where

36. Weights for Foreground and Background Priors • Foreground weight • Pixels closer to should have a higher weight • Pixels with higher should have a higher weight where represents a circular Gaussian roll off function • Background weight • Note if we are using this technique for keying or difference matting we know the value of so there is no need for

37. Solving for and • If is known then is quadratic in and . • Differentiating wrt and and setting to 0 yields Where the identity matrix.

38. Examples (http://grail.cs.washington.edu/projects/digital-matting/image-matting/) Note: Foreground Image for composite taken from CFM Result ()

39. Comparison Against Binary Segmentation Bayesian Matting GrabCut Binary Segmentation

40. Different Backgrounds

41. Closed Form Matting (PAMI 2008) • Powerful method for alpha matting • Can get away with a much sparser tri-map • Proposes a model that allows the global optimum alpha matte for the model to be estimated. • Joint optimisation over all pixels rather than individually as with Bayesian Matting • Avoids need for choosing an order for updating pixel sites. • Decouples estimate of foreground and background from the estimation of the alpha matte • Separate Variational Style Framework for estimating foreground and background colours after the alpha matte is estimated • We will look briefly at how the solution can be derived for greyscale images.

42. Closed Form Matting (PAMI 2008) Consider the grayscale version of the matting equstion Assume that in a small window (typically ) centred on that F and B are approximately constant We can rearrange the matting equation to yield where

43. Energy Function • This Equation forms the energy model for the overall problem • Consider all possible windows in the image, then… Observations: • The is a regularisation term that prevents numerical instability (ie. ). is a positive valued tuning parameter. • is a quadratic function in the unknowns unlike the probabilistic model used in Bayesian Matting. • However we still have unknowns for an image of pixels

44. Solving for alpha It is possible to define the energy function where is of the form • is quadratic in and can be minimised easily • is a sparse matrix and is referred to as the matting Laplacian by the authors.

45. Adding Hard Constraints • is minimised when . • This will have multiple solutions and is always a solution where is a column vector where every value is a constant • Need some hard constraints to get a useful result • Define and • is an diagonal matrix whose diagonal elements are 1 if a hard constraint is provided and 0 otherwise • is a vector that represents the values of alpha for the hard constraints and is 0 otherwise • Minimise • Solve:

46. Colour Version of Model • The equivalent linear model becomes where and are constant over the window Note: • and are not necessarily constant over the window. • See paper for definitions of and . Applying a similar derivation to the greyscale case yields

47. Solving for and • Although solutions for each and are easily obtained once is known, solving for and is trickier • Authors propose a variational framework

48. Examples Matlab Code Available: http://people.csail.mit.edu/alevin/matting.tar.gz

49. Examples

50. Importance of Using the Correct when compositing Using the source image as when compositing Using the estimated as when compositing