Image Matting with the Matting Laplacian

1. Image Matting with the Matting Laplacian Chen-Yu Tseng 曾禎宇 Advisor: Sheng-Jyh Wang

2. Image Matting with the Matting Laplacian • Matting Laplacian • A. Levin, D. Lischinski, Y. Weiss. A Closed Form Solution to Natural Image Matting. IEEE T. PAMI, vol. 30, no. 2, pp. 228-242, Feb. 2008. • Spectral Matting • A. Levin, A. Rav-Acha, D. Lischinski. Spectral Matting. IEEE T. PAMI, vol. 30, no. 10, pp. 1699-1712, Oct. 2008. • Matting for Multiple Image Layers • D. Singaraju, R. Vidal. Estimation of Alpha Mattes for Multiple Image Layers. IEEE T. PAMI, vol. 33, no. 7, pp. 1295-1309, July 2011. Center for Imaging Science, Department of Biomedical Engineering, The Johns Hopkins University

3. Image Matting • Extracting a foreground object from an image along with an opacity estimate for each pixel covered by the object Spectral Matting Result Input Image Conventional Segmentation Result

4. Image Compositing Equation = x α1 L1 Input Image x + α2 L2 x + α3 L3 Alpha Mattes Image Layers

5. Methodology • Supervised Matting • Unsupervised Matting • Spectral Matting Input Image Trimap (user’s constraint) Alpha Matte Input Image Matting Components

6. x x = + Local Models for Alpha Mattes , , and are unknown  ill-posed problem Assume a and b are constant in a small window

7. Color Line Assumption Omer and M. Werman. Color Lines: Image Specific Color Representation. CVPR, 2004. Input Color Distributions

8. Local Models for Alpha Mattes for Multiple Layers Local Models Two color lines A color point and a color point Two color points and a single color line Four color points B G R

9. Local Models Two color lines A color plane and a color point Two color points and a single color line Four color points Color point Unknown color point Color plane

10. Local Models Two color lines A color plane and a color point Two color points and a single color line Four color points = Color point =++ Color plane

12. Local Models for Alpha Mattes for Multiple Layers Local Models Two color lines A color point and a color point Two color points and a single color line Four color points B G R

13. The Matting Laplacian

14. Overview of Spectral Matting Input Data Matting Laplacian Construction Input Image Local Adjacency Spectral Graph Analysis Data Component Generation Laplacian Matrix Output Components Components

15. Spectral Clustering Scatter plot of a 2D data set K-means Clustering Spectral Clustering U. von Luxburg. A tutorial on spectral clustering. Technical report, Max Planck Institute for Biological Cybernetics, Germany, 2006.

16. Graph Construction Similarity Graph Vertex Set Weighted Adjacency Matrix Similarity Graph • Similarity Graph • ε-neighborhood Graph • k-nearest neighbor Graphs • Fully connected graph Connected Groups

17. Graph Laplacian L: Laplacian matrix W: adjacency matrix For every vector D: degree matrix

18. L: Laplacian matrix W: adjacency matrix Example 2 1 3 4 Cost Function 5 Similarity Graph Good Assignment Poor Assignment * 2 1 2 1 3 3 4 4 5 5

19. LaplacianEigenvectors s.t.=1 : Eigenvector : Eigenvalue L is symmetric and positive semi-definite. The smallest eigenvalue of L is 0, the corresponding eigenvector is the constant one vector 1. L has n non-negative, real-valued eigenvalues 0= λ1 ≦ λ 2 ≦ . . . ≦ λ n. Input Image Smallest eigenvectors

20. K-means Projection into eigs space From Eigenvectors to Matting Components Smallest eigenvectors

21. Overview of Spectral Matting Input Data Graph Construction Input Image Local Adjacency Spectral Graph Analysis Data Component Generation Laplacian Matrix Output Components Components

22. Matting Laplacian x + = 1-α B x α F

23. Matting Laplacian Color Distribution

24. Matting Laplacian Matting affinity function Typical affinity function

25. Brief Summary K-means Clustering & Linear Transformation Input Image Laplacian Matrix Smallest Eigenvectors Matting Components

26. Supervised Matting Cost function with user-specified constraint: Foreground Background Unknown Input Trimap

27. Supervised Matting

28. Estimation Alpha Matte for Two Layers

29. Estimation Alpha Matte for Multi-Layers Karusch-Kuhn-Tucker (KKT) condition

30. Construction Assumption The vector of 1s lies in the null space of L, the solution automatically satisfies the constraint

31. Constrained Alpha Matte Estimation Image matting for n≥2 image layers with positivity + summation constraints

32. Karusch-Kuhn-Tucker (KKT) conditions For 0 < < 1 (i,i)=0 and(i,i)=0 Conventional Approaches Directly Clipping Equivalent to Introducing Lagrange Multipliers Refinement is neglected in conventional approaches

33. Experiments

35. Algorithm 1. (b) Algorithm 2. (c) Spectral Matting. (d) SM-enhance.

36. Algorithm 1. (b) Algorithm 2. (c) Spectral Matting. (d) SM-enhance.

38. Summary • Image Matting with the Matting Laplacian • Construction of the Matting Laplacian • Image Compositing Model • Local-Color Affine Model • Supervised Closed-form Matting • Two-layer • Multiple-layer • Spectral Matting • Extended Applications

39. Depth Estimation Compositing Image Likelihood Prior Input Image Estimated Depth MAP Prior Refined Result Confidence Map

40. Input Image Transmission Prior Output Image Refined Transmission

41. Graph Laplacian and Non-linear Filters Global Optima Local Optima