Spectral Matting A. Levin D. Lischinski and Y. Weiss. A Closed Form Solution to Natural Image Matting. IEEE Conf. on Computer Vision and Pattern Recognition (CVPR), June 2006, New York A. Levin, A. Rav-Acha, D. Lischinski. Spectral Matting. Best paper award runner up. IEEE Conf. on Computer Vision and Pattern Recognition (CVPR), Minneapolis, June 2007 A. Levin1,2, A. Rav-Acha1, D. Lischinski1. Spectral Matting. IEEE Trans. Pattern Analysis and Machine Intelligence, Oct 2008. 1School of CS&Eng The Hebrew University 2CSAIL MIT
Hard segmentation and matting compositing Hard segmentation Source image Matte compositing
Unsupervised Previous approaches to segmentation and matting Input Hard output Matte output Spectral segmentation: Shi and Malik 97 Yu and Shi 03 Weiss 99 Ng et al 01 Zelnik and Perona 05 Tolliver and Miller 06
Supervised Unsupervised Previous approaches to segmentation and matting Input Hard output Matte output July and Boykov01 Rother et al 04 Li et al 04
Supervised Unsupervised Previous approaches to segmentation and matting Input Hard output Matte output ? Trimap interface: Bayesian Matting (Chuang et al 01) Poisson Matting (Sun et al 04) Random Walk (Grady et al 05) Scribbles interface: Wang&Cohen 05 Levin et al 06 Easy matting (Guan et al 06)
User guided interface Matting result Scribbles Trimap
x x = + Generalized compositing equation 2 layers compositing
x x = + K layers compositing x x = + x x + + Matting components Generalized compositing equation 2 layers compositing
x x = + x x + + Generalized compositing equation K layers compositing Matting components: “Sparse” layers- 0/1 for most image pixels
Unsupervised matting Input Automatically computedmatting components
Building foreground object by simple components addition + + =
Spectral segmentation Spectral segmentation: Analyzing smallest eigenvectors of a graph Laplacian L E.g.: Shi and Malik 97 Yu and Shi 03 Weiss 99 Ng et al 01 Maila and shi 01 Zelnik and Perona 05 Tolliver and Miller 06
x x = + Problem Formulation Assume a and b are constant in a small window
Derivation of the cost function
Derivation
The matting Laplacian • semidefinite sparse matrix • local function of the image:
The matting affinity
The matting affinity Input Color Distribution
Matting and spectral segmentation Typical affinity function Matting affinity function
Eigenvectors of input image Input Smallest eigenvectors
Null Spectral segmentation Fully separated classes: class indicator vectors belong to Laplacian nullspace General case: class indicators approximated as linear combinations of smallest eigenvectors Binary indicating vectors Laplacian matrix
Zero eigenvectors Binary indicating vectors Laplacian matrix Smallest eigenvectors Linear transformation Spectral segmentation Fully separated classes: class indicator vectors belong to Laplacian nullspace General case: class indicators approximated as linear combinations of smallest eigenvectors Smallest eigenvectors- class indicators only up to linear transformation
linear transformation From eigenvectors to matting components
From eigenvectors to matting components Sparsity of matting components Minimize
From eigenvectors to matting components Minimize Newton’s method with initialization
K-means Projection into eigs space From eigenvectors to matting components 1) Initialization: projection of hard segments Smallest eigenvectors 2) Non linear optimization for sparse components
Extracted Matting Components
Brief Summary Construct Matting Laplacian Smallest eigenvectors Linear Transformation Matting components
Grouping Components + + =
Grouping Components • Unsupervised matting • User-guided matting Completeforeground matte + + =
Unsupervised matting Matting cost function Hypothesis: Generate indicating vector b
Unsupervised matting results
User-guided matting • Graph cut method Energy function Unary term Pairwise term Constrained components
Components with the scribble interface Components (our approach) Levin et al cvpr06 Wang&Cohen 05 Poisson Random Walk
Components with the scribble interface Components (our approach) Levin et al cvpr06 Wang&Cohen 05 Poisson Random Walk
Direct component picking interface Building foreground object by simple components addition + + =
Results
Quantitative evaluation
Spectral matting versus obtaining trimaps from a hard segmentation
Limitations • Number of eigenvectors Ground truth matte Matte from 70 eigenvectors Matte from 400 eigenvectors
Limitations • Number of matting components
Conclusion • Derived analogy between hard spectral segmentation to image matting • Automatically extract matting components from eigenvectors • Automate matte extraction process and suggest new modes of user interaction
