Extracting Smooth and Transparent Layers from a Single Image

1. Extracting Smooth and Transparent Layers from a Single Image Sai-Kit Yeung Tai-Pang Wu Chi-Keung Tang Vision and Graphics Group The Hong Kong University of Science and Technology CVPR 2008 Reporter: Chia-Hao Hsieh Date: 2009/12/29

2. Introduction • Layer decomposition • Under-constrained problem • Easier alternative • Only background layer has substantial image gradients • Small amount of user input • EM-HMM algorithm

3. Outline • Image Decomposition • Methods • Extract F • Expectation Maximization • Hidden Markov Model • Extract β • Results

4. Image Decomposition • Given a single image • Only the background layer has substantial image gradients and structures I: input image F: set of overlapping layers possibly with soft and transparent boundaries B: background layer β: smooth transparent layer

5. Extract F • F: set of overlapping layers possibly with soft and transparent boundaries • EM-HMM algorithm • Soft segmentation • For each pixel i, compute an optimal set of n soft labels • n: total number of color segments in the image • (n=2 in natural image matting if F is largely opaque)

6. Extract F • Color constraints for EM-HMM • From user-scribbled color samples • Expected color & soft labels (chicken & eggs) • Optimize by alternating optimization • EM algorithm • Collect color statistics using Gaussians

7. Extract F • The set of observations • μj and σj are respectively the mean and standard variation of the colors sampled inside region j • Let R = {ri} be the set of hidden variables that describes the classes labels at all pixels • ri = j if pixel i belongs to region j • |R| = the total number of pixels N to be processed (e.g. whole image or the pixels inside the silhouette)

8. Extract F • Objective function • P(O,R|Θ) is the complete-data likelihood to be maximized • Θ = {ci} is a set of parameters to be estimated • ciis the expected color at pixel I • EM algorithm • φ is the space containing all possible R with carinality equal to N

9. Expectation • marginal probability p(O|ri, Θ’) • If ri = j, ci should be similar to μi: • Let p(ri= j|Θ’) = 1/n be the mixture probability • Given Θ’ only,

10. Expectation • αij = p(ri = j | O, Θ’)

11. Maximization • Decompose P(O,R|Θ) into a combination of simple elements based on the Hidden Markov Model (HMM) assumptions • The hidden variable ri depends only on the hidden variables of its first-order-neighbors • The observation at i depends only on the hidden variable at i The HMM model for estimating the set of soft labels at each pixel.

12. Maximization

13. Maximization • Noise model • Since

14. Maximization • To maximize Q(Θ|Θ’), differentiate Q w.r.tci • M-Step, the updating rule (compute ci) • E-Step (compute αij) • E-Step and M-Step are iterated alternately until convergence • The initial assignment of ci is set as the pixel’s color Ii

15. Extract β • The user marks up on the image outside and inside of the transparent layer • The two marked-up regions O = {{μ1,σ1},{μ2,σ2}} • Modify the Bayesian MAP optimization in [15] to estimate βby incorporating α to improve the results • Optimal β* • B* is a rough estimation of the background without transparency attenuation The estimation of B∗: solve a Poisson equation subject to a guidance field [15] T.-P. Wu and C.-K. Tang. A bayesian approach for shadow extraction from a single image. ICCV05

16. Extract β • P(B*|β) • To discern true image structures from image gradients caused by transparency attenuation • Use α to encode the probability of the observed image gradient at pixel x caused by attenuation • σmis the uncertainty in the smoothness measurement

17. Extract β • Define the likelihood P(B*|β) as • which measures the fidelity between the image gradients of I’ and the estimated βB* weighted by m • {x,y} are first-order neighbors in the valid processing region of I’, obtained by masking out irrelevant regions by intelligent scissor and extracting F by EM-HMM • α1 is the standard deviation of the measurement error

18. Extract β • P (β). By assuming the transparent object to be homogeneous, we use the following smoothness prior P (β) weighted by mx,y as • where σ2is the uncertainty in the smoothness prior

19. Results

20. Results Layer decomposition from a single image Colorization

21. Conclusion • Layer separation from a single image • Easier but useful alternative • EM-HMM algorithm • separate smooth layers and the substantially-textured background from a single image • EM alternatively optimizes the soft label and the expected color at each pixel • HMM is used to maintain spatial coherency of the smooth layers • Preserve the image textures of the background layer by solving the Bayesian MAP estimation problem