Image Quilting for Texture Synthesis and Transfer

1. Image Quilting for Texture Synthesis and Transfer Alexei A. Efros (UC Berkeley) William T. Freeman (MERL) Siggraph01’

2. About author? • Alexei A. Efros • Assistant Professor • Computer Science Department • & The Robotics Institute • School of Computer Science • Carnegie Mellon University • From St. Petersburg, Russia • Got PhD from UC Berkeley in 2003 • Then a year as a Visiting Research Fellow in Visual • Geometry Group of Oxford • Joined in CSD and RI in autumn of 2004 • Computer graphics & computer vision

3. About author? • William T. Freeman • Professor of Electrical Engineering & • Computer Science at the • Artificial Intelligence • Laboratory at MIT(September, 2001) • Received a BS in physics and MS in electrical • engineering from Stanford (1979), and an MS • in applied physics from Cornell(1981) • Got his PhD in 1992 from the MIT • Worked at Mitsubishi Electric Research Labs (1992 – 2001, • Cambridge) • Computer vision

4. + = Quilting? Transfer?

5. Image vs. Texture

6. Example-based Texture Synthesis Input Example

7. input image SYNTHESIS True (infinite) texture generated image The Goal of Texture Synthesis Same in perceptual sense

8. The Challenge • Texture analysis: how to capture the essence of texture? • Need to model the whole spectrum: from repeated to stochastic texture Repeated Stochastic Both?

9. Related Work • Local region-growing method • -Pixel-based • -Patch-based • Global optimization-based method • Heeger and Bergen sig95,Pyramid-based texture synthesis • Paget and Longstaff IEEE Tran… 98,Texture synthesis via a noncausal nonparametric multiscale markov random field • ….. • Physical Simulation et al

10. B1 B1 B2 B2 Neighboring blocks constrained by overlap Minimal error boundary cut block Input texture B1 B2 Random placement of blocks

11. 2 _ = overlap error min. error boundary Minimum Error Boundary Cut overlapping blocks vertical boundary

12. Minimum Error Boundary Cut

13. The Image Quilting Algorithm • Pick size of block and size of overlap • Synthesize blocks in raster order • Search input texture for block that satisfies overlap constraints (above and left) • Easy to optimize using NN search [Liang et.al., ’01] • Paste new block into resulting texture • Compute minimal error boundary cut

14. Synthesis Results

15. Synthesis Results

16. Synthesis Results

17. Synthesis Results

18. Synthesis Results

19. Synthesis Results

20. Portilla & Simoncelli Xu, Guo & Shum input image Wei & Levoy Image Quilting

21. Portilla & Simoncelli Xu, Guo & Shum input image Wei & Levoy Image Quilting

22. Portilla & Simoncelli Xu, Guo & Shum input image Wei & Levoy Image Quilting

23. Failures

24. Image Quilting vs. Graph Cut Input Image Quilting Graph Cut (siggraph 03’)

25. Luminance Constraint + Texture Transfer

27. parmesan + = rice + =

28. Conclusion • No multi-scale, no one-pixel-at-a-time! • fast and very simple • Improved stability • Results are not bad

29. Thanks a lot! Happy New Year!

30. Pixel-based Methods • Compare local causal neighbourhoods • Efros and Leung (ICCV ’99) Wei and Levoy (Siggraph 2000) Ashikhmin (I3D 2001) Input Output

31. Patch-based Methods • Copy patches of pixels rather than single pixels Chaos Mosaic, Xu et al, 1997 Patch-Based Sampling, Liang et al(ACM 2001) Image Quilting, Efros and Williams(Siggraph 2001)