1. Texture Optimization for Example-based Synthesis Vivek Kwatra1,2Irfan Essa1Aaron Bobick1Nipun Kwatra1 1GVU Center/College of Computing, Georgia Tech 2Computer Science Dept., UNC Chapel Hill

2. Motivation + = InputTexture InputFlow Field OutputSequence

3. Example-based Texture Synthesis

4. Example-based Texture Synthesis • Dynamic texturing (animation/video) • Greater control: external knobs, e.g., flow, shape, illumination • Explicit texture quality evaluation and refinement

5. Contributions I • Global Optimization for texture synthesis • Global texture quality metric • Iterative multi-scale optimization

6. + = InputTexture InputFlow Field OutputSequence Contributions II • Flow-guided texture animation • Flow incorporated into optimization • Applications: fluid rendering, flow visualization

7. Related Work (a selected sample) TextureOptimization[Kwatra’05] MRF Annealing[Paget’98] Histogram Matching[Heeger’95] Local vs. Global Graphcut Textures[Kwatra’03] Non-parametricsampling[Efros’99,Wei’00] Image Quilting[Efros’01] Pixel vs. Patch

8. Related Work (a selected sample) Better Quality TextureOptimization[Kwatra’05] MRF Annealing[Paget’98] Histogram Matching[Heeger’95] Local vs. Global Graphcut Textures[Kwatra’03] Non-parametricsampling[Efros’99,Wei’00] Image Quilting[Efros’01] Pixel vs. Patch

9. Related Work (a selected sample) Better Quality TextureOptimization[Kwatra’05] MRF Annealing[Paget’98] Greater Controllability Histogram Matching[Heeger’95] Local vs. Global Graphcut Textures[Kwatra’03] Non-parametricsampling[Efros’99,Wei’00] Image Quilting[Efros’01] Pixel vs. Patch

10. Related Work • Flow-guided Synthesis • Advected Textures [Neyret’03] • Flow-based Video Editing [Bhat’04] • Optimization • Image-based Priors [Fitzgibbon’03] • Space-time Video Completion [Wexler’04]

11. Flow-guided Texture Animation • Flow consistency • Perceived motion similar to flow • Texture similarity • Shape, size, orientation of texture elements similar to source Source Texture Texture Similarity FlowConsistency Flowing Target Target Flow

12. Source Texture Texture Similarity Naive Approach • Ignore Texture Similarity • Warp each frame via flow field Source Texture Warp TargetFrames X0 X1 Xn Texture Similarity FlowConsistency Flowing Target Target Flow

13. Naive Approach Texture structure not maintained Source Texture Texture Similarity FlowConsistency Flowing Target Target Flow

14. Warp X0 X1 Xn Instead of… Warp TargetFrames

15. Warp X0 X1 Xn Instead of… Warp TargetFrames

16. SynthesizedFrame X– X Warp Correct W WarpedFrame Our Approach Warp + Correct TargetFrames

17. “Correct” Operation Link to original goals TargetFrames X– X Correct Warp W

18. “Correct” Operation Link to original goals TargetFrames X– X Correct Flow Consistency Warp W

19. “Correct” Operation Link to original goals Source Texture Similarity TargetFrames X– X Correct Flow Consistency Warp W

20. Optimization of Energy / Cost Energy = Flow Energy + Texture Energy Source Texture Similarity TargetFrames X– X Correct Flow Consistency Warp W

21. Flow Energy TargetFrames X– X Warp Flow Energy W

22. Texture Energy • Tricky! • Pixel-by-pixel comparison of source and target not possible • Compare texture “elements” • Local pixel neighborhoods • Want each target neighborhood to be similar to some source neighborhood

23. Texture Energy Z(source) X(target frame)

24. Energy for Single Neighborhood Z(source) p(pixel) X(target frame)

25. Energy for Single Neighborhood Z(source) Xp(neighborhood) X(target frame)

26. Energy for Single Neighborhood Z(source) Xp(neighborhood) X(target frame)

27. Energy for Single Neighborhood (nearest neighbor) Zp Z(source) Xp(neighborhood) X(target frame)

28. Energy for Single Neighborhood (nearest neighbor) Zp Z Xp(neighborhood) Texture Energy(single neighborhood) X

29. Texture Energy for Entire Image Z X

30. Texture Energy for Entire Image Z X =  individual neighborhood energy

31. Optimization • Optimize Total Energy of target frame: • Initialize: X W(target frame  warped frame) • Iteratively improve target frame

32. Optimization: Intuition Zp Texture Force Xp Source Target Frame Flow Force Wp Warped Frame

33. Optimization: Intuition Zp Xp Blend Source Target Frame Wp Warped Frame

34. Overlapping Neighborhoods Zp Xq Zq Blend Source Xp Target Frame Wp Warped Frame

35. We are not done yet! Nearest neighbor may change Zp Xp Source Target Frame Wp Warped Frame

36. Iterative Algorithm • Step 1 • Find Nearest Source Neighborhoods • Step 2 • Blend Source & Warped Neighborhoods • Repeat until convergence

37. Iterative Algorithm • Step 1 • Find Nearest Source Neighborhoods • Minimize E( X ) w.r.t {Zp} • Approximate nearest-neighbors • Hierarchical Clustering [Johnson’67] • Tree-Structured Vector Quantization [Wei’00]

38. Iterative Algorithm • Step 2 • Blend Source & Warped Neighborhoods • Minimize E( X ) w.r.t X • Set  Solve Linear System

39. Robust Optimization • Quadratic Texture Energy Etsensitive to outlier neighborhoods • Non-quadratic Etmore robust • Et( X ) = pkXp – Zpkr, 0 < r < 2

40. Robust Optimization • Quadratic Texture Energy Etsensitive to outlier neighborhoods • Non-quadratic Etmore robust • Et( X ) = pkXp – Zpkr, 0 < r < 2 • Solve weighted quadratic at each iteration Et( X ) = pkXp – Zpkr-2. kXp – Zpk2 = p p . kXp – Zpk2

41. Minimize Texture Energy Alone • Novel Static Texture Synthesis Algorithm Source Texture Similarity X– X Correct Flow Consistency W

42. Texture Energy Graph • Random Initialization • Multiple • Resolution Levels • Neighborhood Sizes • Progressively refined output

43. Texture Energy Graph • Random Initialization • Multiple • Resolution Levels • Neighborhood Sizes • Progressively refined output

44. Results: Texture Synthesis

46. Results: Texture Synthesis Input Wei-Levoy[Wei’00] Image Quilting[Efros’01] Graph-cuts[Kwatra’03] Texture Optimization[Kwatra’05]

47. Results: Texture Synthesis Image Quilting[Efros’01] Graph-cuts[Kwatra’03] Texture Optimization[Kwatra’05]

48. Results: Flow-guided Synthesis

49. Results: Timing • Static texture synthesis • 5-10 iterations per resolution level • Flowing texture synthesis • 20-60 seconds per frame

50. + = InputTexture InputFlow Field OutputSequence Summary • Global Optimization for controllable texture synthesis • Explicit evaluation of texture quality • Iterative progressive refinement of texture • Flow-guided texture animation within optimization framework