Our Setting: 2D Texture Synthesis

1. Fast and High Quality Overlap Repair for Patch-Based Texture SynthesisAndrew NealenMarc AlexaDiscrete Geometric Modeling Group (DGM)Technische Universität Darmstadt

2. nxm Input Texture NxM Output Texture Our Setting: 2D Texture Synthesis • The goal: Synthesize an output texture which isperceptually similarto the input texture. Also ensure that the result contains sufficient variation.

3. Patch-Based Texture SynthesisSome Existing Methods • A Very Popular 2D Texture Synthesis Method • Image Quilting [Efros and Freeman 2001] • Graphcut Texures [Kwatra et. al 2003] • Wang Tiles [Cohen et. al 2003]

4. Patch-Based Texture SynthesisSome Existing Methods • A Very Popular 2D Texture Synthesis Method • Image Quilting [Efros and Freeman 2001] • Graphcut Texures [Kwatra et. al 2003] • Wang Tiles [Cohen et. al 2003]

5. A B Patch-Based Texture SynthesisSome Existing Methods • A Very Popular 2D Texture Synthesis Method • Image Quilting [Efros and Freeman 2001] • Graphcut Texures [Kwatra et. al 2003] • Wang Tiles [Cohen et. al 2003]

6. A B Patch-Based Texture SynthesisSome Existing Methods • A Very Popular 2D Texture Synthesis Method • Image Quilting [Efros and Freeman 2001] • Graphcut Texures [Kwatra et. al 2003] • Wang Tiles [Cohen et. al 2003]

7. A B Patch-Based Texture SynthesisSome Existing Methods • A Very Popular 2D Texture Synthesis Method • Image Quilting [Efros and Freeman 2001] • Graphcut Texures [Kwatra et. al 2003] • Wang Tiles [Cohen et. al 2003]

8. A B 2 _ Patch-Based Texture SynthesisSome Existing Methods • A Very Popular 2D Texture Synthesis Method • Image Quilting [Efros and Freeman 2001] • Graphcut Texures [Kwatra et. al 2003] • Wang Tiles [Cohen et. al 2003]

9. A B 2 _ = overlap error Patch-Based Texture SynthesisSome Existing Methods • A Very Popular 2D Texture Synthesis Method • Image Quilting [Efros and Freeman 2001] • Graphcut Texures [Kwatra et. al 2003] • Wang Tiles [Cohen et. al 2003]

10. A B 2 _ = overlap error Patch-Based Texture SynthesisSome Existing Methods • A Very Popular 2D Texture Synthesis Method • Image Quilting [Efros and Freeman 2001] • Graphcut Texures [Kwatra et. al 2003] • Wang Tiles [Cohen et. al 2003]

11. A B 2 _ = overlap error Patch-Based Texture SynthesisSome Existing Methods • A Very Popular 2D Texture Synthesis Method • Image Quilting [Efros and Freeman 2001] • Graphcut Texures [Kwatra et. al 2003] • Wang Tiles [Cohen et. al 2003]

12. A B 2 _ = overlap error Patch-Based Texture SynthesisSome Existing Methods • A Very Popular 2D Texture Synthesis Method • Image Quilting [Efros and Freeman 2001] • Graphcut Texures [Kwatra et. al 2003] • Wang Tiles [Cohen et. al 2003]

13. Patch-Based Texture SynthesisSome Existing Methods • A Very Popular 2D Texture Synthesis Method • Image Quilting [Efros and Freeman 2001] • Graphcut Texures [Kwatra et. al 2003] • Wang Tiles [Cohen et. al 2003]

14. Patch-Based Texture SynthesisSome Existing Methods • A Very Popular 2D Texture Synthesis Method • Image Quilting [Efros and Freeman 2001] • Graphcut Texures [Kwatra et. al 2003] • Wang Tiles [Cohen et. al 2003]

15. Patch-Based Texture SynthesisHybrid Texture Synthesis (HTS) • Introduced at EGSR 2003 [Nealen and Alexa] • Adaptive Patch Sampling, like Hierarchical Pattern Mapping [Soler et. al 2002] • Per-Pixel Overlap Re-synthesis

16. Input (n x m) Intermediate Result Goal: From nxm, synthesize NxM similar, but not identical Result (N x M) Result (N x M) Hybrid Texture SynthesisMethod

17. Input (n x m) Intermediate Result Hybrid Texture SynthesisMethod Goal: From nxm, synthesize NxM similar, but not identical Result (N x M) Result (N x M)

18. Patch-Search in the Input + Copy to Result + Mark Invalid Pixels Input (n x m) Intermediate Result Result (N x M) Result (N x M) Hybrid Texture SynthesisMethod Goal: From nxm, synthesize NxM similar, but not identical

19. Patch-Search in the Input + Copy to Result + Mark Invalid Pixels Input (n x m) Intermediate Result Per-Pixel Re-synthesis Steps (for each Patch) Goal: From nxm, synthesize NxM similar, but not identical Result (N x M) Result (N x M) Hybrid Texture SynthesisMethod

20. Patch-Search in the Input + Copy to Result + Mark Invalid Pixels Input (n x m) Intermediate Result Per-Pixel Re-synthesis Steps (for each Patch) Goal: From nxm, synthesize NxM similar, but not identical Result (N x M) Result (N x M) Hybrid Texture SynthesisMethod

21. Patch-Search in the Input + Copy to Result + Mark Invalid Pixels Input (n x m) Intermediate Result Per-Pixel Re-synthesis Steps (for each Patch) Goal: From nxm, synthesize NxM similar, but not identical Result (N x M) Result (N x M) Hybrid Texture SynthesisMethod

22. Patch-Search in the Input + Copy to Result + Mark Invalid Pixels Input (n x m) Intermediate Result Per-Pixel Re-synthesis Steps (for each Patch) Goal: From nxm, synthesize NxM similar, but not identical Result (N x M) Result (N x M) Hybrid Texture SynthesisMethod

23. Patch-Search in the Input + Copy to Result + Mark Invalid Pixels Input (n x m) Intermediate Result Per-Pixel Re-synthesis Steps (for each Patch) Result (N x M) Result (N x M) Hybrid Texture SynthesisMethod Goal: From nxm, synthesize NxM similar, but not identical

24. Hybrid Texture SynthesisGeneralization: Pro and Con • Pro: General Method for Overlap Repair • Complementary to other Methods, such as Minimum-Error-Boundary-Cut (MEBC) or Feathering, yet oftentimes produces better results • Generalizes to arbitrary patch shapes, i.e. is applicable to Graphcut Textures, Wang Tiles, etc. • Con: Computationally Expensive • Exhaustive search for each invalid pixel in the overlap, based on mostly irregular valid neighborhood • Has O(rN log N) complexity -> Doesn‘t scale well.

25. Hybrid Texture SynthesisGeneralization: Pro and Con • Pro: General Method for Overlap Repair • Complementary to other Methods, such as Minimum-Error-Boundary-Cut (MEBC) or Feathering, yet oftentimes produces better results • Generalizes to arbitrary patch shapes, i.e. is applicable to Graphcut Textures, Wang Tiles, etc. • Con: Computationally Expensive • Exhaustive search for each invalid pixel in the overlap, based on mostly irregular valid neighborhood • Has O(rN log N) complexity -> Doesn‘t scale well.

26. Hybrid Texture SynthesisGeneralization: Pro and Con • Pro: General Method for Overlap Repair • Complementary to other Methods, such as Minimum-Error-Boundary-Cut (MEBC) or Feathering, yet oftentimes produces better results • Generalizes to arbitrary patch shapes, i.e. is applicable to Graphcut Textures, Wang Tiles, etc. • Con: Computationally Expensive • Exhaustive search for each invalid pixel in the overlap, based on mostly irregular valid neighborhood • Has O(rN log N) complexity -> Doesn‘t scale well.

27. Fast Overlap RepairBasic Idea • Inspiration • Ashikhmin: Synthesizing Natural Textures [2001] termed Coherence Search • Tong et. al‘s extension: k-Coherence Search [2002] • Basic Idea: Intelligently Reduce Search Space • Only search within a set of coherent pixels • Introduce Trade-off between quality and speed

28. Input Texture Intermediate Result + Source Map Fast Overlap Repair Coherence Search • Applying Coherence Search • For each pixel in the output, store its location in the input in a source map (same size as the output texture)

29. Source Map Lookup Input Texture Intermediate Result + Source Map Fast Overlap Repair Coherence Search • Applying Coherence Search • When searching for a new pixel, only consider input pixels which are coherent with neighboring output pixels

30. Source Map Lookup Input Texture Intermediate Result + Source Map Consequence in this example: Only two possible candidates Fast Overlap Repair Coherence Search • Applying Coherence Search • When searching for a new pixel, only consider input pixels which are coherent with neighboring output pixels

31. Example: 64x64 Texture Synthesized from four 32x32 Patches Coherence Exhaustive Fast Overlap Repair Coherence Search • Applying Coherence Search • Simply comparing to the coherent pixels results in seams similar to Image Quilting (MEBC)

32. Example: 64x64 Texture Synthesized from four 32x32 Patches Coherence Exhaustive Fast Overlap Repair Coherence Search • Applying Coherence Search • Simply comparing to the coherent pixels results in seams similar to Image Quilting (MEBC)

33. Example: 64x64 Texture Synthesized from four 32x32 Patches Coherence Exhaustive Fast Overlap Repair Coherence Search • Applying Coherence Search • Simply comparing to the coherent pixels results in seams similar to Image Quilting (MEBC)

34. Source Map Lookup Input Texture Intermediate Result + Source Map Fast Overlap Repair k-Coherence Search • Better: Applying k-Coherence Search

35. Source Map Lookup Input Texture Intermediate Result + Source Map Fast Overlap Repair k-Coherence Search • Better: Applying k-Coherence Search • Extend the set by the k-nearest neighbors (knn) of each coherent pixel (in feature space) and remove duplicates

36. Fast Overlap Repair k-Coherence Search • Precomputation of knn Data Structure • Performed once for each nxm input texture and stored for repeated use • User defines size of box-shaped neighborhood np • For each of the nxm input pixels • Construct feature vector by ordered concatenation of the npx npRGB-triples in the box-shaped neighborhood • Dimension reduction (75-90%) by applying PCA • Compute indices of k-nearest neighbors to each pixel

37. Fast Overlap Repair k-Coherence Search • Source Map Maintenance • Each valid pixel in the overlap region is a linear blend (feathering) of at least two original pixel values, i.e. from at least two different sources • To avoid the maintenance of multiple source maps, simply store the source of the pixel with greatest contribution in a singlesource map Blue: invalid overlap pixels

38. k = 1 k = 4 k = 11 Exhaustive Results varying k

39. k = 1 k = 4 k = 11 Exhaustive Results varying k

40. k = 1 k = 4 k = 11 Exhaustive Results varying k

41. k = 1 k = 4 k = 11 Exhaustive Results varying k

42. Results timings Exhaustive n = 7x7 k-Coherence n = 3x3 | k = 5 k-Coherence n = 5x5 | k = 11 Input scales 64×64 δmax = 0.02 Δmax = 0.05 Pre: 0 sec. Synth: 283 sec. Pre: 6+3 sec. Synth: 226 sec. Pre: 6+4 sec. Synth: 427 sec. rock 128×128 δmax = 0.02 Δmax = 0.05 Pre: 45+62 sec. Synth: 226 sec. Pre: 45+74 sec. Synth: 415 sec. Pre: 0 sec. Synth: 533 sec. stonewall 200×200 δmax = 0.02 Δmax = 0.03 Pre: 0 sec. Synth: 985 sec. Pre: 247+28 s Synth: 178 sec. Pre: 247+37 s Synth: 350 sec.

43. Results timings Exhaustive n = 7x7 k-Coherence n = 3x3 | k = 5 k-Coherence n = 5x5 | k = 11 Input scales 64×64 δmax = 0.02 Δmax = 0.05 Pre: 0 sec. Synth: 283 sec. Pre: 6+3 sec. Synth: 226 sec. Pre: 6+4 sec. Synth: 427 sec. rock 128×128 δmax = 0.02 Δmax = 0.05 Pre: 45+62 sec. Synth: 226 sec. Pre: 45+74 sec. Synth: 415 sec. Pre: 0 sec. Synth: 533 sec. stonewall 200×200 δmax = 0.02 Δmax = 0.03 Pre: 0 sec. Synth: 985 sec. Pre: 247+28 s Synth: 178 sec. Pre: 247+37 s Synth: 350 sec.

44. Results timings Exhaustive n = 7x7 k-Coherence n = 3x3 | k = 5 k-Coherence n = 5x5 | k = 11 Input scales 64×64 δmax = 0.02 Δmax = 0.05 Pre: 0 sec. Synth: 283 sec. Pre: 6+3 sec. Synth: 226 sec. Pre: 6+4 sec. Synth: 427 sec. rock 128×128 δmax = 0.02 Δmax = 0.05 Pre: 45+62 sec. Synth: 226 sec. Pre: 45+74 sec. Synth: 415 sec. Pre: 0 sec. Synth: 533 sec. stonewall 200×200 δmax = 0.02 Δmax = 0.03 Pre: 0 sec. Synth: 985 sec. Pre: 247+28 s Synth: 178 sec. Pre: 247+37 s Synth: 350 sec.

45. Results timings Exhaustive n = 7x7 k-Coherence n = 3x3 | k = 5 k-Coherence n = 5x5 | k = 11 Input scales 64×64 δmax = 0.02 Δmax = 0.05 Pre: 0 sec. Synth: 283 sec. Pre: 6+3 sec. Synth: 226 sec. Pre: 6+4 sec. Synth: 427 sec. rock 128×128 δmax = 0.02 Δmax = 0.05 Pre: 45+62 sec. Synth: 226 sec. Pre: 45+74 sec. Synth: 415 sec. Pre: 0 sec. Synth: 533 sec. stonewall 200×200 δmax = 0.02 Δmax = 0.03 Pre: 0 sec. Synth: 985 sec. Pre: 247+28 s Synth: 178 sec. Pre: 247+37 s Synth: 350 sec.

46. Results timings Exhaustive n = 7x7 k-Coherence n = 3x3 | k = 5 k-Coherence n = 5x5 | k = 11 Input scales 64×64 δmax = 0.02 Δmax = 0.05 Pre: 0 sec. Synth: 283 sec. Pre: 6+3 sec. Synth: 226 sec. Pre: 6+4 sec. Synth: 427 sec. rock 128×128 δmax = 0.02 Δmax = 0.05 Pre: 45+62 sec. Synth: 226 sec. Pre: 45+74 sec. Synth: 415 sec. Pre: 0 sec. Synth: 533 sec. stonewall 200×200 δmax = 0.02 Δmax = 0.03 Pre: 0 sec. Synth: 985 sec. Pre: 247+28 s Synth: 178 sec. Pre: 247+37 s Synth: 350 sec.

47. Results timings Exhaustive n = 7x7 k-Coherence n = 3x3 | k = 5 k-Coherence n = 5x5 | k = 11 Input scales 64×64 δmax = 0.02 Δmax = 0.05 Pre: 0 sec. Synth: 283 sec. Pre: 6+3 sec. Synth: 226 sec. Pre: 6+4 sec. Synth: 427 sec. rock 128×128 δmax = 0.02 Δmax = 0.05 Pre: 45+62 sec. Synth: 226 sec. Pre: 45+74 sec. Synth: 415 sec. Pre: 0 sec. Synth: 533 sec. stonewall 200×200 δmax = 0.02 Δmax = 0.03 Pre: 0 sec. Synth: 985 sec. Pre: 247+28 s Synth: 178 sec. Pre: 247+37 s Synth: 350 sec.

48. Results timings Exhaustive n = 7x7 k-Coherence n = 3x3 | k = 5 k-Coherence n = 5x5 | k = 11 Input scales 64×64 δmax = 0.02 Δmax = 0.05 Pre: 0 sec. Synth: 283 sec. Pre: 6+3 sec. Synth: 226 sec. Pre: 6+4 sec. Synth: 427 sec. rock 128×128 δmax = 0.02 Δmax = 0.05 Pre: 45+62 sec. Synth: 226 sec. Pre: 45+74 sec. Synth: 415 sec. Pre: 0 sec. Synth: 533 sec. stonewall 200×200 δmax = 0.02 Δmax = 0.03 Pre: 0 sec. Synth: 985 sec. Pre: 247+28 s Synth: 178 sec. Pre: 247+37 s Synth: 350 sec.

49. Results timings Exhaustive n = 7x7 k-Coherence n = 3x3 | k = 5 k-Coherence n = 5x5 | k = 11 Input scales 64×64 δmax = 0.02 Δmax = 0.05 Pre: 0 sec. Synth: 283 sec. Pre: 6+3 sec. Synth: 226 sec. Pre: 6+4 sec. Synth: 427 sec. rock 128×128 δmax = 0.02 Δmax = 0.05 Pre: 45+62 sec. Synth: 226 sec. Pre: 45+74 sec. Synth: 415 sec. Pre: 0 sec. Synth: 533 sec. stonewall 200×200 δmax = 0.02 Δmax = 0.03 Pre: 0 sec. Synth: 985 sec. Pre: 247+28 s Synth: 178 sec. Pre: 247+37 s Synth: 350 sec.

50. Results timings Exhaustive n = 7x7 k-Coherence n = 3x3 | k = 5 k-Coherence n = 5x5 | k = 11 Input scales 64×64 δmax = 0.02 Δmax = 0.05 Pre: 0 sec. Synth: 283 sec. Pre: 6+3 sec. Synth: 226 sec. Pre: 6+4 sec. Synth: 427 sec. rock 128×128 δmax = 0.02 Δmax = 0.05 Pre: 45+62 sec. Synth: 226 sec. Pre: 45+74 sec. Synth: 415 sec. Pre: 0 sec. Synth: 533 sec. stonewall 200×200 δmax = 0.02 Δmax = 0.03 Pre: 0 sec. Synth: 985 sec. Pre: 247+28 s Synth: 178 sec. Pre: 247+37 s Synth: 350 sec.