Coherency Sensitive Hashing

1. Efficient Search Methods Coherency Sensitive Hashing & PatchMatch Simon KormanandShaiAvidan C. Barnes, E. Shechtman, A. Finkelstein, and D. B. Goldman. Presented by: Itamar Friedman

2. Nearest Neighbor Fields (NNF) • For each patch – find nearest patch in other image • Dense : Number of patches ≈ image size • Exact NN is infeasible • Error is mean(patch-distances) A B NNF

3. Why NNF? Non-Local Means denoising Non Local Means. Baudes, Coll, Morel. CVPR 2005

4. Why NNF? Image Completion Object Removal by Examplar Based Inpainting. Criminisi et al’, CVPR 2003 Space-Time Completion. Wexler et al’, CVPR 2004

5. Appearance Space k-by-kpatch = k2 dimensional vector Ordinary Nearest Neighbor search • Two popular methods are: • KD-trees [Arya et al, 1998] • Locality Sensitive Hashing (LSH) [Indyk et al, 1998] • These methods • Can be very accurate • Are relatively slow

6. Image Space • Based on coherency

7. Each iteration, for each pixel: Compare two kinds of candidates (patches): 1. Randomly located patches (with probability that drops exponentially with distance from current match) Image Space - the PatchMatch algorithm[Barnes et al, SIGRAPH 2009] current match candidates

8. Each iteration, for each pixel: Compare two kinds of candidates (patches): 2. Propagated neighbor’s matches Image Space - the PatchMatch algorithm[Barnes et al, SIGRAPH 2009] left neighbor current match of left neighbor candidate

9. Each iteration, for each pixel: Compare two kinds of candidates (patches): 2. Propagated neighbor’s matches Image Space - the PatchMatch algorithm[Barnes et al, SIGRAPH 2009] current match of top neighbor top neighbor left neighbor current match of left neighbor candidates

10. An iterative randomized algorithm Simple and fast (runtime is in the seconds) Based on local propagation and mostly-local search Heavily relies on the fact that images are generally coherent Image Space - the PatchMatch algorithm[Barnes et al, SIGRAPH 2009]

11. Coherency Sensitive Hashing (CSH) Combine image space and appearance space • Find good matches • Propagate them across patches that either: • Are neighbors (image space) • Share hash values (appearance space)

12. Walsh Hadamard (WH) Basis 3 4 5 6 7 8 11 10 6 7 8 9 4 5 6 7 8 9 CSH - The LSH Framework • Project each point v on a random segmented line Hash function (code) Bin number LSH table Datar, Immorlica, Indyk, Mirrokni, 2004 The Gray-Code Filter Kernels, Hel-Or, Hel-Or, PAMI 2007

13. CSH indexing • Hash all patches of both images (with hash function g) Hash Table Image ‘A’ Image ‘B’ g g

14. CSH indexing • Hash all patches of both images (with hash function g) • Keep k (=2) representatives of each image in each entry Hash Table k k Image ‘A’ Image ‘B’ g g-1 g-1 g

15. CSH indexing - candidates Candidates = potential matching patches Image ‘A’ Image ‘B’

16. CSH indexing – candidate types Type I (appearance only) = Standard LSH Image ‘A’ Image ‘B’

17. CSH indexing – candidate types Type II (appearance and coherence) Image ‘A’ Image ‘B’ PM candidate

18. CSH indexing – candidate types Type III (appearance only) Image ‘A’ Image ‘B’

19. CSH indexing – candidate types Type 1 Type 2 Type 3 (> PM) (LSH)

20. CSH Algorithm – recap Preprocessing: • Project A’s and B’s patches on WH vectors Iterate (typically 1-3 times): • Hash patches into a table • For each patch in A • Create candidate patches in B • Pick the best one *further details in the paper A B

21. CSH Algorithm – recap Preprocessing: • Project A’s and B’s patches on WH vectors Iterate (typically 1-3 times): • Hash patches into a table • For each patch in A • Create candidate patches in B • Pick the best one *further details in the paper A B

22. CSH Algorithm – Hashing 1 6 5 2 4 3 3 4 2 1

23. CSH Algorithm – Hashing kernel (DC) Y channel – 32 bins kernel (DC) Cb channel – 4 bins kernel Y channel - 8 bins kernel Y channel - 2 bins

24. CSH Algorithm – Hashing • The reasoning being: • number of bins - relative to the dispersion of points along the projection line • Less bins the higher the kernel frequency • Less bins for color channels compared to Y • spacing of bins – attempting to capture a constant number of points in each bin • Use fixed percentiles • over the distribution

25. CSH Algorithm – recap Preprocessing: • Project A’s and B’s patches on WH vectors Iterate (typically 1-3 times): • Hash patches into a table • For each patch in A • Create candidate patches in B • Pick the best one *further details in the paper A B

26. CSH Algorithm – comparing candidates

27. Experiments - setup • Vid-Pairs data-set • 133 related pairs of ~2 MP images • Up to 30 frames apart in the video • Include both camera and object motion A B

28. Experiments - accuracy/efficiency

29. Experiments – Image Reconstruction • Reconstruct image A, given: B and A->B • Each pixel is an avg. of corresponding pixels through the mapping • Main ingredient in many patch based algorithms (retargetting, inpainting, denoising, super-resolution, etc.) Image ‘A’ Image ‘B’ ? PM NNF CSH GT

30. Experiments – Image Reconstruction

31. Experiments – Image Reconstruction

32. Experiments – Image Reconstruction

33. Experiments – Image Reconstruction

34. Experiments – Image Reconstruction PM vs. GT CSH vs. GT

35. Dependence on patch energy

36. Experiments – Image Reconstruction Image ‘A’ Image ‘B’ ? NNF

37. Experiments – Image Reconstruction • Reconstruct image A, • given only image B and an NNF mapping A->B

38. Experiments – Image Reconstruction • Reconstruct image A, • given only image B and an NNF mapping A->B

39. Experiments – Image Reconstruction • Reconstruct image A, • given only image B and an NNF mapping A->B

40. Experiments – Image Reconstruction • Reconstruct image A, • given only image B and an NNF mapping A->B

41. Experiments – Image Reconstruction PM vs. GT CSH vs. GT

42. Experiments – Mapping Incoherency CSH PM

43. Experiments – Mapping (In)coherency Image ‘A’ Image ‘B’ CSH PM ‘x’-coordinate of PM mapping ‘x’-coordinate of CSH mapping

44. Reconstruction-Error vs. Incoherency Good (exact NN) bad

45. Summary - CSH • A new algorithm for NNF • combines appearance space and image space accuracy speed • other pleasing properties • Reconstruction, Incoherency, Texture • Limitations • Small data (images, patches) • Closely related image pairs • Additional memory for hash tables • Currently doesn’t support scale, rotation, illumination

46. Evaluation – CSH & PatchMatch • Well written. Self contained. • Novel. Technically convincing. • Well tested and evaluated. • Provide materials and code.

47. Future Work • The CSH shows inferior results with flat patches. Is it because of: (???) • The WH representation/functions • Noise in the flat patches • Distribution of selected/matching patches • Overcome limitations • “Smart” selection from a big set of data (images, patches) • Support scale, rotation, illumination by using a different representation such as {SIFT,DAISY,…}

48. Reference • PatchMatch: source code, data-set, more info, available at: • http://gfx.cs.princeton.edu/pubs/Barnes_2009_PAR/ • CSH: source code, data-set, more info, available at: • http://www.eng.tau.ac.il/~simonk/CSH • Huge gratitude to CSH team for their presentation published in their site.

49. THANK YOU!