Common Visual Pattern Discovery via Spatially Coherent Correspondences

1. CVPR10 Common Visual Pattern Discovery via Spatially Coherent Correspondences Speaker: Hairong Liu Authors : Hairong Liu, Shuicheng Yan Learning & Vision Research Group, ECE, National University of Singapore http://www.lv-nus.org/

2. What is Common Visual Pattern? Common Visual Pattern: A set of feature points share similar local features as well as similar spatial layout

3. General Correspondence Problem Large number of outliers Different kinds of noises Correspondence Problem Different types of mappings (one-to-one, one-to-many, many-to-many)

4. Schematic Illustration of Our Approach Spatial coherence of two correspondences • First, find all candidate correspondences by local features • Second, construct correspondence graph • Third, obtain all dense subgraphs • Fourth, recover all correspondence configurations T T x* G (b) (c) (a) (d)

5. Candidate Correspondences Two images with common visual pattern A large number of candidate correspondences Local feature is not enough, need spatial information

6. Dynamic Correspondence Graph Spatial coherence of two correspondences l1 • Each vertex represents a candidate correspondence • The weight of edge represents spatial coherence, that is: • Weight is a function of s, and common visual patterns correspond to dense subgraphs at correct scales T Spatial Coherency l2 Weight G sis the scale factor between two images

7. Graph Density Graph density f(x) represents average affinity of a subgraph A is weighted adjacency matrix mis unknown! StQP

8. Graph Mode (Dense Subgraph) Graph Modes f(x) Graph Graph Modes correspond to the peaks of graph density f(x)

9. Probabilistic Coordinate x xi represents the probability of cluster x contains vertex i ri(x)=(Ax)i,the reward at vertex i, represents the affinity between vertex iand the cluster x

10. Graph Mode and Feature Mode Feature Mode Graph Mode • Similar Point: both reflect dense regions in data • Differences: • In graph, no self-contribution • Graph mode is only the composite effect of points within the cluster, thus is inherently robust to noises and outliers

11. Properties of Graph Mode Lagrangian Function r = f(x*) x* KKT Conditions r > f(x*) r < f(x*) x* is a graph mode

12. Solution Replicator Equation Drop vertices to form very dense subgraph

13. Relation Space Analysis Run the procedure in parallel Tessellate Relation space

14. Two Sampling Strategies Each vertex is a sample (ICML10) No sampling strategy can guarantee to find all modes; but in applications, we can find all significant modes with very high probability The sufficiently large neighborhood of each vertex forms a sample (CVPR10)

15. Recover Correspondences • From each maximizer x*, we can recover a correspondence configuration • Large components have high priorities

16. Experiments & Results • Tasks • Point Set Matching. (Randomly generated point sets) • Image Matching. (ETHZ toy images) • Near-duplicate Image Retrieval. (Columbia dataset) • Goal • Robustness to noises • Robustness to large amount of outliers • Different correspondence configurations (one to one, one to many, many to many)

17. Exp-A: Point Set Matching Our method is more robust to noises Add outlies to both point sets Q Our method is nearly not affected by outliers Add noises Rotation P Test the robustness of all three methods to noises and outliers

18. Exp-A: Point Set Matching (cont) Large peaks indicate correct scales Detect similar point sets at different scales

19. Exp-B: Image Matching Our method can find all correspondence configurations Different correspondence configurations, different scales

20. Exp-C: Near-duplicate Image Retrieval Correspondences between near duplicate images

21. Exp-C: Near-duplicate Image Retrieval (cont) 150 near-duplicate pairs and 300 non-duplicate images Although simple, our method still get best result

22. Summary & Future work • Contributions in this work • Dynamic correspondence graph • Obtain all significant dense subgraphs by systematically sampling • Future work • Better approximation method • Better sampling strategies • Automatically select scales

23. Code • We extended our CVPR work (a very preliminary study) and ICML work, and proposed an efficient, non-parametric tool–Graph Shift, which can be thought as the counterpart of mean shift algorithm on graph . The code is available at: http://www.lv-nus.org/GraphShiftCode.zip • If you can formulate your problem into the problem of detecting dense subgraphs, it will be very helpful, especially for HUGE graph. • You can freely test and use it for academic purpose.