Top-Down & Bottom Up Segmentation

1. Top-Down & Bottom Up Segmentation Reem Amara Presented to : Prof.Hagit Hel-Or

2. Content of the slides 1- Present the bottom-up algorithm. 2- Present the top-down algorithm. 3- Present the combined algorithm. Tip - if you don’t like horses this isn’t the right place !

3. Let’s get started >>>

4. Bottom-up segmentation The goal is to identify an object in an image and separate it from the background. The bottom-up approach is to first segment the image into regions and then identify the image regions that correspond to a single object.

5. What are we looking for? Pixels that have something in common. Pixels that belong together. Similar pixels.

6. How to identify the object regions from the image ? Relying mainly on continuity principles . This means we group pixels according to Grey level. Texture uniformity. Smoothness and continuity of bounding contours.

7. Similar colors –intensity assign to color categories What else we can do ?

8. Similar texture assign to texture categories

9. Difficulties an object may be segmented into multiple regions. may merge an object with its background.

10. The bottom-up segmentation tree example input image :

11. Bottom-up segmentation of input image at multiple scales.

12. Segmentation tree. The bottom up are organized in a tree structure The colors of segments match the color of nodes

13. Fast Multiscale Image Segmentation Cast the segmentation problem as a graph clustering problem . Given an image that contains N = n*n pixels. Construct a graph in which each node represents a pixel . Every two nodes representing neighboring are connected by an arc.

14. Planar graph G(V,E,W) E – edges of G. V – vertex of G , index i- [1….N] . Ii – intensity value . W – wij is the weight associated with each edge for example - wij = |Ii –Ij|, reflecting the degree to which they tend to belong to the same segment.

15. The pixel graph (example) strong coupling weak coupling

16. ?How to detect a segment Associate with a graph a state vector u = (u1,u2,……,uN). ui – state variable associate with the pixel i.

17. Energy Function In an ideal segment (with only binary state) E(S) sums the coupling value along the segment boundary.

18. Energy function (2) -predetermined parameter To avoid preference of small/very large segment , the energy function divided by the volume of the segment.

19. Example W54 = 5 W58 = 8 W52 =3W56 = 1 E(5) = 5 +8+3+1 =16 N(5) = 5

21. Salient segment The segments that yield small values for the functional and whose volume is less than half the size of the image, are considered salient.

22. Choosing a Coarse Grid Given a graph . A set of representative nodes so that every node in is strongly connected to C. is associated With C

23. illustration -Coarse Grid

24. Interpolation matrix Because the original graph is local and because every node is strongly connected to C, there exists a sparse Interpolation matrix P:

26. Interpolation matrix – illustration

27. Weighted aggregation every node k in C can be thought of as representing an aggregation of pixels. for example pixel “i “ belongs to the k’th aggregation with weigh Pik(0,1) decomposition of the image into aggregates

28. This eq. used to generate G[s]

29. Influences only the internal weight of an aggregate

31. The bottom-up algorithm Segmentation by Weighted Aggregations (SWA) We are treating our graph as a grid graph, starting from the most Refined grid, and we coarsen it at each step. First choose ½ your nodes as representatives “C” ,Choose those so that each node in your graph is “strongly”

32. The bottom-up algorithm (cont.) Now we will aggregate all the nodes which are strongly coupled to a node in C, to that node, so that we eliminate a big amount of the nodes. Now each node Corresponds to an aggregate of pixels, not just a single one.

33. The bottom-up algorithm (cont.) Recalculate the aggregate properties. Recalculate the edges weight accordingly. Now apply the same to new nodes.

34. Conclusion At the end of this process a full pyramid has been constructed. Every salient segment appears as an aggregate in some level of the pyramid.

36. Results

37. Result(2)

38. Result(3)

39. Result (4) input image scale 11 scale 8 scale 7 the smaller the scale the smaller the segments .

41. What if I told you there is a horse in the image ?

42. Top –down segmentation rely on acquired class-specific information, and can only be applied to images from a specific class. segmentation approach is to use known shape characteristics of objects within a given class to guide the segmentation process.

43. Top- down  jigsaw puzzle The construction of an object by fragments is somewhat similar to the assembly of a jigsaw puzzle, where we try to put together a set of pieces such that their templates form an image similar to a given example.

44. The goal The goal is to cover as closely as possible the images of deferent objects from a given class, using a set of more primitive shapes. How ? to identify useful “building blocks”- a collection of components that can be used to identify and delineate the boundaries of objects in the class.

45. What are we looking for in an image? image fragments that are strongly correlated with images containing the desired object class.

46. Example Class – horse class fragments→ stored in memory

47. Fragments representation in memory . Stage 1 – 1- Divide set of training images into class images (C) and non-class images (NC) . 2- Generates a large number of candidate fragments from the images in C . 3- These sub-images can vary in size and range from 1/50 to 1/7 of the object size.

48. Stage 2 – compare the distribution of each fragment in the C and NC. 1- For a given fragment Fi ,we measure Si . 2-To reach a fixed level of false alarms α in non-class images we determine a threshold θi for Fi by the criterion: p(Si > θi|NC) ≤ α Strength of Response – Maximal normalized correlation of a fragment i with each image I in C and NC

49. Stage 3 1) Order the fragments by their hit rate p(Si > θi|C) and select the K best ones .( k = size of set), and add this reliability value to each fragment. 2) Add new factor to each fragment: a figureground label . Grey level →←figure template ground label

50. Segmentation by Optimal Cover the main stage in the algorithm consists of covering an image with class-based fragments . cover