Segmentation with Scene and Sub-Scene Categories

1. Segmentation with Scene and Sub-Scene Categories Joseph Djugash Input Image Scene/Sub-Scene Classification Segmentation

2. Problem Statement • Goal: • Accurate segmentation of salient objects/regions in any image. • Problems/Issues: • What is a salient object(s)? • How do we identify the presence of these objects? • What metric/feature/cue do we use? • Is this consistent over all images?

3. Outline • Normalized Cut • Learning segmentation • Methods • The Naïve Approach • Affinity Matching • Minimizing Segmentation Error • Discussion of Results

4. Bottom-Up Segmentation • Normalized Cuts, Mean Shift, etc. • Drawbacks • Parameters • Number of Clusters/Segments • Cluster Size (Sigma values) • Performance drops when dealing with a wide variety of images 1. Jianbo Shi; Malik, J. Normalized cuts and image segmentation. PAMI (2000)2. D. Comaniciu, P. Meer. Mean Shift: A Robust Approach toward Feature Space Analysis. PAMI (2002)

5. Class Specific Segmentation • Image patches from training images are fit to the input image • Patch consistency checks correct inaccurate matching and segmentation • Only known object classes can be segmented correctly • Large database required to encompass all possible objects 3. Eran Borenstein, Shimon Ullman. Class-Specific, Top-Down Segmentation. ECCV (2002)

6. Outline • Normalized Cut • Learning segmentation • Methods • The Naïve Approach • Affinity Matching • Minimizing Segmentation Error • Discussion of Results

7. The Naïve Approach Scene Categories Training Phase Parameter Learning • Learnt Parameters: • Cluster Range {min & max # of clusters} • Sigma Values {controls size of cluster/segment} • The parameters are learnt using a supervised reinforcement learning algorithm (Feedback +/o/–) Testing Phase Parameter Look-Up using Bag-Of-Words Scene Categories Image Segmentation

8. Results: The Naïve Approach

9. Outline • Normalized Cut • Learning segmentation • Methods • The Naïve Approach • Affinity Matching • Minimizing Segmentation Error • Discussion of Results

10. Affinity Matching Training Phase Parameters Learnt using reinforcement learning Parameter Learning Testing Phase Nearest Neighbor Matching performed on a set of stored Affinity matrices Query images are classified and learnt parameters are used to Segments Image Nearest Neighbor Classification& Parameter Look-Up Image Segmentation

11. Results: Affinity Matching

12. Outline • Normalized Cut • Learning segmentation • Methods • The Naïve Approach • Affinity Matching • Minimizing Segmentation Error • Discussion of Results

13. Segmentation Error Dissimilarity within ObjectsJN = je() pxInW(pxIn, pxIn) LabelMe Data Minimizing Segmentation Error Training Phase Gist Features Kmeans Clustering J=i(JN + JS + JO) Object Pixels i єImages in Cluster j єLabeled Objects Parameter Learning Minimizing Segmentation Cost to Human Labeled Data Similarity to Non-ObjectsJN = j W(pxIn, pxOut) Cluster CostJS = je(γ*|#Seg – #Obj|) Testing Phase Image Segmentation 4. A. Torralba,  K. P. Murphy, W. T. Freeman and M. A. Rubin. Context-based vision system for place and object recognition. AIM (2003)

14. Results: Minimizing Segmentation Error

15. Results: Minimizing Segmentation Error

16. Discussion • The Naïve Approach • Uses intuitive categories that humans can identify with • Human feedback can be unreliable and inconsistent • Broad category labels lead to segmentation inaccuracy • Affinity Matching • Affinity matrix provides a good feature space for segmentation • Closeness in the affinity matrix might not necessarily imply similarity in segmentation parameters • Minimizing Segmentation Error • A qualitative measure for evaluating the validity of a segmentation • Human segmentation should try to realize image cues • NCut segmentation prefers edge contours in the image

17. Questions?

18. Outline – Detailed • The Naïve Approach • Reinforcement Learning • Bag-of-words scene classification • Affinity Matching • NN classification on Affinity Matrix • Learning similar image structure • Human Segmentation & Minimizing Cost • LabelMe Data • Gist Features

19. Heavy Clutter Scenery Outdoor Objects with Sky Outdoor Objects Buildings Close-Up Objects Partically Cluttered

20. Learning Scene Categories – Bag-of-Words • Scene Categories learnt from clustering on image codewords • Feature Detection and Representation can be done using various Image Cues • Codewords stored in the database need to cover a wide spectrum 3. Fei-Fei and P. Perona. A Bayesian hierarchical model for learning natural scene categories. CVPR (2005)