Large-Scale Image Parsing

1. sky building car Large-Scale Image Parsing road Joseph Tighe and Svetlana Lazebnik University of North Carolina at Chapel Hill

2. Small-scale image parsingTens of classes, hundreds of images Figure from Shotton et al. (2009) He et al. (2004), Hoiem et al. (2005), Shotton et al. (2006, 2008, 2009), Verbeek and Triggs (2007), Rabinovich et al. (2007), Galleguillos et al. (2008), Gould et al. (2009), etc.

3. Large-scale image parsingHundreds of classes, tens of thousands of images Non-uniform class frequencies

4. Large-scale image parsingHundreds of classes, tens of thousands of images Non-uniform class frequencies Evolving training set http://labelme.csail.mit.edu/

5. Challenges • What’s considered important for small-scale image parsing? • Combination of local cues • Multiple segmentations, multiple scales • Context • How much of this is feasible for large-scale, dynamic datasets?

6. Our first attempt: A nonparametric approach • Lazy learning: do (almost) nothing up front • To parse (label) an image we will: • Find a set of similar images • Transfer labels from the similar images by matching pieces of the image (superpixels)

7. Finding Similar Images

8. Ocean Forest Open Field Mountain Which image is most similar? Highway Inner City Then assign the label from the most similar image Street Tall Building What is depicted in this image?

9. Pixels are a bad measure of similarity Most similar according to pixel distance Most similar according to “Bag of Words”

10. Origin of the Bag of Words model • Orderless document representation: • frequencies of words from a dictionary Salton & McGill (1983) US Presidential Speeches Tag Cloudhttp://chir.ag/phernalia/preztags/

11. What are words for an image?

16. Wing Tail Propeller Building Wheel

17. Wing Building Wheel Propeller Jet Engine Tail

18. Wing Building Wheel Propeller Jet Engine Tail

19. Wing Building Wheel Propeller Jet Engine Tail

20. But where do the words come from?

22. Then where does the dictionary come from?

23. Example Dictionary Source: B. Leibe

24. … … … … Another dictionary Source: B. Leibe

25. Fei-Fei et al. 2005

26. Outline of the Bag of Words method • Divide the image into patches • Assign a “word” for each patch • Count the number of occurrences of each “word” in the image

27. Does this work for our problem? 65,536 Pixels 256 Dimensions

28. Which look the most similar?

29. Which look the most similar? sky sky building tree building tree car car road road sky sky sky building building building sand car road sky sky sky sky building building mountain car car tree car road road road

30. Step 1: Scene-level matching Gist (Oliva & Torralba, 2001) Spatial Pyramid(Lazebnik et al., 2006) Color Histogram Retrieval set: Source of possible labels Source of region-level matches

31. Step 2: Region-level matching

32. Step 2: Region-level matching Superpixels(Felzenszwalb & Huttenlocher, 2004)

33. Step 2: Region-level matching Road Tree Pixel Area (size) Sky Building Snow

34. Step 2: Region-level matching Road Absolute mask(location) Sidewalk

35. Step 2: Region-level matching Road Texture Snow Sky Sidewalk

36. Step 2: Region-level matching Road Sidewalk Color histogram Building

37. Step 2: Region-level matching Superpixel features Superpixels(Felzenszwalb & Huttenlocher, 2004)

38. Region-level likelihoods • Nonparametric estimate of class-conditional densities for each class c and feature type k: • Per-feature likelihoods combined via Naïve Bayes: Features of class c within some radius of ri Total features of class c in the dataset kth feature type of ith region

39. Region-level likelihoods Building Car Crosswalk Road Window Sky

40. Step 3: Global image labeling • How do we resolve issues like this? Maximum likelihood labeling Original image sky sky road tree sea sea road sand sand

41. Step 3: Global image labeling • Compute a global image labeling by optimizing a Markov random field (MRF) energy function: Likelihood score for region ri and label ci Smoothing penalty Co-occurrence penalty Neighboring regions Regions Vector of region labels

42. Step 3: Global image labeling • Compute a global image labeling by optimizing a Markov random field (MRF) energy function: Likelihood score for region ri and label ci Smoothing penalty Co-occurrence penalty Neighboring regions Regions Vector of region labels Maximum likelihood labeling Edge penalties Final labeling Final edge penalties sky sky building building window car car road road

43. Step 3: Global image labeling • Compute a global image labeling by optimizing a Markov random field (MRF) energy function: Likelihood score for region ri and label ci Smoothing penalty Co-occurrence penalty Neighboring regions Regions Vector of region labels Maximum likelihood labeling Edge penalties Original image MRF labeling sky sky road tree sea sea road sand sand

44. Joint geometric/semantic labeling • Semantic labels: road, grass, building, car, etc. • Geometric labels: sky, vertical, horizontal • Gould et al. (ICCV 2009) Original image Semantic labeling Geometric labeling sky sky tree vertical car horizontal road

45. Joint geometric/semantic labeling • Objective function for joint labeling: Geometric labels Semantic labels Cost of geometric labeling Cost of semantic labeling Geometric/semantic consistency penalty Original image Semantic labeling Geometric labeling sky sky tree vertical car horizontal road

46. Example of joint labeling

47. Understanding scenes on many levels To appear at ICCV 2011

48. Datasets

49. Datasets

50. Overall performance *SIFT Flow: 74.75