Challenges to image parsing researchers

1. Challenges to image parsing researchers Lana Lazebnik UNC Chapel Hill sky mountain building person car car sidewalk road

2. The past: “closed universe” datasetsTens of classes, hundreds of images, offline learning Figure from Shotton et al. (2009) He et al. (2004), Hoiem et al. (2005), Shotton et al. (2006, 2008, 2009), Verbeek and Triggs (2007), Rabinovich et al. (2007), Galleguillos et al. (2008), Gould et al. (2009), etc.

3. The future: “open universe” datasets Evolving images, annotations http://labelme.csail.mit.edu/

4. The future: “open universe” datasets Non-uniform class frequencies

5. Which “closed universe” techniques can survive in the “open universe” setting? • Combination of local cues? • Multiple segmentations/grouping hypotheses? • Context? • Graphical models (MRFs, CRFs, etc.)? • Offline learning and inference?

6. Learning from all of LabelMe50K images, 232 labels sky window building tree car door road road sun sky sea sky sky building ceiling car car building wall road sidewalk mountain floor Tighe & Lazebnik, work in progress

7. Learning from all of LabelMe50K images, 232 labels Per-class classification rates Tighe & Lazebnik, work in progress

8. Challenge: Parsing high-res images

9. Challenge: Dynamic image interpretation • Image parsing algorithms should become autonomous decision-making agents • Visual “detective task”: Where was this photo taken?

10. Challenge: Dynamic image interpretation • Image parsing algorithms should become autonomous decision-making agents Input

11. Summary • Challenges to image parsing researchers: • Learn to parse images from “open universe” evolving datasets • Try parsing gigapixel images! • Develop active, sequential image interpretation strategies