1. Outline • Overview • Integrating Vision Models • CCM: Cascaded Classification Models • Learning Spatial Context • TAS: Things and Stuff • Descriptive Querying of Images • LOOPS: Localizing Object Outlines using Probabilistic Shape • Future Directions [Heitz et al. NIPS 2008b]

2. Image Queries on Objects • Categorization • Localization • Descriptive • What color is his tail? • Where is his head? • Is he… standing? sitting? bent over? • What is he doing?

3. Related Work Boosted Detectors [ Torralba et al., PAMI 2007 ] [ Opelt, ECCV 2006 ] [ Fei-Fei and Perona, CVPR 2005 ] Coarse Refined Localization “Parts” Models [ Fergus et al., CVPR 2003 ] [ Leibe et al, ECCV 2004 ] [ Bar-Hillel et al, CVPR 2005 ] [ Winn & Shotton, CVPR 2006 ] Localization Models [ Kumar et al., CVPR 2005 ] [ Cootes et al., CVIU 1995 ] [ Borenstein et al., CVPR 2004 ] Fine OUR WORK

4. Shape Representation: Landmarks Set of “keypoint” landmarks Shape defined by connecting piecewiselinear contour Internal landmarks are allowed (but not shown here)

5. Training Data • Images + Outlines

6. State-of-the-art Alternatives • kAS Detector: Edge-based object detector • Pro: No outline required Great at detection • Con: No single outline • OBJ CUT: Object-based segmentation • Pro: Produces outlines • Con: Appearance modelbased on internal texture [ Ferrari et al., CVPR 2007 ] [ Kumar et al., CVPR 2005 ] [ Prasad et al., CVPR 2006 ]

7. LOOPS Pipeline Images + Outlines ConsistentOutlines LOOPS Model Localized Test Outlines Up Down +1 std UP -1 std +1 std DOWN -1 std Localizing Object Outlines using Probabilistic Shape Descriptive Classification register model to images shape basedclassification learn shape& landmark detectors producecorresponded training data

8. Corresponded Outlines Images + Outlines ConsistentOutlines Localizing Object Outlines using Probabilistic Shape • Based on existing work in medical imaging • Intuition: Arc length and curvature should remain consistent [ Hill & Taylor, BMVC 1996 ] producecorresponded training data

9. Learning Shape & Detectors ConsistentOutlines LOOPS Model +1 std -1 std +1 std -1 std Localizing Object Outlines using Probabilistic Shape learn shape& landmark detectors

10. Multivariate Gaussian over landmark locations Shape Model Neck Legs

11. Landmark Detectors • Build on state-of-the-art discriminative methods for detecting “parts” or “objects” Build a detector for each landmark

12. Registration LOOPS Model Localized Test Outlines +1 std -1 std +1 std -1 std Localizing Object Outlines using Probabilistic Shape register model to images

13. “Registering” the Model to an Image ? ? Task: Assign each landmark l L to a pixel plP L – Assignment of Landmarks to Pixels L* = argmax Score(L | I) = argmax ShapeScore(L) + ImageScore(L | I)

14. The LOOPS MRF pairwise image score shape score landmark detectors Registering = MAP Inference over L

15. Outlining Full LOOPS Image Detectors Only

16. Results Rhino Giraffe Llama

17. kAS Detector OBJ CUT LOOPS

18. kAS Detector OBJ CUT LOOPS

19. LOOPS Pipeline Images + Outlines ConsistentOutlines LOOPS Model Localized Test Outlines Up Down +1 std UP -1 std +1 std DOWN -1 std Localizing Object Outlines using Probabilistic Shape Descriptive Classification register model to images shape basedclassification learn shape& landmark detectors producecorresponded training data

20. Descriptive Classification Localized Test Outlines Up Down UP DOWN Localizing Object Outlines using Probabilistic Shape Descriptive Classification shape basedclassification

21. Descriptive Queries Goal: Classify based on shape characteristics Is the giraffe Or 1 0.8 0.6 0.4 1 2 3 4 5 6 7 8 9 10 # Training Instances (per class) “True” shape Close this gap Boosting Accuracy RANDOM

22. Mammals