Segmentation Propagation in ImageNet

1. Segmentation Propagationin ImageNet ECCV 2012, Florence, Italy October 11th, 2012 Daniel Kuettel, Matthieu Guillaumin, Vittorio Ferrari ETH Zurich University of Edinburgh

2. Goal: segment every image in ImageNet • Segmentations can support shape models, object recognition,pose estimation… • 10k+ classes, 10M+ images • 0 segmentations transportation aircraft wheeled vehicle car [Deng CVPR09, ShottonICCV05, Tu IJCV05, Jiang ICCV09]

3. Exploit all availableinformation External GT segmentations Bounding boxes Object class Semantic hierarchy (WordNet) transportation aircraft wheeledvehicle car cat PASCAL VOC10 aircraft • Segmented images help segment images with similar objects • Bounding boxes constrain segmentations • Semantically related object classes can share appearance

4. Segmentation propagation in a hierarchy VOC10 • segmented images (source) help segmenting new ones (target):segmentation transfer • Proceed recursively: propagation transportation wheeled vehicle aircraft car

5. Segmentation propagation in a hierarchy Source VOC10 • Start from the easiest images transportation wheeled vehicle aircraft Target car

6. Segmentation propagation in a hierarchy VOC10 transportation Source wheeled vehicle aircraft Target Segmentationtransfer car

7. Segmentation propagation in a hierarchy VOC10 transportation wheeled vehicle aircraft Joint segmentation car

8. Segmentation propagation in a hierarchy VOC10 Source transportation Segmentationtransfer wheeled vehicle aircraft Source Target car

9. Segmentation propagation in a hierarchy VOC10 transportation wheeled vehicle aircraft Joint segmentation car

10. Segmentation propagation in a hierarchy VOC10 transportation wheeled vehicle aircraft Semantic relation car

11. Segmentation propagation in a hierarchy Source Target 1 VOC10 transportation wheeled vehicle aircraft Source car Target 2 Target 3

12. Segmentation propagation in a hierarchy VOC10 transportation wheeled vehicle aircraft car

13. Segmentation propagation in a hierarchy VOC10 transportation wheeled vehicle aircraft car

14. Segmentation propagation in a hierarchy VOC10 transportation wheeled vehicle aircraft car

15. Segmentation transfer Source Target image • Related to earlier annotation transfer works[Russel NIPS07, Liu CVPR09, Guillaumin ICCV09,RosenfeldICCV11, KuettelCVPR12, Rubinstein ECCV12]

16. Segmentation transfer Source Target image • Sample windows on objects Objectness sampling [AlexeCVPR10]

17. Segmentation transfer Source Target image • Sample windows on objects • Find visually similar windows HOG + compact binary code for efficient retrieval [Dalal CVPR05, TorralbaCVPR08, Gong CVPR11]

18. Segmentation transfer Target image • Sample windows on objects • Find visually similar windows • Aggregate their segmentations [KuettelCVPR12]

19. Segmentation transfer Target image • Sample windows on objects • Find visually similar windows • Aggregate their segmentations [KuettelCVPR12]

20. Segmentation transfer Source Target image • Sample windows on objects • Find visually similar windows • Aggregate their segmentations • Initialize and run GrabCut [RotherSIGGRAPH04]

21. Segmentation transfer Source Target image • Window-level: compositionality • But: increases the number of descriptors in the source-> 1M images X 100 windows = 100M descriptors-> Compact binary codes [Gong CVPR11] 200x faster, 500x smaller [Gong CVPR11, KuettelCVPR12, Liu CVPR09]

22. Segmentation transfer Source Target image • Code available online: http://groups.inf.ed.ac.uk/calvin/software.html

23. Joint segmentation with shared appearance Target = Dog Segment jointly • Related to co-segmentation[Winn ICCV05, RotherECCV06, JoulinCVPR10, Vicente CVPR11, Batra IJCV11] • Related to knowledge transfer[Salakhutdinov CVPR11, Guillaumin CVPR12, Fei-Fei ICCV03] Related class = Canine Transfer appearance

24. Joint segmentation with shared appearance

25. Joint segmentation with shared appearance Transferred mask from source S to image i

26. Joint segmentation with shared appearance Appearance model for image i

27. Joint segmentation with shared appearance Appearance model for class C

28. Joint segmentation with shared appearance Appearance model for related classes

29. Joint segmentation with shared appearance Label smoothness

30. Joint segmentation with shared appearance Iterate minimization and appearance model re-estimation, as in Grabcut [RotherSIGGRAPH04, Batra IJCV11]

31. Joint segmentation with shared appearance

32. Joint segmentation with shared appearance • Learning α: fixed appearance models --> independent images • Use structured SVM: • Loss Δi=sum over pixels: exact and efficient computation of most violated constraint with GraphCut • Learn specific α for each stage of the propagation [TsochantaridisJMLR05, SzummerECCV08]

33. Experiments on iCoseg • 38 classes, 643 images. No propagation, no related classes • Within-class appearance sharing helps • Segmentation transfer helps • Outperforms Joulin10, Vicente11and much faster(using unrelated segmented images from PASCAL VOC10) [RotherSIGGRAPH04, Joulin CVPR10, Vicente CVPR11, Batra IJCV11, Kuettel CVPR12]

34. Experiments on ImageNet Kangaroo Lemur Killer whale

35. Experiments on ImageNet Sail boat Megaphone Monitor

36. Experiments on ImageNet • Overall experiment: 0.5M images, 577 classes • Full propagation scheme • 10 images X 446 classes annotated with AMT • 77.1% accuracy vs 71.0% for GrabCut • Performance degrades gracefully over stages

37. Conclusion • Segmentation Propagation: • An efficient scheme to recursively segment images in ImageNet • Successfully combines ideas from segmentation transfer,co-segmentation, binary codes for image retrieval, structured output learning • Excellent results on: • iCoseg • ImageNet