Hierarchical Novelty Detection for Visual Object Recognition

1. Hierarchical NoveltyDetection for Visual ObjectRecognition Kibok Lee*, Kimin Lee†, KyleMin*, Yuting Zhang*, Jinwoo Shin†, HonglakLee*‡ University of Michigan*, KAIST†, GoogleBrain‡

2. Conventionalnoveltydetectionframeworkdoesnotprovidemore • informationthan“novelty”ofanobject. • Supposewehavetrainingdatalike… Why Hierarchical NoveltyDetection? Persiancat Siamesecat Pomeranian Welshcorgi

3. Then,supposewehavetestimageslike… • Testimage: Why Hierarchical NoveltyDetection? Persiancat Siamesecat Pomeranian Welshcorgi

4. Testimage: Why Hierarchical NoveltyDetection? animal cat Persiancat Siamesecat dog Pomeranian Welshcorgi

5. Ourhierarchicalnoveltydetectionframeworkaimstofindthemost specific class label of any data on the hierarchical taxonomy built with knownlabels. Why Hierarchical NoveltyDetection? Testimage:

6. Thisframeworkcanbepotentiallyusefulforautomaticallyor interactively organizing a customizedtaxonomy • Company’s productcatalog • Wildlifemonitoring • Personal photolibrary • bysuggestingclosestcategoriesforanimagefromnovel categories. • new consumerproducts • unregistered animalspecies • untagged scenes orplaces Why Hierarchical NoveltyDetection?

7. Ourtaxonomyhasthreetypesofclasses. • Known leaf classes are seen duringtraining • Superclassesareancestorsofleafclasses,alsoknown • Novel classes are unseen duringtraining • Theirexpectedpredictionistheclosestsuperclassinthetaxonomyinourtask HierarchicalTaxonomy

8. HierarchicalTaxonomy

9. HierarchicalTaxonomy

10. HierarchicalTaxonomy

11. HierarchicalTaxonomy

12. HierarchicalTaxonomy

13. HierarchicalTaxonomy

14. Multi-stageclassification • Untilarrivingataknownleafclass Approach - Top-down (TD)Method .2 .8 .7 .1 .2

15. Multi-stageclassification • Untilarrivingataknownleafclass • Or classification is unconfident(novel) Approach - Top-down (TD)Method .5 .5

16. Classificationrule: • Classification is confidentif Approach - Top-down (TD)Method

17. Trainingobjective: Approach - Top-down (TD)Method

18. Representallprobabilitiesofknownleafandnovelclassesina singlevector • Add virtual novelclasses Approach - FlattenMethod

19. Representallprobabilitiesofknownleafandnovelclassesina singlevector • Add virtual novelclasses • Andthenflattenthestructure Approach - FlattenMethod

20. Representallprobabilitiesofknownleafandnovelclassesina singlevector • Add virtual novelclasses • Andthenflattenthestructure Approach - FlattenMethod

21. Classificationrule: • We proposetwostrategiestotrainthismodel. Approach - FlattenMethod

22. Datarelabeling • Fill novel classes by hierarchicalrelabeling Approach - FlattenMethod animal cat dog Persiancat Siamesecat Pomeranian Welshcorgi

23. Datarelabeling • Fill novel classes by hierarchicalrelabeling Approach - FlattenMethod animal cat dog Persiancat Siamesecat Pomeranian Welshcorgi

24. Datarelabeling • Fill novel classes by hierarchicalrelabeling Approach - FlattenMethod animal cat dog Persiancat Siamesecat Pomeranian Welshcorgi

25. Datarelabeling • Fill novel classes by hierarchicalrelabeling • Relabeling ratecan be chosen by validation Approach - FlattenMethod animal cat dog Persiancat Siamesecat Pomeranian Welshcorgi

26. Datarelabeling • Trainingobjective: Approach - FlattenMethod animal cat dog Persiancat Siamesecat Pomeranian Welshcorgi

27. Leave-one-out (LOO)strategy • Generate deficienttaxonomies • Andthentrainthemodelwiththem Approach - FlattenMethod animal cat dog Persiancat Siamesecat Pomeranian Welshcorgi

28. Leave-one-out (LOO)strategy • e.g.,when Approach - FlattenMethod animal cat dog Persiancat Siamesecat Pomeranian Welshcorgi

29. Leave-one-out (LOO)strategy • e.g.,when Approach - FlattenMethod animal novelanimal dog Pomeranian Welshcorgi

30. Leave-one-out (LOO)strategy • Trainingobjective: Approach - FlattenMethod animal novelanimal dog Pomeranian Welshcorgi

31. ComputeandenumeratetheoutputofTD • AndthenfeedittoLOO • Combinedmethodutilizestheircomplementarybenefits. • Top-down method leverages hierarchical structureinformation. • Butitsuffersfromerroraggregationoverhierarchy. • Flattenmethodavoidserroraggregationoverhierarchy. • Butitdoesnotleveragehierarchicalstructureinformation. Approach - Combined Method(TD+LOO)

32. Quantitativeresults Experiments - Hierarchical NoveltyDetection

33. Comparedalgorithms • Baseline: DARTS (Deng et al.,2012) • Ours: Relabel, LOO,TD+LOO Experiments - Hierarchical NoveltyDetection J.Deng,J.Krause,A.C.Berg,andL.Fei-Fei.“Hedgingyourbets:Optimizingaccuracy-specificitytrade-offsinlargescale visualrecognition.”InCVPR,2012

34. Datasets • ImageNet: 1k known, 16k novelclasses • AwA2: 40 known, 10 novelclasses • CUB: 150 known, 50 novelclasses Experiments - Hierarchical NoveltyDetection

35. Metrics • Novel class accuracy @ knownclass accuracy = 50% • Byaddinganappropriatescorebiastoallnovelclasses • Area under known-novel class accuracycurve • By varying the novel class scorebias Experiments - Hierarchical NoveltyDetection

36. Metrics • Novel class accuracy @ knownclass accuracy = 50% • Byaddinganappropriatescorebiastoallnovelclasses • Area under known-novel class accuracycurve • By varying the novel class scorebias • (a)ImageNet (b)AwA2 Experiments - Hierarchical NoveltyDetection (c)CUB

37. Metrics • Novel class accuracy @ knownclass accuracy = 50% • Byaddinganappropriatescorebiastoallnovelclasses • Area under known-novel class accuracycurve • By varying the novel class scorebias • (a)ImageNet (b)AwA2 Experiments - Hierarchical NoveltyDetection (c)CUB

38. Metrics • Novel class accuracy @ knownclass accuracy = 50% • Byaddinganappropriatescorebiastoallnovelclasses • Area under known-novel class accuracycurve • By varying the novel class scorebias • (a)ImageNet (b)AwA2 Experiments - Hierarchical NoveltyDetection (c)CUB

39. Qualitativeresults Experiments - Hierarchical NoveltyDetection Novel class: Americanfoxhound

40. Qualitativeresults Experiments - Hierarchical NoveltyDetection

41. Qualitativeresults Experiments - Hierarchical NoveltyDetection

42. Qualitativeresults Experiments - Hierarchical NoveltyDetection

43. Qualitativeresults Experiments - Hierarchical NoveltyDetection

44. Quantitativeresults Experiments - Generalized Zero-ShotLearning

45. Semanticembeddings • Attributes (numeric attributevalues) • Wordvector(similarityamongwordsinrealcoordinatespace) • Hierarchicalembedding Experiments - Generalized Zero-ShotLearning

46. Compared hierarchicalembeddings • Baseline: Path (Akata etal., 2015) • Distance between classes onhierarchy • Ours: Top-down(TD) • Expectedoutputofourtop-downmodel Experiments - Generalized Zero-ShotLearning Z.Akata,S.Reed,D.Walter, H.Lee,andB.Schiele.“Evaluationofoutputembeddingsforfine-grainedimageclassification.”InCVPR,2015.

47. Datasets • AwA1,2: 40 known, 10 novelclasses • CUB: 150 known, 50 novelclasses Experiments - Generalized Zero-ShotLearning

48. Metrics • Unseen class accuracy(ZSL) • Area under seen-unseen curve(GZSL) • By varying the unseen class scorebias Experiments - Generalized Zero-ShotLearning

49. Metrics • Unseen class accuracy(ZSL) • Area under seen-unseen curve(GZSL) • By varying the unseen class scorebias • AwA1 (b)AwA2 Experiments - Generalized Zero-ShotLearning (c)CUB

50. Metrics • Unseen class accuracy(ZSL) • Area under seen-unseen curve(GZSL) • By varying the unseen class scorebias • AwA1 (b)AwA2 Experiments - Generalized Zero-ShotLearning (c)CUB