SlideServe
Télécharger Obtenir Pro
Menu
Accueil Pro
Préférences
ApparenceLight Aide et commentaires
© SlideServe · v2026

Scalable Multi-Label Annotation

kaori avatar
kaori
32 diapositives · 588 Views · Jul 11, 2014

Scalable Multi-Label Annotation. Jia Deng Olga Russakovsky Jonathan Krause, Michael Bernstein Alexander Berg Li Fei-Fei. Multi-label annotation. Labels. Data Item. Task: Crowdsource object labels for images. . Application: Benchmarking, training, modeling. Generalization :

Scalable Multi-Label Annotation - slide 1
01 / 32Scalable Multi-Label Annotation
1 / 17

Télécharger la présentation

Scalable Multi-Label Annotation
Une image / un lien ci-dessous est fourni (tel quel) pour télécharger la présentation

Politique de téléchargement : le contenu du site vous est fourni TEL QUEL pour votre information et votre usage personnel ; il ne peut être vendu, concédé sous licence ni partagé sur d’autres sites sans l’accord de son auteur.Le contenu vous est fourni TEL QUEL, pour votre information et votre usage personnel uniquement. Téléchargez la présentation en cliquant sur ce lien. Lors du téléchargement, si vous ne parvenez pas à télécharger une présentation, l’éditeur a peut-être supprimé le fichier de son serveur.Pendant le téléchargement, si vous ne parvenez pas à obtenir une présentation, le fichier a peut-être été supprimé par l’éditeur.

F I N

Plus de contenu associé

Transcription de la présentation

  1. Scalable Multi-Label Annotation JiaDeng Olga RussakovskyJonathan Krause, Michael Bernstein Alexander Berg Li Fei-Fei

  2. Multi-label annotation Labels Data Item Task: Crowdsource object labels for images. Application: Benchmarking, training, modeling • Generalization: • musical attributes of songs • actions in movies • sentiments in documents

  3. Large-Scale Visual Recognition Challenge ILSVRC 2010-2014 Current focus: 200 Category Detection (~100,000 fully labeled images)

  4. Naïve approach: ask for each object • cost: • estimation: • use the crowd-machine diagram • show UI. Is there a table? Question Machine Crowd Answer Yes

  5. Naïve approach: ask for each object • cost: • estimation: • use the crowd-machine diagram • show UI. Is there a table? Question Machine Crowd Answer Yes

  6. Naïve approach: ask for each object • cost: • estimation: • use the crowd-machine diagram • show UI. Is there a chair? Question Machine Crowd Answer Yes

  7. Naïve approach: ask for each object • cost: • estimation: • use the crowd-machine diagram • show UI. Is there a horse? Question Machine Crowd Answer No

  8. Naïve approach: ask for each object • cost: • estimation: • use the crowd-machine diagram • show UI. Is there a dog? Question Machine Crowd Answer No

  9. Naïve approach: ask for each object • cost: • estimation: • use the crowd-machine diagram • show UI. Is there a cat? Question Machine Crowd Answer No

  10. Naïve approach: ask for each object • cost: • estimation: • use the crowd-machine diagram • show UI. Is there a bird? Question Machine Crowd Answer No

  11. Naïve approach: ask for each object Cost: O(NK) for N images and K objects

  12. Animal Mammal Furniture Hierarchy

  13. Animal Hierarchy Mammal Furniture Correlation Sparsity

  14. Animal Better approach: exploit label structure Mammal Furniture Question Machine Crowd Answer

  15. Animal Better approach: exploit label structure Mammal Furniture Question Is there an animal? Machine Crowd No Answer

  16. Animal Better approach: exploit label structure Mammal Furniture Question Is there an animal? Machine Crowd No Answer

  17. Better approach: exploit label structure Mammal Animal Furniture Question Is there furniture? Machine Crowd Yes Answer

  18. Better approach: exploit label structure Mammal Animal Furniture Question Is there a table? Machine Crowd Yes Answer

  19. Better approach: exploit label structure Mammal Animal Furniture Question Is there a chair? Machine Crowd Yes Answer

  20. Better approach: exploit label structure Mammal Animal Furniture Question Is there a chair? Machine Crowd Yes Answer

  21. Selecting the Right Question Goal: Get as much utility (new labels) as possible, for as little cost (worker time) as possible, given a desired level of accuracy

  22. Accuracy constraint • User-specified accuracy threshold, e.g., 95% • Majority voting assuming uniform worker quality [GAL: Sheng, Provost, Ipeirotis KDD ‘08] • Might require only one worker, might require several based on the task

  23. Cost: worker time (time = money) expectedhuman time to get an answer with 95% accuracy

  24. Utility: expected # of new labels Yes No Is there a table? utility = 1

  25. Utility: expected # of new labels Yes No Is there a table? utility = 1 Pr(Y) = 0.5 Pr(N) = 0.5 Is there an animal? utility = 0.5 * 0 + 0.5 * 4 = 2

  26. Selecting the Right Question Pick the question with the most labels per second

  27. Results • Dataset: 20K images from ImageNet Challenge 2013. • Labels: 200 basic categories (dog, cat, table…), 64 internal nodes in hierarchy

  28. Results • Dataset: 20K images from ImageNet Challenge 2013. • Labels: 200 basic categories (dog, cat, table…), 64 internal nodes in hierarchy • Setup: • 50-50 training test split • Estimate parameters on training, simulate on test • Future work: online estimation

  29. Results: accuracy Annotating 10K images with 200 objects

  30. Results: cost Annotating 10K images with 200 objects

  31. Results: cost Annotating 10K images with 200 objects 6 times more labels per second

  32. Conclusions Animal Mammal Furniture Speeds up crowdsourced multi-label annotation by exploiting the structure and distribution of labels. Could be a bargain for you! Hierarchy Sparsity Correlation