Bag-of-Words based Image Classification

1. Bag-of-Words based Image Classification Joost van de Weijer

2. What is in the image ? Is there a suit-case ? Is there a person ? Is there car ? image classification: answers the question what is in the image.

3. Inspiration The VOC Pascal challenge: a competition on image classification. Participants have to classify 20 classes in over 10.000 images.

4. Inspiration http://pascallin.ecs.soton.ac.uk/challenges/VOC/voc2010/results/index.html

5. The Event Data Set • 7 event classes: basketball, polo, rowing, castells, marathon, sailing, skiing. • each class has 50 images, devided in 30 training and 20 test images.

6. Project I title:Bag-of-Words basedImage Classification. goal: build an image classification system which can successfully classify sport images. competition: do so better than the other groups.

7. Why is this difficult ? Variations in pose. Variations in viewpoint and zoom.

8. Why is this difficult ? lighting changes. Inter-class variation.

9. Why is this difficult ? similar backgrounds- different classes. Back-ground variation. Maybe the background could help ?

10. Given a new image: • detect local regions from a set of image. • assign every region to its nearest visual word. • compute visual word-image histogram assign to visual word N from images to frequency histogram • Compute visual words: • detect local regions from a set of images. • describe every local region by a descriptor • texture • color • cluster all descriptors into visual words

11. Bag of Visual Words representation Bag-of-Words representation normalize patches Feature Detection No spatial relations.

12. Bag of Visual Words representation pi(w|Miro) pi(w|Dali)

13. 1. Feature detection 4. BOW Image 5. SVM/ distance measures 2. Extraction shape texture color image classification image retrieval 3.Learn vocabulary Shape Voc shape words The Framework Image Representation

14. 1. Feature detection 1.random 4. BOW 4. nearest neighbor Image 2. RGB 5. SVM/ distance measures 5. linear SVM 2. Extraction shape texture color image classification image retrieval 3.Learn vocabulary 3. random 50 % classification score Shape Voc shape words Existing Implementation: Image Representation

15. Existing Implementation: • properties of BOW implementation: • you can improve any of the subroutines and analyze the changes based on the classification results. • several team members can work on feature detection while others work on feature description. • the final classification results allow us to compare the results between the groups.

16. Project I: Bag Bag-of-Words based Image Classification goal : build an image classification system which can successfully classify sport images. • teaching objectives • you will learn: • to represent images robust to changes of cameras, object orientation, and illuminant color. • what photometric invariance theory is and how to apply it to a real-world problem. • understand and use the SIFT descriptor. • how to discretize image features (colors, shapes, and textures). • what the strong and weak points of BOW representations for images are. • how to evaluate retrieval and classification results.

17. Practical information: • Group Size: • The project has to be made in groups of 3 students. Each group should decide on the following roles: • responsible competition. • responsible presentation. • responsible report • If it is hard to work as a group you can partition the tasks: • feature detection • feature description • vocabulary construction • learning/evaluation All group-members should understand all steps in the final program ! Practical Information: All practical information can be found in the student guide (http://cat.cvc.uab.es/~joost/master.html )

18. Practical information: Important Dates: 22 jan- 19 Feb. : The project will last 1 month. 22 jan. : Start project. 29 Jan. : Extra assignment will be handed out. Submission of first results in AP. 5 Feb. : Discussion meeting + submission second results in AP. 11 Feb. : Publication of final test set. 12 Feb. : Discussion meeting with groups separately. 15 Feb. : Final submission of classification results in AP for all classes. 19 Feb. : Presentation of the project. 22 Feb. : Final submission date for report. Supervision: There will be project meetings on Tuesdays afternoon to discus progress. For any questions during the three weeks of the project email (joost@cvc.uab.es) or come to office O/119 in the CVC. Use “PROJECT I” as subject of your emails, which makes it easier to manage.

19. Practical information: • Notes • The final note will be based on: • participation (15%) • presentation (25%) • report (50%) • competition (10%) Bugs: For sure there will be several bugs in the code. If you find one, mail me, and I will notify the other groups. Thanks !

20. Practical information: Competition: Dates: 29 Jan. : Submission of first results in AP (before 15:00). 5 Feb. : Submission second results in AP (before 15:00) 19/22 Feb. : Your report/final presentation is based on the labeled test set ! labeled test set labeled train set

21. Practical information: Competition: Dates: 11 Feb. : Publication of final test set. 15 Feb. : Final submission results in AP for all calsses. labeled train set no labels for test set !

22. Practical information: • Final Report • The final report has to be submitted on 22th of February. The report should contain the following chapters. • Introduction ( max 1 page ) • Feature Detection (max 2 pages). • Feature Description (max 3 pages). • Visual Vocabulary and BOW representation (max 2 pages) • Classification (max 2 pages) • Object Detection (optional: max 2 pages) • Results (max 2 pages). • Conclusions (max 1 page)

23. What to do next ? • make groups of and assign : • responsible competition • (send an email to me today or tomorrow ) • install the programs and play with the code. • ( http://cat.cvc.uab.es/~joost/master.html ) • This week you should already start working on a feature detector.

24. What to do next ? Good Luck !