SVM Active Learning For Image Retrieval

1. SVM Active Learning For Image Retrieval Morris LeBlanc

2. Outline • Why Image Retrieval is Hard? • Problems with Image Retrieval • Support Vector Machines • Active Learning • Image Processing • Texture and Color • Relevance Feedback

3. Why is Image Retrieval Hard? • What is the topic of this image? • What are right keywords to index this image • What words would you use to retrieve this image? • The Semantic Gap

4. Problems with Image Retrieval • A picture is worth a thousand words • The meaning of an image is highly individual and subjective

5. Support Vector Machines(SVMs) • Is a set of related learning methods used for classification and regression • Views data in two sets of vectors in a n-dimensional space • With this we are able to label “relevant” and “non-relevant” images • Based on distance from a labeled instance

6. Updating SVM • SVM training process proceeds as follows: • Choose some working subset of the query images • Construct classifier – i.e. create a new surface: • Optimize the weights associated with the working subset of images (feature vectors) • Update optimality conditions for images (vectors) not in working subset • Broadcast working subset images (vectors) and weights • Update optimality conditions for all images in query (Map) • Reduce to find greatest violating image (vector) not contained in working subset (Reduce)

7. Updating SVMs Cont’d • Update working subset to include greatest violating image (vector) • Iterate until all images (vectors) satisfy optimality conditions • Repeat steps 2 through 4 until correct images are returned

8. Example This image shows the multiple current version space chosen by the user (wi) and all instances found later. The closet one is what will be shown to the user.

9. Active Learning • Here, one allows the learner the flexibility to choose the data points that it feels are most relevant for learning a particular task • An analogy is that a standard passive learner is a student that sits and listens to a teacher while an active learner is a student that asks the teacher questions, listens to the answers and asks further questions based upon the teacher's response

10. Image Processing for Retrieval • Representing the Images • Segmentation • Low Level Features • Color • Texture

11. Image Features • Information about color or texture or shape which are extracted from an image are known as image features • Also a low-level features • Red, sandy • As opposed to high level features or concepts • Beaches, mountains, happy, serene, George Bush

12. Image Segmentaton • Do we consider the whole image or just part ? • Whole image - global features • Parts of image - local features

13. Local Features • Segmentimages into parts • Two sorts: • Tile Based • Region based

14. Regioning and Tiling Schemes Tiles Regions

15. Tiling • Break image down into simple geometric shapes • Similar Problems to Global • Plus dangers of breaking up significant objects • Computational Simple • Some Schemes seem to work well in practice

16. Regioning • Break image down into visually coherent areas • Can identify meaningful areas and objects • Computationally intensive • Unreliable

17. Color • Produce a color signature for region/whole image • Typically done using color correllograms or color histograms

18. Color Histograms • Identify a number of buckets in which to sort the available colours (e.g. red green and blue, or up to ten or so colours) • Allocate each pixel in an image to a bucket and count the number of pixels in each bucket. • Use the figure produced (bucket id plus count, normalised for image size and resolution) as the index key (signature) for each image

19. Global Histograms

20. Texture • Produce a mathematical characterization of a repeating pattern in the image • Smooth • Sandy • Grainy • Stripey

21. Texture • Reduces an area/region to a (small - 15 ?) set of numbers which can be used a signature for that region • Proven to work well in practice • Hard for people to understand

22. Relevance Feedback • Well established technique in text retrieval • Experimental results have always shown it to work well in practice • Unfortunately experience with search engines has show it is difficult to get real searchers to adopt it - too much interaction

23. Basic Idea • User performs an initial query • Selects some relevant results • System then extracts terms from these to augment the initial query • Requeries

24. Relevance Feedback Algorithm • Identify the N top-ranked images • Identify all terms from the N top-ranked images • Select the feedback terms • Merge the feedback terms with the original query • Identify the top-ranked images for the modified queries through relevance ranking

25. Rocchio Vector Space RelevanceFeedback • Q’ = aQ + b sum(R) - c sum(S) • Q: original query vector • R: set of relevant document vectors • S: set of non-relevant image vectors • a, b, c: constants (Rocchio weights) • Q’: new query vector

26. References • “SVM Active Learning For Image Retrieval” Simon Tong, Stanford University and Edward Chang, UCSB • John Tait, University of Sunderland, UK tait.ppt • http://robotics.stanford.edu/~stong/research.html -Simon Tong’s website

27. Questions and Comments