Learning in Large Scale Image Retrieval Systems

1. Learning in Large Scale Image Retrieval Systems Under the guidance of: Dr. C. V. Jawahar & Dr. Vikram Pudi by Pradhee Tandon Roll No. 200607020

2. Image Retrieval Explosive growth in images Easy access to most of these on the web Contemporary systems used tags The best commercial systems are still tag based Inadequate and unreliable Manual tagging is infeasible Content based retrieval is the best option

3. Content Based Image Retrieval Query Feature Extraction Comparison Module Results Image Feature Database Query Results

4. Content Based Image Retrieval Query Feature Extraction Comparison Module Results Image Feature Database Feature Index

5. Content Based Image Retrieval Query Feature Extraction Comparison Module Results Rf Relevance Learning Feature Index Memory & Logs

6. Content Based Image Retrieval Query Feature Extraction Comparison Module Results Rf Relevance Learning Learning Memory Feature Index

7. Features Color Histograms Texture Filters Shape Context SIFT GLOH Spatial indexing methods Kd – trees R-tree Distance Metrics Euclidean Mahalanobis KL Divergence Relevance Feedback Short term learning Long term learning Content Free Retrieval Active Learning Diversity Retrieval Scope of work

8. Requirements • Efficient Image Retrieval • Learning relevant features in images • Learning image-image relationships • Diversity in retrieval for improved learning

9. Requirements • Efficient Image Retrieval • Learning relevant features in images • Learning image-image relationships • Diversity in retrieval for improved learning

10. FISH – The System

11. Implementation of FISH • Image Representation in FISH • MPEG-7 Colour Structure Descriptor • Maximum Response Filters for Textures Developed on the LAMP stack, using C/C++, Perl, PHP, HTML, MySQL and Apache TPIE toolkit from Duke University for B+ tree implementation

12. Indexing Scheme Interactive response over large databases (less than a second) Efficient scalable index (dynamic with data) Similarity indexing scheme (r-tree, kd-tree, ss-tree) Support for changing similarity metrics (metric changes with learning) B+ tree based index Nataraj et. al, MMM 2007, Efficient Search with Changing Similarity Measures on Large Multimedia Datasets

13. The Retrieval Algorithm Query Feature Extraction Comparison Module Results Rf Relevance Learning Feature Index Learning Memory Retrieval in FISH

14. Retrieval Performance Retrieval times with increasing DB sizein (secs) & #dimensions fixed at 10 Retrieval times with increasing #Dimensionsin (secs) & DB size fixed at 1 lakh

15. Requirements • Efficient Image Retrieval • Learning relevant features in images • Learning image-image relationships • Diversity in retrieval for improved learning

16. Content Based Image Retrieval Query Feature Extraction Comparison Module Results Rf Relevance Learning Feature Index Memory & Logs

17. Learning - expectations Effective – capture user intent correctly Efficient – interactive retrieval response Scalable – limited computational overhead for large collections Adaptive – caters to individual user’s subjectivities • Intra-query or short term learning (STL) Evolving – incrementally improves across users and queries • Inter-query or long term learning (LTL) Dynamic – seamlessly absorbs changes in the collection

18. Learning - Method • Relative relevance of features using feedback • Numerous methods can be used • Discriminative variance is as - • Weights are incrementally learnt over iterations using – • At the end of the session long term learning is updated for the relevant images using – • Image to image dissimilarity is computed using – • Weighted Mahalanobis

19. Improved accuracy Precision across sessions using LTL Rank Convergence of top N relevant samples Sum of ranks of Top 10 relevant images converges close zero (downshifted) over multiple sessions with long term learning

20. Improved retrieval System learns the rock and sky pattern over sessions System learns the yellow flower in the hedge over sessions Top 9 results for queries across 3 different sessions (left-most are queries too)

21. Optimized Retrieval

22. Content from Learning • Long term memory allows learning of relevant image features • Converges to popular content over sessions • For example, • Assume, features are associated with individual pixels, colors • Consider a gray image, pixels for more relevant features are colored brighter Actual image Content image

23. Visual Content Extraction Over sessions After a large number of sessions

24. Requirements • Efficient Image Retrieval • Learning relevant features in images • Learning image-image relationships • Diversity in retrieval for improved learning

25. Content Based Image Retrieval Query Feature Extraction Comparison Module Results Rf Relevance Learning Feature Index Memory & Logs

26. Image-Image Relations Query Given a history of patterns in behavior and a current partial pattern, collaborative filtering predicts the next pattern for the latter Content Free Image Retrieval orCFIR, uses feedback logs to predict the next set of results for the current pattern

27. Hybrid Image Retrieval We integrate them in a Bayesian inference like framework, • The a priorirelationships from logs • Theevidence from visual similarity • Retrieval is an a posteriori estimation problem

28. Bayesian Image Retrieval System Architecture of the proposed Bayesian Image Retrieval System

29. Bayesian Image Retrieval... posterior = prior * evidence Efficient a priori updates • The prior probabilities are not stored, reduces updates • Co-relevance between images are stored in a matrix • The a prioriis estimated using the co-relevance values Evidence computation • Weights are learnt using discriminative variance method • WeightedMahalanobis for (dis)similarity

30. Concept Discovery a priorimatrix has embedded patterns of similar co-relevances Co-relevance patterns can be summarized into ‘k’ concepts • cluster the patterns into V concepts 1…k. • clustering is repetitive but offline • exhaustive comparisons are avoided

31. Accuracy with Bayesian Gain in precision with Bayesian Gain in precision across sessions Using real human user feedback logs Using annotation based feedback logs

32. Accuracy with Bayesian CBIR results and Bayesian results

33. Requirements • Efficient Image Retrieval • Learning relevant features in images • Learning image-image relationships • Diversity in retrieval for improved learning

34. Diversity in Image Retrieval Query Query

35. Skylines – the natural solution Results should be similar in a variety of different ways Skylines return non-dominated samples Non-dominated samples are closer to the query than all the others, in at-least one way (attribute)

36. Skyline Extraction Architecture of the proposed skyline based similarity retrieval system

37. Diversity with Skylines

38. Efficient Skylines Synthetic data with 10 dimensions and 10000 and 15000 data points Real image data with 12 and 9 dimensions with 11901 real images

39. Preferential Skylines Relevance feedback represents user’s preference Weights learned using feature relevance Skylines are then computed in user space

40. Contributions Designed and implemented a web-based image retrieval system, called FISH Proposed an efficient feature relevance learning algorithm Integration of complimentary CFIR and CBIR a Bayesian inference framework Skylines to retrieve diversely similar samples for a given query

41. Future directions Videos are richer and the next step Efficient higher level concept discovery is needed Skylines with preference should be explored further

42. Publications Pradhee Tandon, Piyush Nigam, Vikram Pudi, C. V. Jawahar, “FISH: A Practical System for Fast Interactive Image Search in Huge Databases”, in Proceedings of the 7th ACM International Conference on Image and Video Retrieval (CIVR ’08), July 6-8, 2008, Niagara Falls, Canada. Pradhee Tandon, C. V. Jawahar, “Long Term Learning for Content Extraction in Image Retrieval”, in Proceedings of the 15th National Conference on Communications (NCC ’09), January 16-18, 2009, Guwahati, India. Pradhee Tandon, C. V. Jawahar, “Bayesian Image Retrieval” submitted to 3rd International Conference on Pattern Recognition and Machine Intelligence (PReMI ’09), December 16-20, 2009, New Delhi, India.

43. Thank you 

44. Addendum

45. The Retrieval Algorithm *Learning discussed in detail later

46. Bayesian Image Retrieval • The a priori probability of retrieving image ‘a’with query ‘q’ is P(R) = n(q,a)/n(a) • where n(a) denotes relevant retrievals for ‘a’ • The evidence from visual similarity is computed as p(S|R) = f(w,q,a) • where weights ‘w’ are refined using relevance feedback • The posterior probability of retrieval is computed as p(R|S) = p(S|R) P(R) • the denominator can be ignored • PicHunter is a hybrid but does no feature learning • Zhong et. al, use Bayes inference for a probabilistic decision only

47. Skyline Extraction