A Novel Log-based Relevance Feedback Technique in Content-based Image Retrieval

1. ACM Multimedia 2004 12th Annual Conference, October 10 -16, 2004 New York City, Columbia University A Novel Log-based Relevance Feedback Technique in Content-based Image Retrieval Steven Chu-Hong Hoi & Michael R. Lyu Department of CSE The Chinese University of Hong Kong Shatin, Hong Kong SAR {chhoi, lyu}@cse.cuhk.edu.hk

2. Outline • Introduction & Motivation • Log-based Relevance Feedback • Soft Label Support Vector Machine • Experimental Results • Conclusions and Future Work

3. Introduction • Content-based Image Retrieval (CBIR) • Attract much interest, studied for many years • An important component in multimedia retrieval • Query based on low-level visual content: color, texture, shape, etc. QBE Challenge: the semantic gap between low-level features and high-level concepts

4. Introduction • Relevance Feedback (RF) in CBIR • A powerful technique, attack the semantic gap problem • Using interactive mechanisms, soliciting users’ interactions, learning users’ high-level concepts • Boosting retrieval performance effectively • Many popular techniques: MARS, QEX, MindReader, Optimizing learning, SVM (active), Boosting, etc. • Problems • Regular relevance feedback techniques: a lot of times of feedback which will cost much time and make users boring

5. Motivation Relevance Feedback ? Users’ Feedback Logs Problem Can users’ feedback logs information be used to improve the regular relevance feedback?

6. LRF: Log-based Relevance Feedback • Problem Formulation • Construct a Relevance Matrix: RM • Each log session: (N = + ) N images are marked: relevant & irrelevant instances • Values: relevant (+1), irrelevant (-1), unknown (0) Image samples in the image database -1 -1 1 1 1 -1 -1 0 1 -1 1 Log Sessions -1 1 -1 -1 -1 -1 -1 0 -1 1 -1

7. Log-based Relevance Feedback (cont’d) • Relationship Measurement • For each given session k , if the image i is marked as ‘relevant’ (positive) and the image ‘j’ is marked as ‘irrelevant’ (negative), then the elements are represented as RM (k, i) = 1 and RM (k, j) = -1 • For every two images: i and j, their relationship can be measured by a modified correlation function:

8. LRF Algorithm • Collection of training Samples • Regular relevance feedback • Learn only with a limited number of training samples • Cannot achieve good performance without enough training samples • Idea: finding more samples based on N initial samples • For an initial positive sample i, the relevance degrees between every image sample jof the database are computed by a soft label function: • By ranking the soft label values, we can collect a number of samples with larger soft label values corresponding to the sample i.

9. LRF Algorithm (cont’d) • The learning issue of the algorithm • Based on the initial marked samples and the log information, we can collect a large number of positive and negative training samples associated with soft labels which represent their confidence degrees. • Problem: how to develop the algorithm to learn the data associated with soft labels ? • Proposed Solution: Soft Label Learning • Soft Label Support Vector Machine (SLSVM)

10. SLSVM: Soft Label Support Vector Machine 1 • Problem Formulation 1 1 SVM 0.5 1 1 1

11. SLSVM (cont’d)

12. SLSVM (cont’d)

13. Experimental Results • Datasets • Images selected from COREL image CDs • 20-Category: 2000 image instances • 50-Category: 5000 image instances • Each category contains a specific semantic meaning

14. Experimental Results (cont’d) • Image Representation • Color Moment • 9-dimension • Edge Direction Histogram • 18-dimension • Canny detector, 18 bins of 20 degrees • Wavelet-based texture • 9-dimension • Daubechies-4 wavelet, 3-level DWT • 9 subimages are selected to generate the feature

15. Experimental Results (cont’d) • Log Format • Define a Log Session (LS) as a basic log unit, that corresponds to a relevance feedback round • Each log session contains 20 images marked by users • Log Collection • Collect logs from 10 users • Non-noisy logs: 100 LS • Noisy logs: • 20-Category: 103 LS, 7.2% noise • 50-Category: 138 LS, 8.1% noise

16. Experimental Results (cont’d) • Compared Schemes • EU (Euclidean distance - baseline) • RF_QEX (QEX: query expansion) • Multiple instance sampling, pick N nearest samples recursively • RF_SVM • Regular relevance feedback by SVM • LRF_QEX • Similar to RF_QEX, but we pick the samples weighted by soft labels in our framework (the larger the label, the smaller the distance) • LRF_SLSVM

17. Experimental Results (cont’d) • Settings • Same Kernels: e.g. RBF kernel • Evaluation metric: Average Precision = # of relevance / # of returned • Automatic evaluation: Taking average precision over 200 query executions

18. Experimental Results (cont’d) • Performance Comparison

19. Experimental Results (cont’d) • Performance Comparison

20. Experimental Results (cont’d) • Performance Comparison

21. Conclusions • In this paper we proposed a new scheme to study users’ feedback logs for improving the performance of regular relevance feedback in CBIR. • We introduce the soft label learning concept and developed a modified SVM technique, i.e. Soft Label SVM, to construct the algorithm for log-based relevance feedback. • We evaluate our proposed method compared with traditional techniques and demonstrate promising results.

22. Limitations & Future Work • The proposed LRF with SLSVM algorithm still suffers performance drop when many noisy logs are appeared. • Much noise may be involved when the scale of the image database is increased. • When the number of log sessions is large, the dimension of the relevance matrix may be a problem. • Training time of SLSVM need be considered for large scale datasets. Open questions: • Can we work out more effective Soft Label Learning techniques in the future? • Can we include some noise filtering techniques into our framework?

23. Thank You! Q & A

24. References (part) • [He & King 2003] X. He, O. King, W.-Y. Ma, M. Li, and H. J. Zhang. Learning a semantic space from user’s relevance feedback for image retrieval. IEEE Transactions on Circuits and Systems for Video Technology, 13(1):39–48, Jan. 2003. • [Huang & Zhou 2001] T. S. Huang and X. S. Zhou. Image retrieval by relevance feedback: from heuristic weight adjustment to optimal learning methods. In Proceedings of IEEE International Conference on Image Processing (ICIP’01), Thessaloniki, Greece, Oct. 2001. • [Hong & Huang 2000] P. Hong, Q. Tian, and T. Huang. Incorporate support vector machines to content-based image retrieval with relevant feedback. In Proc. IEEE International Conference on Image Processing (ICIP’00), Vancouver, BC, Canada, 2000. • [Rui & Huang 1999] Y. Rui and T. S. Huang. A novel relevance feedback technique in image retrieval. In Proc. ACM Multimedia (MM’99), pages 67–70, Orlando, Florida, USA, 1999.on Image Processing (ICIP’00), Vancouver, BC, Canada, • [Tong & Change 2001] S. Tong and E. Chang. Support vector machine active learning for image retrieval. In Proceedings of the ninth ACM international conference on Multimedia, pages 107–118. ACM Press, 2001.

25. Appendix • Kernel Comparison

26. CBIR