Image Retrieval: Current Techniques, Promising Directions, and Open Issues

1. Image Retrieval: Current Techniques, Promising Directions, and Open Issues Yong Rui, Thomas Huang and Shih-Fu Chang Published in the Journal of Visual Communication and Image Representation. Presented by: Deepak Bote

2. Presentation Outline • History of image retrieval – Issues faced • Solution – Content-based image retrieval • Feature extraction • Multidimensional indexing • Current Systems • Open issues • Conclusion

3. History of Image Retrieval • Traditional text-based image search engines • Manual annotation of images • Use text-based retrieval methods • E.g. Water lilies Flowers in a pond <Its biological name>

4. Limitations of text-based approach • Problem of image annotation • Large volumes of databases • Valid only for one language – with image retrieval this limitation should not exist • Problem of human perception • Subjectivity of human perception • Too much responsibility on the end-user • Problem of deeper (abstract) needs • Queries that cannot be described at all, but tap into the visual features of images.

5. Outline • History of image retrieval – Issues faced • Solution – Content-based image retrieval • Feature extraction • Multidimensional indexing • Current Systems • Open issues • Conclusion

6. What is CBIR? • Images have rich content. • This content can be extracted as various content features: • Mean color , Color Histogram etc… • Take the responsibility of forming the query away from the user. • Each image will now be described by its own features.

7. CBIR – A sample search query • User wants to search for, say, many rose images • He submits an existing rose picture as query. • He submits his own sketch of rose as query. • The system will extract image features for this query. • It will compare these features with that of other images in a database. • Relevant results will be displayed to the user.

8. Sample Query

9. Sample CBIR architecture

10. Outline • History of image retrieval – Issues faced • Solution – Content-based image retrieval • Feature extraction • Multidimensional indexing • Current Systems • Open issues • Conclusion

11. Feature Extraction • What are image features? • Primitive features • Mean color (RGB) • Color Histogram • Semantic features • Color Layout, texture etc… • Domain specific features • Face recognition, fingerprint matching etc… General features

12. Mean Color • Pixel Color Information: R, G, B • Mean component (R,G or B)= Sum of that component for all pixels Number of pixels Pixel

13. Histogram • Frequency count of each individual color • Most commonly used color feature representation Corresponding histogram Image

14. Color Layout • Need for Color Layout • Global color features give too many false positives • How it works: • Divide whole image into sub-blocks • Extract features from each sub-block • Can we go one step further? • Divide into regions based on color feature concentration • This process is called segmentation.

15. Example: Color layout ** Image adapted from Smith and Chang : Single Color Extraction and Image Query

16. Texture • Texture – innate property of all surfaces • Clouds, trees, bricks, hair etc… • Refers to visual patterns of homogeneity • Does not result from presence of single color • Most accepted classification of textures based on psychology studies – Tamura representation • Coarseness • Contrast • Directionality • Linelikeness • Regularity • Roughness

17. Segmentation issues • Considered as a difficult problem • Not reliable • Segments regions, but not objects • Different requirements from segmentation: • Shape extraction: High Accuracy required • Layout features: Coarse segmentation may be enough

18. Presentation Outline • History of image retrieval – Issues faced • Solution – Content-based image retrieval • Feature extraction • Multidimensional indexing • Current Systems • Open issues • Conclusion

19. Problem of high dimensions • Mean Color = RGB = 3 dimensional vector • Color Histogram = 256 dimensions • Effective storage and speedy retrieval needed • Traditional data-structures not sufficient • R-trees, SR-Trees etc…

20. 2-dimensional space Point A D2 D1

21. 3-dimensional space

22. Now, imagine… • An N-dimensional box!! • We want to conduct a nearest neighbor query. • R-trees are designed for speedy retrieval of results for such purposes • Designed by Guttmann in 1984

23. Presentation Outline • History of image retrieval – Issues faced • Solution – Content-based image retrieval • Feature extraction • Multidimensional indexing • Current Systems • Open issues • Conclusion

24. IBM’s QBIC • QBIC – Query by Image Content • First commercial CBIR system. • Model system – influenced many others. • Uses color, texture, shape features • Text-based search can also be combined. • Uses R*-trees for indexing

25. QBIC – Search by color ** Images courtesy : Yong Rao

26. QBIC – Search by shape ** Images courtesy : Yong Rao

27. QBIC – Query by sketch ** Images courtesy : Yong Rao

28. Virage • Developed by Virage inc. • Like QBIC, supports queries based on color, layout, texture • Supports arbitrary combinations of these features with weights attached to each • This gives users more control over the search process

29. VisualSEEk • Research prototype – University of Columbia • Mainly different because it considers spatial relationships between objects. • Global features like mean color, color histogram can give many false positives • Matching spatial relationships between objects and visual features together result in a powerful search.

30. ISearch – my own system

31. ISearch – my own system

32. ISearch – my own system

33. Feature selection in ISearch

34. Database Admin facility in ISearch

35. Presentation Outline • History of image retrieval – Issues faced • Solution – Content-based image retrieval • Feature extraction • Multidimensional indexing • Current Systems • Open issues • Conclusion

36. Open issues • Gap between low level features and high-level concepts • Human in the loop – interactive systems • Retrieval speed – most research prototypes can handle only a few thousand images. • A reliable test-bed and measurement criterion, please!

37. Presentation Outline • History of image retrieval – Issues faced • Solution – Content-based image retrieval • Feature extraction • Multidimensional indexing • Current Systems • Open issues • Conclusion

38. Conclusion • Satisfactory progress, but still… A long way to go…!!

39. Acknowledgements • Dr. Padma Mundur • Mr. Yong Rao • Mr. Sumit Jain, Software Engineer, KPIT Cummins, India • Mr. Ajay Joglekar, Software Engineer, Veritas India.

40. References • Y. Rui, T. S. Huang, and S.-F. Chang, “Image retrieval: Current techniques, promising directions, and open issues” • S. Jain, A. Joglekar, and D. Bote, ISearch: A Content-based Image Retrieval (CBIR) Engine, as Bachelor of Computer Engineering final year thesis, Pune University, 2002

41. THANK YOU!!! Questions?