Content-based Image Retrieval

1. Content-based Image Retrieval Presentation by Charlie Neo

2. Introduction • Why • Digital image database growing rapidly in size • Professional needs – Logo Search • Difficulty in locating images on the web • Example • Find a picture of me and jack on the bus

3. Application CBIR • Search for one specific image. • General browsing to make an interactive choice. • Search for a picture to go with a broad story or search to illustrate a document. • Search base on the esthetic value of the picture.

4. Two Classes of CBIRNarrow vs. Broad Domain • Narrow • Medical Imagery Retrieval • Finger Print Retrieval • Satellite Imagery Retrieval • Broad • Photo Collections • Internet

6. Challenges • Semantic gap • The semantic gap is the lack of coincidence between the information that one can extract from the visual data and the interpretation that the same data have for a user in a given situation. • User seeks semantic similarity, but the database can only provide similarity by data processing. • Huge amount of objects to search among. • Incomplete query specification. • Incomplete image description.

7. Description Of Content: Image Processing • Color • Local Shape • Texture

8. Color Image Processing • Problems with color variances • Surface Orientation • Camera Viewpoint • Position of Illumination • Intensity of the Light

9. Color Image Processing • Approaches • Opponent color axes • Advantage of isolating the brightness information on the third axis. • Invariant to changes in illumination intensity and shadows. • HSV-representation • Invariant under the orientation of the object with respect to the illumination and camera direction. • Search for clusters in a color histogram to identify which pixels in the image originate from one uniformly colored object.

10. Image Processing for Local Shape • Problems • Occlusion • Different Viewpoint • Approaches • Collect all properties that capture geometric details in the image. • Invariant Descriptors.

11. Image Texture Processing • Problems • Offer little semantic referent. • Approaches • Markovian analysis • Wavelets • Generated by groups of dilations or dilations and rotations • Some semantic correspondent. • Great For • Satellite images • Images of documents

12. Description Of Content using Features • Grouping Data • Strong Segmentation • Region T = 0 (object) • Shape and Object features • Weak Segmentation • T subset of 0 • Salient features • Sign Detection • Signs Probabilities • Partitioning • Global feature

13. Global and Accumulating Features • Accumulation of features using a histogram • The set of features F(m) ordered by histogram index m. • 64-bin histogram, has discriminating power up to 25,000 images

14. Salient Features • Weak segmentation. (grouping of data into homogeneous regions.) • Salient feature calculations lead to sets of regions or points with known location and feature values. • Innovation of content-based retrieval. • Expected to receive much attention in the further expansion of the field.

15. Signs • Typical signs are an icon, a character, a traffic light, or a trademark. • Strong semantic interpretation is within grasp • Analysis tends to become application-oriented.

16. Shape and Object Features • Object segmentation is hard. • Possible for narrow domain. • Fortunately, for our purpose it only requires detection of the object’s presents.

17. Description of Structure and Lay-Out • Structural feature • Feature values • Relationships between object sets. • captured in a graph • Lay-out descriptions • characterized by locations, size, and features.

19. Interpretation And Similarity • Semantic Interpretation • Derive interpretation from feature set. • Features generate a probability distribution. • MAVIS2-system: four semantic layers. • Similarity • Similarity measure Sq,d between the images q and d • Sq,d = s(Fq,Fd). • s(Fq,Fd) = g( d(Fq,Fd) ) • e.g. dcan be just the Euclidean distance.

20. User Interaction • Query Space: Definition and Initialization • Q = {IQ,FQ,SQ,ZQ} • IQ is a selection of images from the large image archive I. • FQ is a selection of features from feature set F. • SQ similarity function. • ZQ is a set of labels to capture goal dependent semantics.

21. Query Specification

22. Query Space Display • Besides just showing the images that match the query … • Images are placed in such a way that distances between images in the display reflect SQ. • Highlight parts indicating which parts of the image fulfill the criteria. (exact query)

23. Interacting with Query Space • The process of query specification and display is iterated, where, in each step, the user revises the query. • user feedback leads to an update of query space: • Both positive and negative examples is used. • Each iteration, the probability of being the target for an image in IQ is increased or decreased

24. SYSTEM ASPECTS:Storage and Indexing • Standard, Linear File System • O(N) • Three classes of indexing methods • Space partitioning • Data partitioning • Distance-based technique • Varies tree structure • O(log N)

25. SYSTEM ASPECTS:System Architectures • Separate indexing and retrieval. • Image retrieval as a plug-in module to an existing database system. • Analysis, indexing and training as modules

26. SYSTEM ASPECTS:System Evaluation • Relevance is subjective. • Human subjects to produce idea ordering for a query.

27. Cortina: A System for large scale, content based Web Image Retrieval • Built for Web Image Retrieval • 3 million images • Clusters • Data Mining for Semantics

28. Cortina: A System for large scale, content based Web Image Retrieval

29. Cortina: A System for large scale, content based Web Image Retrieval • Four global feature descriptor • Homogeneous Texture Descriptor • Edge Histogram Descriptor • Scalable Color Descriptor • Dominant Color Descriptor • Linear combination of the 4 features, as the distance for K-NN search.

30. Cortina: A System for large scale, content based Web Image Retrieval

31. Conclusion • There is a need for CBIR • Image retrieval does not entail solving the general image understanding problem. It may be sufficient that a retrieval system present similar images, similar in some user-defined sense. • Interaction • The need for database • The semantic gap