1 / 101

Outline

Outline. Content-Based Image Retrieval Query-by-Example Query-by-Feature Feature Vector. CBIR and CBR. Content-based Image Retrieval (CBIR) as example of Content-based Retrieval (CBR) concentrates on low-level features. Main Ideas of CBIR:

mandy
Télécharger la présentation

Outline

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Outline • Content-Based Image Retrieval • Query-by-Example • Query-by-Feature • Feature Vector

  2. CBIR and CBR • Content-based Image Retrieval (CBIR) as example of Content-based Retrieval (CBR) • concentrates on low-level features. • Main Ideas of CBIR: • Represent an image as a set of feature descriptors. • Define similarity measures of the descriptors. • When a user specifies a query, the system returns images, which are sorted by similarity.

  3. CBIR Architecture

  4. CBIR of Butterflies • To allow non-expert users to find out some possible species of the butterflies they saw by the appearance of the butterflies • The appearance: • Color, Texture, Shape

  5. Problems • How can you describe a butterfly? • How can you communicate with a machine?

  6. Problems • Different users have different perception. • Users may not remember the appearance of the butterfly clearly. • Users usually do not have enough knowledge to describe the butterflies like experts. • Users usually do not have patience to browse too many query results.

  7. Solutions • An user-driven interactive query process: QBF/QBE query process • Query By Features and Query By Example • Fuzzy feature description for each butterfly • An “What You See Is What You Get” query interface • A representative set for our collection of butterflies

  8. QBF/QBE query process (1) • QBF query: • A QBF query is to choose some features of butterflies and expect that the system returns all butterflies with those features. • Features of butterflies: • Dominant color, texture pattern, shape. • QBE query: • A QBE query is to point an image and expect that the system returns all butterflies similar to that.

  9. QBF/QBE query process (2) • Properties of QBF: • Rough search • When to use: • The first query and when users want to enlarge the view in the search space • Properties of QBE: • Fine search • When to use: • Usually the last query and when users want to see the neighbors of the query one in the search space.

  10. QBF/QBE query process (3) • Result page: (Each result page should contain two parts) • Result Images: • These are the butterfly images satisfy the query conditions. • Users can invoke QBE queries from these images. • Related Features: • These are the features related to the previous query conditions. • Users can invoke QBF queries from these features.

  11. Feature Description (1) • Feature Description for a butterfly: • Like metadata which describe the appearance of this butterfly. • This makes QBF queries possible. • Feature Description consists of some feature descriptors. • Feature descriptor: • A ( “feature value” , “match level” ) pair.

  12. Feature Description (2)

  13. Feature Description (3) Color

  14. Feature Description (4) Texture

  15. Feature Description (5) Shape

  16. Feature Description (6) • QBF query: • Single feature query: • Result images: images with its corresponding degree of match > 0. • Ranked by: degree of match. • We call this ranked sequence “Feature sequence.” • Multiple features query: • Merge the corresponding feature sequences.

  17. Result Presentation • For QBF query: • Property: rough search • Presentation: representative butterflies only • For QBE query: • Property: fine search • Presentation: • For very similar images: present them all • For less similar images: representative ones

  18. Feature Vector Indexing • Goal: • To make search efficiently. • Problems of Indexing in CBIR: • Dimension of feature space is very high. • Index structure should support Euclidean and non-Euclidean similarity measures. • Solution: • Dimension reduction: KLT, DCT, DWT. • Similarity indexing: R*-tree, SS-tree, SR-tree.

  19. Semi-Automatic Feature Extraction • Segmentation: • Background segmentation • Butterfly object segmentation • Feature extraction: • Color: color histogram • Texture: manual annotation • Shape: manual annotation

  20. Classic CBIR with Color Feature • Most of the CBR systems rely on the notion of color, this may differ: • Dominant color • Scalable color based on color histograms (local for one region, global for the whole image) • Color Structure Descriptor (incoporates the spatial structure) • Let’s see the CBIR color theory and describe how MPEG-7 copes with it. • Let’s start why we focus on color and 90% of the CBIR is on color.

  21. What color is the apple ? We are so visual !!!! I’d say it is Bright Red I really couldn’t tell you (I am color blind) I think it is “Crimson” It is Red!

  22. Cosmic rays X-rays UV IR Radio waves 400 nm-700 nm “visible light” Visible light is part of a wide electro-magnetic spectrum 100 1.0 0.8 0.6 0.4 0.2 0.0 Relative Sensitivity 0 400 500 600 700 200 400 600 800 1000 Wavelength (nm) Wavelength (nm) • Response of human eye (luminance response) • Response of Silicon

  23. RGB Color Space • Hardware Oriented Model: RGB Color Space: 3 values to represent a color. Red Green Blue

  24. HSV Color Space • HSV – Hue Saturation Value • close to human perception • 3 values to represent a color. Value Green (120o) Yellow (60o) Cyan (180o) Red (0o) Blue (240o) Magenta (300o) White Black Hue Saturation

  25. YCbCr Color Space • Y is the Luminance and Cb and Cr are the Chrominance Values of this Color Space. • Decouples intensity and color information • A monochrome color representation has only the Y value. • Very Perceptual Model, see next slide.

  26. Luminance and Chrominance Achromatic + Red cone + + (Luminance) + Rods + Red-Green - + Green Cone + (Chrominance) Yellow - Blue-Yellow + Blue cone (Chrominance) • Eye has higher sensitivity for Luminance

  27. HMMD Color Space • HMMD (Hue-Max-Min-Diff) color space is closer to a perceptually uniform color space. • Min = min(R,G,B); • Max = max(R,G,B); • Diff = Max – Min; • Sum = (Max + Min)/2; • -Hue the same as in HSV

  28. Max: indicates how much black color it has, giving a flavor of shade or blackness. Min: indicates how much white color it has, giving a flavor of tint or whiteness. Diff: indicates how much gray it contains and how close to the pure color, giving a flavor of tone or colorfulness. Sum: simulates the brightness of the color.

  29. Color Space: Representation • RGB: [int, int, int] • Y Cb Cr: [int, int, int] • HSV: [int, int, int] • Value • lightness of the color • Saturation • how dominant a color is • Hue • dominant spectral tone of the color

  30. v = max(R,G,B); s = (v-min(R,G,B))/v; if(R=max&&G=min) h=5+(R-B)/(R-G); else if(R=max&&B=min) h=1-(R-G)/(R-B); else if(G=max&&B=min) h=1+(G-R)/(G-B); else if(G=max&&R=min) h=3-(G-B)/(G-R); else if(B=max&&R=min) h=3+(B-G)/(B-R); else if(B=max&&G=min) h=5-(B-R)/(B-G);

  31. Color Space: Distance • Weighted Euclidean Distance (L-2 metric) • Better Distance for HSV • w.r.t human perceptual system • interpreted as cylinder v h s

  32. Distance Problem in Color Spaces: • a difference between green and greenish-yellow is relatively large, whereas the distance distinguishing blue and red is quite small. CIE 1931, chromaticity diagram

  33. CIE-LAB Uniform Color Space • Uniform Color Spaces: CIE Lab • CIE solved this problem in 1976 with the development of the Lab color space.

  34. Color Histogram: Representation • A list of Color-Percentage pairs: • Describe the colors and its percentages in an image.

  35. Color Quantization • Indexed Colors • A jpg Image with 256-color components in each RGB channel • 256 x 256 x 256 colors in total → n groups, e.g, in 256 groups, that makes a reduction 256x256, I.e., that each group takes 256 colors to count.

  36. In each group we compute how many pixels fall into this group, this gives e.g., 145. • Quantization means also the encoding of the bin values with a number of bits, • Criteria:

  37. Color Representation in General T H Q B R b[m] S h[m] G V indexing B Linear or non-linear transformation Quantization Transform to binary

  38. Example • Color space selection & quantization • Use RGB channels • Each channel is divided into 2 intervals • Total number of bins = 23 = 8

  39. Example (cont.) • H(I): Color histogram for Image I • Three images of 8x8 pixels, each pixel in one of eight color bins C1 to C8 • Image 1 has 8 pixels in each eight color bins • H1 = (8, 8, 8, 8, 8, 8, 8, 8) • Image 2 has 7 pixels in each C1 to C4, and 9 pixels in each C5 to C8 • H2 = (7, 7, 7, 7, 9, 9, 9, 9)

  40. Image 3 has 2 pixels in each C1 and C2, and 10 pixels in each C3 to C8 • H3 = (2, 2, 10, 10, 10, 10, 10, 10) • Quantization to 4 bins -> • H1 = (16, 16, 16, 16) • Quantization to 3bits -> • H1 = (7, 7, 7, 7)

  41. Similarity Measures - Overview • Minkowski Similarity • Distance L1 : r = 1 • Distance L2 : r = 2 • Quadratic Similarity • Intersection Similarity (Swain et Ballard 1991)

  42. Example (cont.) • Minkowski Similarity • Is a L-1 metric where Ik and Jk is the number of pixels in bin k for image I and J • Distance between above three images • D(H1, H2) = 1 + 1 + 1 + 1 + 1 + 1 + 1 + 1 = 8 • D(H1, H3) = 6 + 6 + 2 + 2 + 2 + 2 + 2 + 2 = 24 • D(H2, H3) = 5 + 5 + 3 + 3 + 1 + 1 + 1 + 1 = 23

  43. Example (cont.) • Minkowski Similarity • Is a L-2 metric • Distance between above three images • D(H1, H2) = (1 + 1 + 1 + 1 + 1 + 1 + 1 + 1)1/2 = 2.8 • D(H1, H3) = (36 + 36 + 4 + 4 + 4 + 4 + 4 + 4)1/2 = 9.8 • D(H2, H3) = (25 + 25 + 9 + 9 + 1 + 1 + 1 + 1) 1/2 = 8.5

  44. QBIC distance • Weighted Euclidean distance (QBIC) • Is a L-2 metric(?) distance between histogram H1 and H2: D = (H1 - H2)T A (H1 - H2) where A is a symmetric color similarity matrix A (i, j) = 1 –d (ci, cj) / dmax where ci and cj are the i-th and j-th color bins, d (ci , cj) is the color distance in the color space, and dmax is the maximum distance between any two colors in the color space

  45. Limitation • Ignore similarity between colors • Example • Two color bins • Bin-1 color range: 1 – 10 • Bin-2 color range: 11 – 20

  46. Three color pixels • Pixel 1 is Color 10  Bin-1 • Pixel 2 is Color 11  Bin-2 • Pixel 3 is Color 20  Bin-2 • Pixel 2 is similar to Pixel 3 than Pixel 1  unreasonable !

  47. Limitation (cont.) • Ignore spatial relationships among pixels Different image with same histogram

  48. Spatial RelationshipsAmong Pixels • Segment image into regions & calculate histogram for each region • Remove background & calculate histogram of foreground • Correlogram • Use spatial layouts (see MPEG-7)

  49. Color Representation in MPEG-7 MPEG-7 distinguishes: • Color Space and Color Quantization, • Descriptive Elements: • dominant, scalable, (histogram based, counting of pixels that fulfill criteria), • color-structure, layout, (spatial layout), • GoF/GoP (for video frames), • For both global image and regions of image.

  50. Color Space Description in MPEG-7 Like : RGB, YCrCb, HSV, HMMD, linear transformation with reference to RGB

More Related