1 / 52

Selected Advanced Topics

Selected Advanced Topics. Storing and Retrieving Images Content-based Image/Video Indexing and Retrieval. Image/Video Database. Problem. Find all images contain horses …. Text-based technology. Annotation : Each image is indexed with a set of relevant text phrases, e.g.,

georgesimmons
Télécharger la présentation

Selected Advanced Topics

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Selected Advanced Topics Storing and Retrieving Images Content-based Image/Video Indexing and Retrieval

  2. Image/Video Database Problem Find all images contain horses …..

  3. Text-based technology • Annotation: Each image is indexed with a set of relevant text phrases, e.g., • Retrieval: based on text search technology Appropriate phrases to describe the content of this image include: Mother, Child, Vegetable, Yellow, Green, Purple ….

  4. Text-based technology - Drawbacks Annotation - subjective • different people may use different phrases to describe the same or very similar image/content

  5. Text-based technology - Drawbacks • Annotation - Laborious • It will take a lot of man-hours to label large image/video databases with 1m+ items

  6. Content-based Technology Using Visual Examples Image/Video Database

  7. r% g% b% Content-based Technology Using Visual Features Image/Video Database

  8. Content-based Technology • Content-based image indexing and retrieval (CBIR), is an image database management technique, which indexes the data items (images, or video clips) using visual features (e.g., color, shape, and texture) of the images or video clips. • A CBIR system lets users find pictorial information in large image and video databases based on visual cues, such as colour, shape, texture, and sketches.

  9. Content-based Technology • The visual features, computed using image processing and computer vision techniques are used to represent the image contents numerically. • Image Content - a high level concept, e.g., this image is a sunset scene, a landscape scene, etc. • Numerical Content Representations - Low level numbers, often the same set of numbers can come from very different images, making the task very hard!

  10. Content-based Technology • Techniques for Computing Visual Features/Representing Image Contents – • some are very sophisticated, and many are still not matured • hence the computational processes • in some cases are automatic • but in other cases are semi-automatic • in the most difficult cases, it may have to be done manually

  11. Content-based Technology • Comparing Image Content/Retrieving Images based on Content • Simple approaches - compute the metric distance between low level numerical representations • Advanced Approaches - using sophisticated pattern recognition, artificial intelligence, neural networks, and interactive (relevant feed-back) techniques to compare the visual content (low level numerical features)

  12. Content-based Technology - IBM QBIC System • The IBM’s QBIC (Query by Image and Video Content) system is one of the early examples of CBIR system developed in 1990s. • The system lets users find pictorial information in large image and video databases based on color, shape, texture, and sketches.

  13. Content-based Technology - IBM QBIC System • The User Interfaces Module • Let user specify visual query by drawing, selecting from a color wheel, selecting a sample image … • Display results as an ordered set of images • The Database Population and Database Query Modules • Database population - process images and video to extract features describing their content - colors, textures, shapes and camera and object motion, and store the features in a database • Database Query - let user compose a query graphically, extract features from the graphical query, input to a matching engine that finds images or video clips with similar features

  14. Content-based Technology - IBM QBIC System • The Data Model • Still image, or scene - full image • Objects contained in the full image - subsets of an image • Videos - broken into clips called shots - sets of contiguous frames • Representative frames, the r-frames, are generated for each shot • R-frames are treated as still image - from which features are extracted and stored in the database. • Further processing of shots generates motion objects - e.g., a car moving across the screen.

  15. Content-based Technology - IBM QBIC System • Queries are allowed on • Objects - e.g., Find images with a red round object • Scenes - e.g., Find images that have approximately 30% red and 15% blue colors • Shots - e.g., Find all shots panning from left to right • A combination of above - e.g., Find images that have 30% red and contain a blue textured objects

  16. Content-based Technology - IBM QBIC System • Similarity Measures • Similarity queries are done against the database of pre-computed features using distance functions between the features • Examples include, Euclidean distance, City-block distance, …. • These distance functions are intended to mimic human perception to approximate a perceptual ordering of the database • But, it is often the case that a distance metric in a feature space will bear little relevance to perceptual similarity.

  17. color color color color color texture texture texture texture texture shape shape shape shape shape positions positions positions positions positions …. …. …. …. …. Record n Record3 Record2 Record1 Record4 color texture shape positions …. Content-based Technology - Basic Architecture Similarity Measures Imagery Meta data Query Image Database

  18. What soft drink Which fruit? Colour - An effective Visual Cue Colors can be a more powerful visual cue than you initially thought!

  19. Colour - An effective Visual Cue In many cases, color can be very effective. Here is an example Results of content-based image retrieval using 4096-bin color histograms

  20. B Cyan White Magenta G gray R Yellow Colour Spaces Colour Models RGB Model: This colour model uses the three NTSC primary colours to describe a colour within a colour image. Sometimes in Computer Vision, it is convenient to use rg chromaticity space r = R/(R+G+B) g= G/(R+G+B)

  21. Colour Spaces YIQ Model: The YIQ models is used in commercial colour TV broadcasting, which is a re-coding of RGB for transmission efficiency and for maintaining compatibility with monochrome TV standard. In YIQ, the luminance (Y) and colour information (I and Q) are de-coupled. YCbCr Model Y = 0.299R + 0.587G + 0.114B Cb = -0.169R - 0.331G + 0.500B Cr = 0.500R - 0.419G - 0.081B

  22. Perceived Color Differences • One problem with the RGB colour system is that colorimetric distances between the individual colours don't correspond to perceived colour differences. • For example, in the chromaticity diagram, a difference between green and greenish-yellow is relatively large, whereas the distance distinguishing blue and red is quite small. r = R/(R+G+B) g= G/(R+G+B)

  23. CIELAB • CIE (Commission Internationale de l'Eclairage)solved this problem in 1976 with the development of the Lab colour space. A three-dimensional color space was the result. In this model, the color differences which you perceive correspond to distances when measured colorimetrically. The a axis extends from green (-a) to red (+a) and the b axis from blue (-b) to yellow (+b). The brightness (L) increases from the bottom to the top of the three-dimensional model. • With CIELAB what you see is what you get (in theory at least).

  24. Colour Histogram • Given a discrete colour space defined by some colour axes (e.g., red, green, blue), the colour histogram is obtained by discretizing the image colours and counting the number of times each discrete colour occurs in the image. • The image colours that are transformed to a common discrete colour are usefully thought of as being in the same 3D histogram bin centered at that colour.

  25. Colour Histogram Construction • Step 1 Colour quantization (discretizing the image colours) • Step 2 Count the number of times each discrete colour occurs in the image.

  26. Colour Quantization • A true colour, 24-bit/pixel image (8 bit - R, 8 bit - G, 8 bit -B), will have 224 = 16777216 bins ! • That is, each image will have to be represented by over 16 million numbers • computationally impossible • in practice not necessary • Colour quantization - reduce the number of (colours) bins

  27. Simple Colour Quantization • Simple Colour Quantization (Non-adaptive) • Divide each colour axis into equal length sections (different axis can be divided differently). • Map (quantize) each colour into its corresponding bin

  28. R G B 0 0 0 31 31 31 63 63 63 95 95 95 127 127 127 159 159 159 191 191 191 223 223 223 255 255 255 Simple Colour Quantization Example: In RGB space, quantize each image colour into one of 8x8x8 = 512 colour bins Colour Bin Colour Bin (123,23,45) (3, 0, 1 ) (122, 28, 46) (3, 0, 2) (132, 29,50) (4, 0, 1) (122, 172, 27) (3, 5, 0) (121,26,48) (x, x, x) (142, 28, 46) (x, x, x)

  29. Advanced Colour Quantization • Adaptive Colour quantization (Not required) • Vector Quantization • K-means clustering • K representative colours • The colour histogram consists of K bins, each corresponding to one of the representative colours. • A pixels is classified as belonging to the nth bin if the nth representative colour is the one (amongst all the representative colours) that is closest to the pixel. A pixel is a point in the 3D colour space B G R Representative colours

  30. Colour Histogram Construction - An Example • A 3 x 3, 24-bit/pixel image has following RGB planes • Construct an 8-bin colour histogram (using simple colour quantization, treating each axis as equally important). Green 213 24 77 11 232 239 22 12 12 Blue 23 24 77 12 24 69 22 123 123 Red 23 24 77 11 24 69 22 12 12 Bin (0,0,0) = Bin (0,0,1) = Bin (0,1,0) = Bin (0,1,1) = Bin (1,0,0) = Bin (1,0,1) = Bin (1,1,0) = Bin (1,1,1) =

  31. Colour Histogram Construction - An Example • Quantized Colour Planes • Count the number of times each discrete colour occurs in the image. Green 1 0 0 0 1 1 0 0 0 Blue 0 0 0 0 0 0 0 0 0 Red 0 0 0 0 0 0 0 0 0 Bin (0,0,0) = 6 Bin (0,0,1) = 0 Bin (0,1,0) = 3 Bin (0,1,1) = 0 Bin (1,0,0) = 0 Bin (1,0,1) = 0 Bin (1,1,0) = 0 Bin (1,1,1) = 0

  32. # of pixels # of pixels 10 0 0 0 100 10 30 0 0 = Color Distribution (10,0,0,0,100,10,30,0,0) Colour Based Image Indexing The histogram of colours in an image defines the image colour distribution

  33. = Colour Distribution (10,0,0,0,90,10,40,0,0) Colour based Image Retrieval Images are similar if their histograms are similar! Colour Distribution = (10,0,0,0,100,10,30,0,0) Dissimilar Similar! Colour Distribution = (0,40,0,0,0,0,0,0,110,0)

  34. = Colour Distribution (10,0,0,0,100,10,30,0,0) = Colour Distribution (0,40,0,0,0,0,0,110,0) Formalizing Similarity 1 2 Similarity(Image 1, Image 2) = D (H1, H2) where D( ) is a distance measure between vectors (histograms) H1 and H2

  35. a b c Metric Distances A distance measure D( ) is a good measure if it is a metric! D(a,b) is a metric if D(a,a) = 0 (the distance from a to itself is 0 D(a,b) = D(b,a) (the distance from a to b = distance from b to a) D(a,c) <= D(a,b) + D(b,c) ( triangle inequality [ the straight line distance is always the least!] ) D(a,b) + D(b,c) should be no smaller than D(a,c)

  36. HI(H1,H2) = Common Metric Distance measures Histogram Intersection, HI H1 = (10, 0, 0, 0, 100, 10, 30, 0, 0) H2 =( 0, 40, 0, 0, 0, 6, 0, 110, 0) Similarity = HI(H1, H2) = 0 + 0 + 0 + 0 + 0 + 6 + 0 + 0 = 6

  37. Root-mean square error Common Metric Distance measures Euclidean or straight-line distance or L2-norm, D2 H1 = (10, 0, 0) H2 =( 0, 40, 0) Similarity = D2(H1, H2) = sqrt(100 + 1600 +0) = 41.23

  38. sum of absolute differences Common Metric Distance measures Manhattan or city-block or L1-norm, D1 H1 = (10, 0, 0) H2 =( 0, 40, 0) Similarity = D1(H1, H2) = (10 + 40 +0) = 50

  39. Histogram Intersection vs City Block Distance Theorem: if H1 and H2 are colour histograms and the total count in each is N (there are N-pixels in an image) then: (Histogram Intersection inversely proportional to a metric distance!) Proof (by definition) (1) (2)

  40. Histogram Intersection vs City Block Distance (3) Substituting (2) and (3) in (1) (4) (5)

  41. Build Histogram (1) Histogram Database (2) Colour Histogram Database

  42. How well does Color histogram intersection work ? 66 test histograms in the database Swain Original Test: 31 query images Recognition rate almost 100% Indeed, because color indexing worked so well it is at he heart of almost all image database systems

  43. Google Image Search

  44. Google Image Search After clicking this colour patch

  45. Problems with color histogram matching 1. Color Quantization problem: = Colour Distribution (0,40,0,0,0,0,0,110,0) Because, the two images have slightly different color distributions their histograms have nothing in common! 0 intersection! = Colour Distribution (0,0,40,0,0,0,0,0,110) Sources of quantization error: noise, illumination, camera

  46. Problems with color histogram matching 2. The resolution of a color histogram = Colour Distribution (0,40,0,0,0 … ,0,0,110,0) For the best results, Swain quantized colour space into 4096 distinct colours => Each colour distribution is a 4096-dimensional vector. => Histogram intersection costs O(4096) operations (some constant * 4096) 4096 comparisons per database histogram => histogram intersection will be very slow for large databases Many newer methods work well using 8 - 64 D features

  47. Problems with color histogram matching 3. The colour of the light Under a yellowish light all image colours are more yellow than they ought to be

  48. Problems with color histogram matching 4. The structure of colour distribution All four images have the same color distribution - need to take into account spatial relationships!

  49. = Mean = Variance/ Covariance Problem solution => Use statistical moments 1st order statistics 2nd order statistics

  50. Statistical similarity = Colour Distribution (50,50,50) Compare mean RGBs (In general compare all statistical measures) = Colour Distribution (20,70,40) Statistical similarity = (Euclidean distance between corresponding statistical measures)

More Related