1 / 36

Glasgow 02/02/04

Glasgow 02/02/04. NN k networks for content-based image retrieval Daniel Heesch. Overview. CBIR – a problem of image understanding? Approaches to feature weighting The NN k technique for retrieval Getting connected: NN k networks for browsing Future work. Challenges: Semantic gap.

rasul
Télécharger la présentation

Glasgow 02/02/04

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Glasgow 02/02/04 NNk networks for content-based image retrieval Daniel Heesch

  2. Overview • CBIR – a problem of image understanding? • Approaches to feature weighting • The NNk technique for retrieval • Getting connected: NNk networks for browsing • Future work

  3. Challenges: Semantic gap What is the relationship between low-level features and image meaning?

  4. Challenges: Image polysemy one image - multiple meanings

  5. Feature Weights

  6. Approaches to feature weighting (1) Post retrieval relevance feedback • effectiveness relies on a good first retrieval result • useful for fine-tuning weights

  7. Approaches to feature weighting (2) SVM Metaclassifier (Yavlinsky et al., ICASSP 2004) • Given a set of queries and ground truth, for each query: • sample at random m positive and negative images and for each build a score vector consisting of the feature-specific similarities between that image and the query • Use an SVM to obtain a hyperplane that separates positive and negative score vectors with least error • For an unseen image, compute similarity to the query as the distance of its score vector to the hyperplane.

  8. Approaches to feature weighting (2) • The distance of a vector to a hyperplane is a weighted sum of the vector’s components, which is just the aggregation formula shown to you previously. • The hyperplane represents a set of feature weights that maximise the expected mean average precision

  9. Approaches to feature weighting (2) • ~ 300 score vectors needed to establish near-optimal weights for a subset of the Corel collection on average (6192 images) • No query-specific weight optimization

  10. Approaches to feature weighting (3) Query-specific optimization (Aggarwal et al., IEEE Trans. Multimedia 2002) • Modify query representation along each feature axis and regenerate modified query, ask user whether new query image is still relevant • Interesting idea but limited applicability in practice

  11. NNk retrieval

  12. The idea of NNk – a two-step approach • Retrieve with all possible weight sets -> returns a set of images (NNk) each associated with a particular weight • Retrieve with the weights associated with the relevant images the user selects

  13. The first retrieval step: finding the NNk • For each feature combination w, determine nearest neighbour of the query • Record for each nearest neighbour the proportion of w for which it came top as well as the average of these w • NN for nearest neighbour, k for the dimensionality of the weight space(= length of the weight vector w)

  14. F3 F2 F1 The first retrieval step: finding the NNk 1 1 1

  15. The first retrieval step: finding the NNk

  16. The first retrieval step: finding the NNk • With fixed number of grid points, time complexity is exponential in the number of features (k) • Useful theorem: if for any two weight sets w1 and w2 that differ only in two components the top ranked image is the same, then this image will be top ranked for all linear combinations of w1 and w2

  17. The first retrieval step: finding the NNk

  18. Visualization of NNk

  19. The second retrieval step: • Retrieve with each weight set in turn • Merge ranked lists

  20. Performance evaluation • Comparison of NNk with two post-retrieval methods for weight-update1. Our own: minimize 2. Rui’s method (Rui et al., 2002)

  21. Performance evaluation • Corel Gallery 380,000 Package • Given a subset of images, treat each image as a query in turn and retrieve from the rest • For RF: retrieve with equal weight sets, gather relevance data and retrieve with new weight set • For NNk: determine NNk, gather relevance data and retrieve with new weight sets • Determine MAP after second retrieval

  22. Performance evaluation

  23. NNk networks

  24. Network Construction • Vertices represent images • An arc is established between two images X and Y, iff there exist at least one instantiation of the weight vector w, for which Y is the nearest neighbour of X • Record for each nearest neighbour the proportion of w, for which it came top-> edge weight, measure of similarity • Storage: for each image, its nearest neighbours and their frequencies

  25. Rationale • exposure of semantic richness • user decides which image meaning is the correct one • network precomputed -> interactive • supports search without query formulation

  26. Graph topology: small world properties • small average distance between any two vertices (three nodes for 32000 images) • high clustering coefficient: an image‘s neighbours are likely to be neighbours themselves

  27. Graph topology: scale-freeness • Degree distribution follows power-law

  28. Image Browsing • Initial display: retrieval result using search-by-example OR cluster display using Markov-Chain Clustering (MCL) technique (van Dongen, 2000) • Clicking on an image displays all adjacent vertices in the network • Distance inversely proportional to edge weight

  29. Evaluation of NNk networks: TRECVID2003 • search collection: 32000 keyframes from news videos • 24 topics: example images + text • Four system variants:Search + Relevance Feedback + BrowsingSearch + Relevance FeedbackSearch + BrowsingBrowsing

  30. Future work

  31. User interaction • What can we infer from the history of images in the user‘s search path?The location of the target? Changing information needs?

  32. Network construction and analysis • Hierarchical sampling of points in weight space • Incremental update of network while preserving small-world properties • Optimal network structures

  33. Thanks

More Related