1 / 41

Advances in Image Retrieval: Bridging Text and Visual Data through Integrated Learning

This study explores innovative methodologies in image retrieval, focusing on the integration of text and visual data. It discusses Bag-of-Words models, visual words' spatial locations, and part-based models, alongside discriminative methods for segmentation and recognition. A significant emphasis is placed on joint learning approaches that enhance large-scale retrieval, utilizing noisy labels to improve search algorithms like Google's Image Search. Key findings include optimized image ranking techniques, advanced data collection methods, and insights from seminal works within the field presented at ICCV and CVPR conferences.

iliana
Télécharger la présentation

Advances in Image Retrieval: Bridging Text and Visual Data through Integrated Learning

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Agenda • Introduction • Bag-of-words models • Visual words with spatial location • Part-based models • Discriminative methods • Segmentation and recognition • Recognition-based image retrieval • Datasets & Conclusions

  2. Retrieval domains Internet image search Video search for people/objects Searching home photo collections

  3. Learning from Internet Image Search • Joint learning of text and images • Large scale retrieval

  4. Noisy labels

  5. Improving Google’s Image Search • Fergus, Fei-Fei, Perona, Zisserman, ICCV 2005 • Variant of pLSA that includes spatial information

  6. Re-ranking result: Motorbike Topics in model Automatically chosen topic

  7. Animals on the Web Berg and Forsyth, CVPR 2006 Gather images using text search Use LDA to discover “good” images using features based on nearby text, shape, color

  8. Boostrapping of Image Search Schroff, Zisserman, Criminisi, Harvesting Image Databases from the Web, ICCV 2007 Images returned with PENGUIN query Final rankingusing SVM Removal of drawings and abstract images Naives Bayes ranking using noisy metadata Train SVM……. 4 2

  9. OPTIMOL Li, Wang, Fei-Fei CVPR 07

  10. Learning from Internet Image Search • Joint learning of text and images • Large scale retrieval

  11. Matching Words and Pictures • Barnard, Duygulu, de Freitas, Forsyth, Blei, Jordan, JMLR 2003

  12. Text to Images

  13. Images to text • Use Blobworld or nCuts to segments images into regions • Need to deduce labels attached to each image

  14. Images to text result

  15. Names and Faces in the News Berg, Berg, Edwards, Maire, White, Teh, Learned-Miller, Forsyth. CVPR 2004 Collected 500,000 images and text captions from Yahoo! News Find faces (standard face detector), rectify them to same pose. Perform Kernel PCA and Linear Discriminant Analysis (LDA). Extract names from text. Cluster faces, with each name corresponding to a cluster. Use language model to refine results

  16. Initial clusters

  17. Clusters refined withlanguage model

  18. Learning from Internet Image Search • Joint learning of text and images • Large scale retrieval

  19. Vocabulary tree Nistér & Stewénius CVPR 2006. KD-tree in descriptor space Inverse lookup of features Specific object recognition Not category-level

  20. Slide from D. Nister

  21. Slide from D. Nister

  22. Slide from D. Nister

  23. Slide from D. Nister

  24. Slide from D. Nister

  25. Slide from D. Nister

  26. Slide from D. Nister

  27. Slide from D. Nister

  28. Slide from D. Nister

  29. Slide from D. Nister

  30. Slide from D. Nister

  31. Slide from D. Nister

  32. Slide from D. Nister

  33. Slide from D. Nister

  34. Slide from D. Nister

  35. Slide from D. Nister

  36. Slide from D. Nister

  37. Slide from D. Nister

  38. Slide from D. Nister

  39. Pyramid Match Hashing • Grauman & Darell, CVPR 2007 • Combines Pyramid Match Kernel (efficient computation of correspondences between two set of vectors) with Locality Sensitive Hashing (LSH) [Indyk & Motwani 98] • Allows matching of the set of features in a query image to sets of features in other images in time that is sublinear in # images • Theoretical guarantees

  40. Semantic Hashing • Salakhutdinov and Hinton, SIGIR 2007 • Torralba, Fergus, Weiss, CVPR 2008 • Map images tocompact binary codes • Hash codes for fastlookup

More Related