1 / 17

Associating Relevant Photos to Georeferenced Textual Documents through Rank Aggregation

Associating Relevant Photos to Georeferenced Textual Documents through Rank Aggregation. Rui Candeias and Bruno Martins Instituto Superior Técnico : INESC-ID. ICSW Workshop Terra Cognita 2011 Bonn , Germany. Introduction. The illustration problem , a.k.a. Cross Media Retrieval :

zody
Télécharger la présentation

Associating Relevant Photos to Georeferenced Textual Documents through Rank Aggregation

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Associating Relevant Photos to GeoreferencedTextual Documents through Rank Aggregation Rui Candeiasand Bruno Martins Instituto Superior Técnico : INESC-ID ICSW Workshop Terra Cognita 2011 Bonn, Germany

  2. Introduction • The illustration problem, a.k.a. Cross Media Retrieval: • Given a textual document (e.g., a travelogue) as the query • Find relevant images (e.g., photos from Flickr) to illustrate the text • Many practical applications • Illustrating travelogues with landmark photos • Very challenging problem • Semantic gap between photos and textual documents • Vocabulary mismatch between document terms and photo tags • Proposal : Geographically-aware cross media retrieval!

  3. Related Work • Geographic Information Retrieval • Resolving place references in textual documents • Retrieving documents through geospatial similarity • Multimedia and Cross Media Retrieval • Explore descriptions (e.g., tags) associated to photos • Associating Photos to Travelogues [Lu et al., 2010] • Probabilistic topic models to avoid vocabulary mismatch

  4. The Proposed Method • Resolve place references in the document • Collect nearby georeferencedphotos from Flickr • Select the best photos to associate to the document • Compute multiple relevance estimators • Combine the estimators through rank aggregation • Select the top-ranked photo(s)

  5. Resolving Places and Collecting Photos Yahoo! Placemaker Delimiting placenames and associating them to the corresponding geospatial coordinates Flickr’s API Retriving metadata (e.g., tags) for photos taken close to a given pair of geospatial coordinates

  6. Resolving Places and Collecting Photos The Bonn Minster is one of Germany's oldest churches having been built between the 11th and 13th centuries. Since the 13th century, when the people of Bonn included the Minster in their city ’s coat of arms, it has been the emblem of the City of Bonn. The basilica of Bonn as we know it today was built on the site of the graves of the two martyrs Cassius and Florentius, the city’s patrons. The whole of its development is recorded, from its beginnings as a small place of worship in the late Roman period to its becoming the first large church complex in the Rhineland, and later a significant example of medieval Rhenish church architecture.

  7. Resolving Places and Collecting Photos The Bonn Minster is one of Germany's oldest churches having been built between the 11th and 13th centuries. Since the 13th century, when the people of Bonn included the Minster in their city ’s coat of arms, it has been the emblem of the City of Bonn. The basilica of Bonn as we know it today was built on the site of the graves of the two martyrs Cassius and Florentius, the city’s patrons. The whole of its development is recorded, from its beginnings as a small place of worship in the late Roman period to its becoming the first large church complex in the Rhineland, and later a significant example of medieval Rhenish church architecture. <extents> <center> <latitude>51.3346</latitude><longitude>1.31407</longitude> </center> <southWest> <latitude>38.7051</latitude><longitude>-91.5391</longitude> </southWest> <northEast> <latitude>55.0581</latitude><longitude>15.0421</longitude> </northEast> </extents> <placeDetails><placeId>2</placeId> <place> <woeId>640161</woeId> <type>Town</type> <name>Bonn, North Rhine-Westphalia, DE</name> <centroid> <latitude>50.7323</latitude> <longitude>7.10169</longitude> </centroid> </place> </placeDetails>

  8. Resolving Places and Collecting Photos The Bonn Minster is one of Germany's oldest churches having been built between the 11th and 13th centuries. Since the 13th century, when the people of Bonn included the Minster in their city ’s coat of arms, it has been the emblem of the City of Bonn. The basilica of Bonn as we know it today was built on the site of the graves of the two martyrs Cassius and Florentius, the city’s patrons. The whole of its development is recorded, from its beginnings as a small place of worship in the late Roman period to its becoming the first large church complex in the Rhineland, and later a significant example of medieval Rhenish church architecture. <extents> <center> <latitude>51.3346</latitude><longitude>1.31407</longitude> </center> <southWest> <latitude>38.7051</latitude><longitude>-91.5391</longitude> </southWest> <northEast> <latitude>55.0581</latitude><longitude>15.0421</longitude> </northEast> </extents> <placeDetails><placeId>2</placeId> <place> <woeId>640161</woeId> <type>Town</type> <name>Bonn, North Rhine-Westphalia, DE</name> <centroid> <latitude>50.7323</latitude> <longitude>7.10169</longitude> </centroid> </place> </placeDetails> <photoid="3882008927" dateuploaded="1251930164" isfavorite="0“ views="483“> <ownernsid="26021670@N00" username="Claude@Munich" realname="Claudia" location="Munich, Germany“/> <title>BonnMinster</title> <descriptio> The Bonn Minster is one of Germany's oldest churches having been built between the 11th and 13th centuries. … </description> <datesposted="1251930164" taken="2009-08-2621:07:05” lastupdate="1306931029" /> <comments>6</comments> <tags> <tagraw="Germany" author="26021670@N00”>germany</tag> <tagraw="Nordrhein-Westfalen" author="26021670@N00”>nordrheinwestfalen</tag> <tagraw="church" author="26021670@N00”>church</tag> <tagraw="BonnMinster"author="26021670@N00“>bonnminster</tag> <tagraw="Minster"author="26021670@N00“>minster</tag> </tags> <locationlatitude="50.733155" longitude="7.100451“ place_id="Uvujcu1XVrp6hVQ" woeid="640161"> <localityplace_id="Uvujcu1XVrp6hVQ" woeid="640161">Bonn</locality> <countyplace_id="MlysCMBQUL9JMVQ_Og" woeid="12597065">StadtkreisBonn</county> <regionplace_id="RoiqFqRTUb6tlYAO" woeid="2345487">NorthRhine-Westphalia</region> <countryplace_id="h7eZVDlTUb50Btij9Q" woeid="23424829">Germany</country> </location> </photo>

  9. The Relevance Estimators • Textual Similarity • Photos having words (title, tags) that occur in the text of the document are more likely to be relevant • Geospatial Proximity • Photos taken close to the places mentioned in the document are more likely to be relevant • Temporal Cohesion • Photos taken in the same semester as the temporal period discussed in the document are more likely to be relevant • Photo Importance and Interestingness • Photos having more visualizations or more comments should be more interesting, and thus also more likely to be relevant

  10. Computing The Estimators • Textual Similarity • Term Frequency x Inverse Document Frequency (TF-IDF) • Stopwords were first removed • Tags considered twice more important than title • Geospatial Proximity • Great circle distance • Since documents have multiple locations, we used the minimum and the average distances

  11. Score-Based Rank Aggregation • The relevance estimators are combined through score-based rank aggregation schemes: • Multiple scores are computed from the relevance estimators • Scores for each relevance estimator are normalized through min-max procedure • Final ranking is obtained by aggregating the normalized scores [Fox & Shaw, 1999?] • The CombSUMmethod • Multiple scores are summed • The CombMNZ method • Multiple scores are summed and multiplied by the number of non-zero scores

  12. Experimental Evaluation • Dataset with 450 georeferenced Flickr photos • Total of 50 photos for each of 9 popular tourist destinations (i.e., capitals) • Large textual descriptions, containing placenames, used as the queries • Photos described with tags, title, geospatial coordinates, ... • Experiments with different combinations of the proposed method • Text Similarity versus different combinations of relevance estimators • CombSUM versus CombMNZ method • Results evaluated with Precision@1 and Reciprocal Rank • Only one relevant photo per document

  13. The Obtained Results

  14. The Obtained Results

  15. Conclusions • We proposed and evaluated a novel geographically-aware cross media retrieval method • The method leverages on resolved place references to avoid the semantic gap between photos and texts • Combining relevance estimators leads to more accurate cross-media retrieval results • The CombMNZ and CombSUM rank aggregation methods are adequate to the task, obtaining similar results

  16. Future Work • Supervised Learning to Rank (L2R) • Experiments with many different L2R algorithms • Experiments made afterwards showed significantly better results • Many more relevance estimators • Topical similarity with basis on LattentDirichlet Allocation (LDA) model • Features from visual image clusters • Experiments made afterwards showed slight increase in result quality • Outlier removal • Remove outlier cities recognized by Yahoo! Placemaker • Improve the evaluation procedure • Still to be made...

  17. Thanks for your attention! rui.candeias@ist.utl.pt

More Related