Some ideas around PIADA: (Picture Indexing: Affect, Description and Availability)

1. Some ideas around PIADA: (Picture Indexing: Affect, Description and Availability) Diana Santos

2. PIADA in a nutshell • Context: making sense of images • High-level concerns: • Purpose • Interaction with text/context • Cultural factors • Jokes, irony, metaphor, affect, humour • Practical setup: Web archive, image repositories, Wikimedia • Expected outcomes • Picture ontologies (as well as integration with other NLP ontologies) • Reasoning with images • Cross-cultural understanding of image differences

3. Presentation outlook • A short description of the main ideas of PIADA • 5 things not yet solved • How to go about it? Data; 3 kinds of reasoning • SINTEF’s LEG (Language Engineering Group) • PIADA in a Norwegian context

4. 1. Find images with feelings • Most image descriptions are about “objective” features like who (names), where (places), dates, colours and objects • Asterix sad – no way, and we know there exist these pictures • Try Hillary happy and Hillary angry and you get mostly the same pictures: those of Hillary Clinton together with texts that may report whatever about happiness or sadness of her competitors or fans

5. 2. Find culturally implicit information • Japanese people in the underground (tube): most probably posted by Japanese people, not with captions saying that people are Japanese... • Try Google images and give up • Man reading. This can be a good enough caption in an European museological context, but certainly not in an Asian, or African context

6. 3. Find synergy between images and text • why this caption or this illustration? why do they work together? creative use of pictures in text: or: how many of these images of Sócrates are jokes? • from the obvious: a happy baby in a diaper’s advertisement to the provocative or very very subtle humour (was it?)

7. 4. Find the picture’s purpose http://staff.science.uva.nl/~marx/ • Why is this picture here? • Reasoning about a multimedia world... finding the cause for humour and the important details • Vi roser Anne! How my students end up....

8. 5. Help discuss or describe pictures for different purposes Properties of pictures are often mentioned in some contexts • (didactical, antropological, documentary, scientific) to focus on particular details: see the character behind Jesus, see the tree on the left, note the tool on his hand, look at the back wings, at the dark clouds, at the tumor ... • (police or law courts) to explain why the picture was taken: revolver under the table, near the corpse • (artistic setting): in sunlight, in rainy wather, with a special lins...

9. In a nutshell, we believe that • there are many aspects of picture description and reasoning that have not received any or enough attention, namely • emotions • creativity and humour • crosscultural differences • intertextuality with pictures • a natural language processing angle is the right way to attack them

10. Real applications • Significantly enhancing image bank providers activity • Helping professionals that need images and text • teachers • museum staff • encyclopedia authors (the Wikipedia community) • other multimedia content providers: for games, educational CDs, textbooks • advertisers • historians and biographers • Common multilingual image search

11. How to go about it? Data • Image collections to study • Image ontologies and folksonomies available • “Text and image” collections • Wikipedia • Web pages (Internet archive) • Special sites and or multimedia products (guided tours) • Image search logs • Elicitation collections: sets of stories about image search • Game results: eliciting similarities or associations among images

12. How to go about it? Reasoning 1 • If you want to find a picture of a strong healthy man, or of a genius, you probably find an instance of a person that illustrates these qualities (such as Johny Weissmuller or Einstein and look for them) • If you want to find a picture of Asterix angry, you can look for more “objective” descriptions such as Asterix shouting or Asterix beating • If you want to illustrate the property black you probably look for concepts where black is stereotypical, such as coffee...

13. How to go about it? Reasoning 2 • You need to know the context of the annotation or text to know what should be implicit and what should be probably commented out • searches in .pt for Sócrates are probably about the prime-minister but in .no, after the philosopher, comes the Brazilian football player  • the stereotypical image for man or woman is obviously different depending on the gender of the beholder, no matter sexual orientation, and the same for places and cultures • a “typical” restaurant (and its food) varies widely • pictures captioned Lisbon (or Dublin, or Helsinki) are most often by tourists. People who live in Lisbon give the precise names of what they take the picture of, or don’t even bother to specify location

14. How to go about it? Reasoning 3 • Why are the images chosen? • What is the kind of connection? • What is their import? • Which other associations -- interpictuality -- they bring? • Why can they be considered offensive or funny? • Do they feel old-fashioned? Do they feel modern?

15. Language engineering at SINTEF • Question answering • Ontologies • Geographical reasoning • Contrastive studies • Information extraction • Corpus search We believe all these pieces will help us to address the image search and indexing issue.

16. PIADA in a Norwegian context We want to develop specific knowledge on images in Norwegian • The vocabulary of images and image search in Norway • Demo collections • Study of user behaviour: what do people ask for, what do they want to see? • Picture reasoning ontologies • We hope that by cooperating with commercial actors we will do something useful not only for research purposes

17. Specific proposal • brukerstyrte innovasjonsprosjekter (BIP) OR • kompetanseprosjekter med brukermedvirkning (KMB) • Scanpix Norge as prime contractor • SINTEF writes most of the proposal • ABM-utvikling is also involved • Other Norwegian actors also contacted