Connections of Image and Text in Digital and Handwritten Documents

1. Connections of Image and Text in Digital and Handwritten Documents Ágnes Veszelszki 2nd Visual Learning Conference, BudapestDec. 2 2011

2. Overview • digitalcommunication and images • connections between image and text • priority of images over words • images—in themselves—are not unambiguous • images and words in interaction • emoticons • in digital communications • in handwritten texts(classroom notes, private letters)

3. 1. Pictorial turn • history of philosophy ~ a series of turns(Rorty) • classical and medieval philosophy: things • 17-19th centuryphilosophy: ideas • 20th century philosophy: words (linguistic turn) • + pictorial turn(W. J. T. Mitchell) • linguistic turn • everything is a text? • pictorial turn(Mitchell 1992)/iconic turn (Boehm 1994) • everything is an image?

4. 2. Digital age and visuality • basic forms of writing • handwriting - Manu-Skript • printing - Typo-Skript • computer writing - Compu-Skript(Dürscheid 2006) • handwriting–typewriting–digital writing • visuality: stronger connection between handwriting and digital writing • digital media: creating, accessing,modifying images • impact on our communication customs + on our way of thinking

5. 3. Connections between image and text • three forms(summed up by Nyíri 2011): • 1. images take priority over words • 2. images (in themselves) are not unambiguous • 3. images and words are in interaction, they have the same level of importance

6. 3.1. Images taking priority • images as self-evident natural signs (Gombrich 1978) • from an evolutionary point of view:images takeprecedence over words(Genova 1993; Nyíri 2011) • images can convey information which cannot be coded verbally (Gombrich 1972) • examples: • profile pictures shared on social websites (Veszelszki 2011) • funny pictures

7. 3.2. Images (in themselves) are not unambiguous • images are not obvious without words (Gombrich1969, Nyíri 2011: 5) • verbal clues (title, label, cutline, photo caption) • example: dog on a house wall in Pompeii

8. 3.2. Images are not unambiguous Examples: image on a social website ‘[Tibetan] nomads hang their animals born with developmental disorders on a tree as peace offerings, in order to return them to evil spirits’ • tabloid website • presenter of a talent contest + Monchhichi doll • ‘big feathery Monchhichi’

9. 3.3. Image and text in interaction, having the same level of importance • neither ‘mental images, nor words constitute independent processes’ (Paivio 1971, Nyíri 2011) • examples: • (web based) comics • picture poems • word clouds • demotivatingpictures • emoticons

10. 4. Emoticons • graphic signs/symbols, charactercombinations • text-image conglomerates • mutually contextualise and clarify each other (Schmitz 2003) • avoid the misunderstanding of the text written (Bódi–Veszelszki 2006, Keszler 2004, Runkehl 2005, Veszelszki 2005b) • source of empiric data on emoticons: • non-representative online survey, 2010 (Veszelszki 2010)

11. 4.1. Types of emoticons • two types • ASCII characters, punctuation marks • main types: :-) :-( • Japanese emoticons: emojis ‘facial signs’ • smiling face(^^) • two eyes ^^ • blushing cheeks (٭^^٭) • graphic symbols • standing (static) • moving (dynamic) emoticons

12. 4.2. Meaning of emoticons • Which emoticons are used most in the course of digital communication? • :) (mentioned339 times/635): positive attitude (happiness, easiness, joking, kindness, reaction to kindness) • :( :-( (296 times): sadness, reaction to bad news, compassion • :D :-D (275times): laughter • :P:-P (113 times): tongue sticking out, playful attitude, lack of intent to comment the previous remark • XD xD(100 times): open-mouthed laughter • :O (100 times): amazement, surprise, shock • <3(L)love, affection

13. 4.2. Meaning of emoticons • broader/alternative meaning > general meaning • theylack any specific meaning • emoticonsreceive their meaning only in relation to the context and the cotext • current meaning is only revealed in the context • „An image sometimes can tell more talkative than a thousand words—but […] what the meaning conveyed by an isolated static image will only be understandable in the context of words, described by words” (Nyíri 2000)

14. 4.3. Functions of emoticons • expression of emotions, supplementsuprasegmental and extralinguinstic features, visual content • e.g.: hahaha, this is a good one! :))) • speed up communication • break the dominance of text, make the text more spectacular • a source of humour (Bódi–Veszelszki 2006) • alter the meaning of the text • e.g.: U R very clever :S • reducing the possibility of misunderstanding • fill up pauses in simultaneous communication (e.g. on chat) • maintaining the social relationship

15. 4.4. Emoticons and text types • What types of texts permit and what types of texts prohibit the use of emoticons? • 90%: chat, email, sms or social websites • 50% <: non-digital text forms (handwritteninformal letters, handwritten messages, notes, classroom paper messages, postcards): emoticons frequently • even in speech (e.g. laughter by saying xd) • 10% don’tavoid using smileys in official situations (e.g. in formal letters or test papers) • informal situations • but: survey vs. reality!

16. 5. Summaryand implicationsforfutureresearch • connections between emoticon use, printed advertisement and official communication • dialogue letter(handwritten private letter written in class)

17. Thankyouforyourattention! Thx 4 UR attention! :)