Gesture and language: Iconicity and viewpoint

1. Gesture and language:Iconicity and viewpoint Eve Sweetser Dept. of Linguistics University of California, Berkeley sweetser@berkeley.edu CogSci Faculty Retreat, Dec. 7, 2007

2. Gesture is a universal People gesture when they speak. In every culture. Gesture is minutely co-timed with speech production in what is clearly a common neural production package. (McNeill, Hand and Mind) What varies: Size of gesture space Precise pattern of co-timing of speech and gesture (This takes a lot of learning! Cf. McNeill, Goldin-Meadow) Conventional “emblems”

3. How much of a gesture is “meaningful” - and to whom? Speakers gesture in absence of physical interlocutor seeing the gestures. (on the telephone, for example) Speakers’ lexical access is impeded by impeding gesture - especially lexical access to spatial vocabulary. Speakers gesture LESS without physical interlocutor: in particular, INTERACTIVE gestures are diminished. (Bavelas et al.) (Though nods and head-shakes may persist!) “Content” gestures are still there. Speakers alter gestural patterns to take addressees into account (Özyurek 2000).

4. Gesture structure Gestures have prosodic structure like speech. A linguistic PHRASE often coincides with a single GESTURE, therefore called a gestural “phrase” by some (Kendon). A linguistic phrase has various stages including: a preparatory phase possibly a pre-stroke “hold” a STROKE (the major motion phase of a gesture, often temporally associated with some specific constituent of the linguistic phrase, in English often the VERB) a post-stroke hold retraction

5. Intermodal meaning overlap Gestural “phrases” are temporally associated with linguistic forms whose meaning is related to theirs. Gesture can add information not present in the linguistic form. It can also give “interactional” meaning about how to take the linguistic content. And it can contradict (and win out over) linguistic information in the listener/viewer’s interpretation. HOW and WHY does gesture accompany language this way? Attentional focus is one aspect of the interaction - Mischa will talk about that.

6. Things to notice in “Listen” The use of the INTER-SPEAKER space: the floor is gained by reaching gesturally into the space between the two speakers’ spaces - and even into the interlocutor’s space. The use of the “unclaimed” adjacent space: the |cupboard| could NOT just as well be in between the interlocutors, but the |stacking| can be done in the speaker’s own gesture space.

7. Listen

8. Listen transcript S1: …the underside. S2: OK, this is what would happen. S1: You’d stack dishes. S2: Listen. We did stack dishes buut I’d like reach in to get a plate to get ready to eat and there’d be like [grea- S1: [laughs] S2: - there’d be like grease on the bottom] S1: yeah S2: And I’d be like…

9. Listen transcript S1 mimes washing the underside of the dishes as she says “the underside,” then shapes a stack of plates (or makes a stacking gesture) as she says “you’d stack dishes.” S2 is meanwhile trying to break into S1’s high-involvement feedback, which is keeping her from the floor. She first says, “OK, this is what would happen,” with hands shaping a new topic in her own gesture space, which return to rest as she fails to get the floor. She then tries again with “Listen” - accompanying her attempt with three left-hand D-points (on “listen”, “did” and “buut”), which reach well out of her own space into the shared interactional gesture space. She gains the floor.

10. Listen

11. Iconicity ICONICITY: a representation is iconic if the form in some way resembles the meaning. Spoken language examples: (1) phonosymbolism: meow, crash, pop. (2) She talked on and on and on. But (cf. Taub, Language from the Body) the VISUAL-GESTURAL medium allows for a greater variety of effective iconic mechanisms than the auditory one. Signed languages therefore share this extra-iconic character with gesture, although they are conventional in ways that gesture is not.

12. Visual-gestural iconicity Iconic mappings (thanks to S. Taub) ASL TREE iconically represents a schematic concept of a tree. DH FOREARM = TRUNK DH HAND & FINGERS = BRANCHES (note: doesn’t mean a tree with 5 branches!) NDH FOREARM = GROUND Upper arms, signer’s head & trunk: NOT MAPPED

13. How conventional is gesture? There are cultural and crosscultural regularities in how people gesture about both concrete and abstract domains. (Metaphor and iconicity can be conventional, we know; and culture-specific. HKSL “Tree” vs. ASL “Tree.” Local “catchments” (cf. McNeill and Duncan, in McNeill (2000)) regularly arise in interaction. Just as local repeated phrasings do. Reduced forms of these catchments still carry meaning. (cf. Also LeBaron and Streeck. McNeill’s Snow White experiment.

14. Iconic gestures can then be interpreted metaphorically Gesture forwards - does that mean physically ahead of me, or in the future? Rotation of hand - does that mean some physical object is rotating, or does it show repetitive or ongoing aspect of an action? Hand up, palm out - is the speaker trying to prevent the addressee from approaching her (or fend off a projectile), or is she metaphorically “fending off” questions Cf. Parrill and Sweetser 2004.

15. Metaphor All languages use METAPHOR, because all cultures have metaphoric cognitive patterns. One way of looking at metaphor: understanding more abstract things in terms of more concrete things. TIME IS SPACE FUTURE IS IN FRONT OF EGO PAST IS BEHIND EGO last year is behind us; look ahead to next year Source and target domains

16. Metaphoric gestures are iconic for the SOURCE domain of the metaphoric mapping. Gesture forwards for future: FUTURE IS AHEAD. Hand up, palm out to forestall questions: IDEAS ARE OBJECTS, COMMUNICATION IS OBJECT EXCHANGE… Palm-up “offering” hand, meaning “it’s obvious”, or “now I’m sure you see”: Again, COMMUNICATION IS OBJECT EXCHANGE.

17. “Put you away” iconic mappings 2 B hands, palms facing each other (thumb side up) = The hands of someone putting something away. Motion of the gesturer’s hands = Motion of the hands putting something away. Space between gesturer’s hands = object being put away (box?) (Invisible surrogate! Moves when hands do.)

18. “Put you away” metaphoric mappings CONVICTED CRIMINAL = AN OBJECT (a box?) PUTTING CRIMINAL IN PRISON = PUTTING THE OBJECT AWAY AGENT PUTTING CRIMINAL IN PRISON = AGENT PUTTING THE OBJECT AWAY RESULTING STATUS OF CRIMINAL (STUCK IN ONE PLACE, MONITORED) = RESULTING STATUS OF OBJECT (WHEN PUT AWAY, WE KNOW WHERE TO FIND IT AND WE DON’T THINK IT WILL GET LOST).

19. Sign language examples TOMORROW, YESTERDAY KNOW, IDEA VOTE, TEABAG DIPLOMA DRIVE, FLY(PLANE) NEPHEW, NIECE, COUSIN

20. Things to note in “concepts” Setting up of two spatial areas, |concepts| and |forms|. Mapping between them. The encircling gestures for “framework.”

21. Concepts

22. Transcript of “concepts” Clip name: concepts map onto the world (lecture by Mark Johnson) ...have fixed definitions and they map onto the world...um... and that knowledge consists in...framing a set of concepts that neatly map onto states of affairs in the world whether those states of affairs have to do with morality or politics or um...or um...quantum physics or whatever.

23. Gesture transpript of “concepts” Clip name: concepts map onto the world (lecture by Mark Johnson) Two B hands to Left, delineate a definition/concept space. They then move to the Right and delineate a World space. Delineation of a globe-shaped central space is |framing a set of concepts|. Moving hands from one side to the other is |mapping| Handwaving shows “it doesn’t matter” at the discourse level (cf. whatever)

24. Concepts

25. Viewpoint, language and body Language is basically, intrinsically viewpointed. Cognition is basically, intrinsically viewpointed. (BECAUSE) The body is basically, intrinsically viewpointed.

26. Implications Cognition could not be genuinely independent of bodily experience, and language could not be independent of (embodied) cognition. The surprising thing would be if we did NOT exploit our constant use of irrealis space understandings for other less obviously or immediately “functional” purposes. It would also be really surprising if fictional characters and situations lacked viewpoint.

27. Linguistic viewpoint Message in a bottle: Meet me here tomorrow. Deictic marking (e.g., here/there, this/that) is pervasive in human language. In a physical scene it marks speaker’s physical viewpoint. BUT: cf. Rubba 1986, or Hanks 1990: a distal deictic can just as well mean social non-identification. This cooking-fire can mean the one I cook on, while that cooking fire can mean the one I don’t get to cook on. This part of town can mean my part of town, and that kind of neighborhood can mean the kind I lack an ethnic affiliation with.

28. Linguistic viewpoint 2 First, second and third person seem to be linguistic universals: I, You, Other = Speaker, Addressee, Third Party. No big surprise, since in actual communication, these distinctions are inevitable. BUT (Rubba 1986) there are differences between so-called “impersonal” uses of English 2nd-person you and 3rd-person one or they. You shows more identification of the speaker with the referent, even though all the referents are third-person.

29. All the different ways that content is presented and construed differently depending on (among other things!): Where the Speaker and Addressee are assumed to be, and what they be able to see, be able to reach, etc. (The Real Space.) here, there, this, that, next door, …. When S and A are assumed to be: now, then, tomorrow, last year… What an imagined participant can see, reach, etc., from an imagined location in some imagined space. What the Speaker and Addressee are assumed to know, think, presuppose, and be able to calculate mentally about whatever mental space is involved. The/a, if/when/since, choice of formal/informal pronouns, presuppositional verbs like stop,... Linguistic viewpoint markers 1

30. What the Speaker and Addressee feel about the contents of the relevant spaces - how they evaluate them affectively, culturally, etc. Thrifty/stingy, maybe, hopefully,…. And what imagined participants know, think, presuppose, calculate, feel, etc. about relevant spaces. (And more, including possession, social identification and differentiation, ….) In short, language seems affected by just about anything about the way that a particular individual’s mental space construal is specific to that individual’s cognitive and perceptual access. Linguistic viewpoint markers 2

31. What is “viewpoint”? Literal origo of visual access and perspective. Social “viewpoint” Literary “viewpoint” Cognitive “viewpoint” We co-experience: Our physical visual perspective on events Our self-location and definition of peripersonal space Our tactile and other sensory access to the situation Our cognitive assessment of the situation Our emotional reaction to the situation Video systems may give us multiple simultaneous visual perspectives on the same event; but normally all we get is one: our own.

32. Front-back: Asymmetric physical access, manual affordances Asymmetric visual access Asymmetric movement affordances Up-down: We have no experience of life outside an asymmetric gravitic field. Motion, vision, etc are all affected by this. Left-right: Dominant/non-dominant hand; asymmetric manual affordances Relative spatial languages vs. absolute spatial languages. Inherent and transferred asymmetry of non-human entities. The asymmetric Self

33. Asymmetries in visual access correlate with asymmetries in: Informational access attendant on visual access Physical access to object manipulation Motion affordances This complex set of correlations with located visual viewpoint also correlate with evaluative differences in assessing the situation. The kid with the plate of cookies in front of her, as opposed to the one who doesn’t, not only has different visual, tactile, etc., access, but a reason to plan differently, make different inferences, and experience the situation differently from an emotional viewpoint. What is viewpoint, cont.

34. What the evidence seems to show is that children from the start react differently to humans vs. nonhumans and animates vs. inanimates, even though they only gradually develop the adult concepts of human-ness and animacy. Relatively early shared attention loci, attention to direction of caregiver’s eye-gaze. Even at early stages of language-learning, children show considerable interactional ability to cope with the fact that caregivers may disapprove of their actions or try to thwart them, and versatile mechanisms for getting approval. They also request information and transmit it. Why these belong together

35. Moreover, meta-awareness of even some of the basic perceptual aspects of differentiation develops slowly. Small children clearly understand that they can’t see or know everything their parents see or know - they ask questions expecting the caregiver to know more than they do, and they ask to be picked up to get a better view, as they know the grownup is getting one. BUT they tend to think that interlocutors (esp. adults??) not only know but actually see everything they do. They point and say “this” and “that” when speaking on the telephone, or to a caregiver who can’t see the object in question. Levels of theory of mind

36. Levels of theory of mind 2 A late stage in all this is the acquisition of a conscious meta-awareness that other people have MINDS whose STATES may be DIFFERENT from theirs: other people may know or believe different things from what they know and believe (Wimmer, Perner, Tomasello,….), and people may feel differently about the same stimuli. (E.g. celery and goldfish crackers - cf. Gopnik.)

37. Incomplete shared viewpoint The ability to “put yourself in someone else’s position” cognitively and emotionally is one that adults never fully learn (if they did, one supposes they would essentially have “out-of-body experiences” in other people’s situations).

38. BUT humans can’t unlearn or do without such viewpoint sharing. We can’t help having physical and emotional responses to film images of humans involved in eating, crying, laughing, kissing, or hitting each other. Reading a newspaper story about strangers who have lost their jobs, or can’t get ex-husbands to pay child support, or have overcome odds to win a sports event - these things, to a greater or lesser degree, put us in sympathy with the participants’ viewpoints. Inasmuch as we are considerate of others in daily life, much of our consideration derives not just from “following the rules” but from being able to imagine “how the other person would feel” if we did the opposite. Incomplete shared viewpoint 2

39. All primates appear to have mirror neuron circuits which (at least for certain aspects of physical interaction and spatial relations) are activated both by the Self’s motions of hand, mouth, foot and the Self’s peripersonal space, and by an observed primate’s actions and spatial relations. The ability to maintain local coherence between differing viewpoints apparently follows from our physical perceptions of motion, space, etc, including mirror neurons; NOTE, we do not have trouble tracking which things are in our interlocutor’s field of vision, even though it may be very different from ours. This is a plausible basis on which to build later higher-level awareness of different viewpoints as parts of a coherent larger scene. Local and global coherence

40. Add to this a basic experience of INTERACTION with another human with viewpoint (babies interact with care-givers actively from the start). A contrast between SPEAKER and ADDRESSEE - or more generally between a communicatively expressive agent and the intended interpreting observer - is thus another deeply entrenched experiential correlation. From the start, we experience ourselves in BOTH of these two roles. And, thanks to our understanding that everyone has Viewpoint, Viewpoint blends freely with either the Speaker or the Hearer role in the Speaker-Hearer contrast. Other people’s viewpoints

41. (1) Have the experiential correlations involved in having a Viewpoint “from the inside”. (2) Project that kind of experiential correlation of Viewpoint onto other people, assuming that they also have that kind of perspectival experience of the world. (3) meta-navigate this system - be able to go back and forth between representing one viewpoint and another, and know when/whether our language requires or allows particular viewpoint representations of situations. (May S, or must S, say “I’d love to come to your party” rather than using go?) (4) also have some natural representation of global, less viewpointed knowledge (e.g. spatial knowledge) (5) be able to represent many situations linguistically in “global” as well as in “participant” viewpoints. Adult cognition requires that speakers:

42. Adult cognition requires that speakersalso: (6) Project much of their systematic spatiomotor “viewpoint” structure onto other understanding of less concrete domains such as Time, social relations, cognition, selfhood. (7) Maintain more and less perspectival models of these domains as well, and know when to use language which shows the appropriate perspectival construal. (8) Be able to TAKE APART viewpoint blends, maintaining personal “I”ness and “you” ness separate from (for example) a proximal/distal structure, and use linguistic forms appropriate to this dissection.

44. 8

45. Most languages seem to have at least a two-way distinction between • this and that, here and there. • Some have more complex three-way distinctions (here, there, yonder). • Basis in Speaker’s visual field and manual access field: Here = within S’s manual access range There = within the visual field but outside manual access range • Yonder = outside both visual field and manual access field. • In a two-term system, the manual access field • would be the central sense of here, the area outside the manual • access and vision fields would be unmistakably there, and the area • of the visual field beyond manual access would be negotiable, • depending on what objects were being contrasted. Experiential basis of deixis

46. (2) Basis in the Speaker/Hearer contrast: Here = near S There = near H Yon = away from both. (Once again, in a two-term system, things get fuzzy.) This is not necessarily in opposition to the analysis in terms of S’s different fields of access; we might expect that in a prototypical communicative exchange, S and H will be within visual range of each other, and that there may very possibly be more overlap between their visual fields than between their fields of manual access. So “near H” (or “nearer to H than to S”) might well also refer to a location beyond S’s manual access, but inside S’s visual field. Experiential basis of deixis, 2

47. Problems with the spatial view of deixis (1) It doesn’t, on its own, explain systematic extensions to time, or the independent system of temporal deixis; more on this soon! (2) It doesn’t explain all the SOCIAL, non-spatial uses of deixis. (cf. Rubba 1996, Hanks Referential Practice). (3) It doesn’t explain language-specific spatial uses (is the “this” term or the “that” term, the “come” verb or the “go” verb, the unmarked member of the pair, for example?) A combination of problems (2) and (3) can be noted in French: Professeur Jones n’est pas ici. (She works at UCLA, not at UCB)) Professeur Jones n’est pas là. (She is not at her desk just now)

48. Metaphoric viewpoint spaces Social uses of here/there, this/that can readily be seen as FURTHER blends, between physical space and social space. This should be seen as coherent with the very general metaphorical spatialization of our concepts of Self and social structure. SOCIAL RELATIONSHIP IS PHYSICAL CLOSENESS SOCIAL ALIENATION IS PHYSICAL DISTANCE

49. The speaker's body is one of our most basic landmarks for understanding whatever she says. It is because bodily viewpoint infuses our cognitive and interactional structure that deixis and perspective are so pervasively manifested in language. Yet actual here-and-now bodily viewpoint is very flexibly displaced to represent other imagined ones; we don't really know the meaning of here or the reference of a pointing gesture unless we know whether, for example, the speaker is enacting some irrealis situation . Such displacement phenomena are present equally in language and gesture, and in spoken and signed languages. They can usefully be analyzed as blends of the here-and-now deictic space with an imagined one. Displaced deixis

50. The visual/gestural modality uses S’s body to represent (among other things!): - itself, at other times and places - other human bodies and animal bodies Every representation of a body brings with it the physical asymmetries of affordance and sensory access which are characteristic of bodies: in other words, VIEWPOINT can represent VIEWPOINT DEICTIC CENTER can represent DEICTIC CENTER. BODILY AFFORDANCES can represent BODILY AFFORDANCES This is crucial to the ways in which gesture can represent social and abstract concepts. Real-space viewpoint blends