Real time studies of comprehension and production in dialogue: insights from eye movements

1. Real time studies of comprehension and production in dialogue: insights from eye movements Michael K. Tanenhaus Sarah Brown-Schmidt University of Rochester

2. Overview • Part 1. Eye movements and spoken language • Sensitive time-locked response measure • Cohort effects • Point-of-disambiguation effects • Dynamically updated contextually rich referential domains • Part 2: Standard view or “emergent alignment” • Need real-time studies of non-scripted interactive dialogue • Part 3: Some feasibility studies • Referential domains, POD and alignment (Blocks) • Generating referential expressions: interplay between message and utterance formulation (Islands) • In progress study (Questions)

3. 1. Language unfolds over time. beaker • beetle, beacon, beak, beep… • Put the apple on the towel… 2. Processing is closely time-locked to the unfolding utterance. 3. Requires monitoring moment-by-moment processing.

4. Why eye movements? • Closely time-locked to attentional shifts • Natural, well-understood response measure • Ballistic response • Participants are unaware • Can be used with continuous speech • Low threshold, sensitive response • Plausible linking hypothesis • Closely time-locked to linguistic processing • (Cooper, 1974; Tanenhaus, SpiveyKnowlton, Eberhard & Sedivy, 1995; Griffin & Bock, 2000)

5. Allopenna, Magnuson & Tanenhaus (1998) Eye camera Scene camera Pick up the beaker

6. Fixation Proportions over Time Trials 1 2 3 4 5 Time 200 ms Proportion of fixations Time + Target = beaker Cohort = beetle Unrelated = carriage Look at the cross. Click on the beaker.

7. Fixation Proportions summed over an interval Trials 1 2 3 4 5 Proportion of fixations Proportion of fixations cohort Time unrelated +

8. Allopenna, Magnuson & Tanenhaus (1998) Eye camera Scene camera Pick up the beaker

9. Allopenna et al. Results 200 ms after coarticulatory information in vowel

10. How do addressees establish and update referential domains in real-time comprehension? Listeners (and speakers) dynamically update referential domains: (Altmann & Kamide, Chambers et al.; Eberhard et al.) Are domains influenced by: real-world properties of potential referents? goal-based constraints? interlocutor’s knowledge? If so, do these domains, affect “core” linguistic processes in comprehension, e.g., syntactic processing? Explore intersection of domain and processing of linguistic expression

11. Put the apple on the towel in the box E y e t r a c k i n g M i c r o p h o n e V i d e o d e v i c e D i s p l a y CPU VCR Action-based visual world paradigm (Tanenhaus et al., 1995) • Participants perform actions in response to instructions. • Language is “about” the world. • Speakers and addressees have well-defined behavioral goals • Fixations provide a window into processing

12. Schematic of situation and an empty martini glass Experimenter who occasionally makes a mistake, incorrectly describing the object she is putting down Confederate speaker asking for objects from a script Subject following instructions to give speaker what she requests

13. Point of Disambiguation Late Match Early Match Pick up the empty martini glass ...

14. Point of Disambiguation Late Match Early Match Pick up the empty martini glass ... Pick up the empty martini glass ...*

15. Point of Disambiguation Late Match Early Match Pick up the empty martini glass ... Pick up the empty martini glass ...* * Assumes listeners will immediately interpret scalar adjectives such as empty contrastively (i.e.,there should be one empty X and one not empty X). See Sedivy, Tanenhaus, Chambers & Carlson (1999) Cognition for supporting evidence.

16. Late Match Early Match Pick up the empty martini glass ... 1.00 Target 0.80 TargetO 0.60 Comp 0.40 CompO 0.20 Other 0.00 Time since onset of determiner (ms)

17. Late Match Early Match Pick up the empty martini glass ... 1.00 Target 0.80 TargetO 0.60 Comp 0.40 CompO 0.20 Other 0.00 Time since onset of determiner (ms) Pick up the empty martini glass ... 1.00 Target 0.80 TargetO 0.60 Comp 0.40 CompO 0.20 Other 0.00 Time since onset of determiner (ms)

18. Late Match Early Match Pick up the empty martini glass ... 1.00 Target 0.80 TargetO 0.60 Comp 0.40 CompO 0.20 Other 0.00 Time since onset of determiner (ms) Pick up the empty martini glass ... 1.00 Target 0.80 TargetO 0.60 Comp 0.40 CompO 0.20 Other 0.00 Time since onset of determiner (ms) POD effect as expected

19. Chambers et al., JML, 2002 “ P i c k u p t h e c u b e . N o w p u t i t i n t h e / a c a n . ” Is the referential domain dynamically updated taking into account context-specific properties and actions? Display - Chambers: Uniqueness Big Can (potential target) Small Can (potential target) Bowl (unique competitor) Cube (small OR big)

20. Chambers et al., JML, 2002 “ P i c k u p t h e c u b e . N o w p u t i t i n t h e / a c a n . ” Is the referential domain dynamically updated taking into account context-specific properties and actions? Display - Chambers: Uniqueness Big Can (potential target) Small Can (potential target) Bowl (unique competitor) Cube (small OR big)

21. Chambers et al., JML, 2002 “ P i c k u p t h e c u b e . N o w p u t i t i n t h e / a c a n . ” 7 0 0 6 0 0 5 0 0 4 0 0 3 0 0 2 0 0 1 0 0 Is the referential domain dynamically updated taking into account context-specific properties and actions? Display - Chambers: Uniqueness Big Can (potential target) Small Can . (potential target) Bowl (unique competitor) Cube (small OR big) E y e m o v e m e n t l a t e n c i e s t o t a r g e t Small cube T w o C o m p a t i b l e T a r g e t s m s D e f i n i t e I n d e f i n i t e

22. Chambers et al., JML, 2002 “ P i c k u p t h e c u b e . N o w p u t i t i n t h e / a c a n . ” 7 0 0 6 0 0 5 0 0 4 0 0 3 0 0 2 0 0 1 0 0 Context is dynamically updated taking into account context-specific properties and actions Display - Chambers: Uniqueness Big Can (potential target) Small Can . (potential target) Bowl (unique competitor) Cube (small OR big) Could you put it in/the/a can E y e m o v e m e n t l a t e n c i e s t o t a r g e t Big cube O n e C o m p a t i b l e T a r g e t Small cube T w o C o m p a t i b l e T a r g e t s m s Size matters But only when you have to do it D e f i n i t e I n d e f i n i t e

23. Extensions • POD • Reference resolution • Cohorts • Prosody • Disfluency

24. Joint action or “emergent alignment” • Work in the language-as-action tradition, including work in formal and computational semantics and pragmatics, assumes that interlocutors seek and use information about each others likely intentions, commitments, and likely knowledge, distinguishing between information that is in common ground and information that is privileged for each participant. Recently, however, several lines of investigation have suggested that whether knowledge is mutual or private may have little effect on real-time language is processing.

25. Keysar et al. Evidence against shared/private distinction w/common ground based on physical co-presence

26. What about considerations of speaker and addressee information states, likely knowledge (private vs. shared), commitments, etc.? Howdo we account for wide range of phenomena without these notions: e.g.,Questions vs. declarative questions, vs declaratives: Is the coffee ready? The coffee is ready The coffee is ready?

27. Problems with confederates andscripted instructions 1.Declarative request assumes addressee can perform an action to satisfy request 2.Matcher role in combination with Director, who instructs, creates assumption of omniscience. 3.Subject relinquishes control, tries not to act dumb, assuming evaluation. Consequence: May encourage egocentrism

28. Error and a martini glass with olives Experimenter who occasionally makes a mistake, incorrectly describing the object she is putting down Confederate speaker asking for objects from a script Subject following instructions to give speaker what she requests

29. Taking the speaker’s perspective Pick up the empty martini glass Confederate speaker, asking for objects from a script (blind to manipulation) Subject following instructions to give speaker what she requests

30. Late Match Mismatch

31. Late Match Mismatch

32. Late Match Mismatch POD effect: VERY early perspective-taking!

33. Referential domains in interactive conversation Task need to be truly collaborative. Language must be created by the interlocutors (non-scripted). Need real-time measure.

34. Bit of dialog from Brown-Schmidt, Campana & Tanenhaus • *ok, ok I got it* ele…ok • alright, *hold on*, I got another easy piece • *I got a* well wait I got a green piece RIGHT above that • above this piece? • well not exactly right above it • it can’t be above it • it’s to the…it’ doesn’t wanna fit in with the cardboard • it’s to the right, right? • yup • w- how? *where* • *it’s* kinda line up with the two holes • line ‘em right next to each other? • yeah, vertically • vertically, meaning? • up and down • up and down

35. Brown-Schmidt, Campana & Tanenhaus A sheet separates the subjects. eyetracker 2 1

36. Participants replace stickers / with blocks This is mid-way through the task: Subject 2’s Board Subject 1’s Board

37. Put it above the red block. Put it above the red vertical block. Definiteness and Referential Domains Definite referring expressions assume a uniquely identifiable referent with respect to a circumscribed referential domain.

38. the red vertical the red vertical the red vertical Referential expression was linguistically underspecified on half the trials!

39. What’s going on? Speakers linguistically disambiguate only when there is a salient (proximal and task-relevant) competitor in their referential domain (Beun & Cremer, 2003)

40. Task compatibility. Task compatibilityrefers to constraints on block placement due to the size and shape of the board, as well as the idiosyncratic systems that partners used to complete the task. • 1. ok, you’re gonna line it up… it’s gonna go <pause> one row ABOVE the green one, directly next to it. Only one green square has open board above it.

41. Are addressees confused?

42. the red vertical

43. Are addressees confused? No!

44. Summary Speakers are “underspecifying” Listeners are not confused Speakers disambiguate only when there is a salient (proximal and task-relevant) competitor in their referential domain. Speakers and addressees have similar, task-relevant, referential domains

45. Additional evidence about alignment • Further evidence comes from the absence of cohort effects: We had some blocks with pictures--names were cohorts, (e.g, clown and cloud) • Standard cohort effects when experimenter interrupted the conversation for a “calibration” check (e.g., “Look at the cloud, now look at the penguin”) • No hint of cohort effects when the participants referred to a picture block with a cohort (competitor was outside of the referential domain).

47. Use of scalars No contrast: 1% scalars Contrast on same island 72% Increase in disfluency ratea when contrast present but not modified. Eye movements: No looks to contrast member 26% Look to contrast member 91% Timing relates to form of utterance

48. First look to contrast member (e.g., small horse the big horse theee uh big horse the horse….oh the BIG one ms from onset of determiner

49. 1. In row four, does the horse with the hat match with what you have? (common) 2. In column one, The horse with the hat matches one of my guys with hats. (common) 3. In column 3, what is in the squares I can’t see? (private, addressee) 4. The horse with a hat is on your very left of row two? (common, hidden from speaker).

50. Typical trial Pick up the empty martini glass Confederate speaker, asking for objects from a script (blind to manipulation) Subject following instructions to give speaker what she requests