Module u1: Speech in the Interface 6: Human Communication

Module u1: Speech in the Interface 6: Human Communication Jacques Terken SAI User-System Interaction U1, Speech in the interface: 6. Human communication

Motivation • provide knowledge about basic concepts of human communication relevant to the design of speech interfaces • explore use of knowledge about human communication as model for man-machine interaction SAI User-System Interaction U1, Speech in the interface: 6. Human communication

Contents • basic scheme for communication • the structure of messages • the structure of communication • “how to do things with words”1: communication as process • implications for man-machine interaction • key points • human-human communication as model for human-computer interaction • 1 J.L. Austin: “How to do things with words”, 1962 SAI User-System Interaction U1, Speech in the interface: 6. Human communication

context context sender receiver channel feedback basic scheme • Context • Situational context  situated communication: time, place, objects • Social context • Cultural context • Dialogue context SAI User-System Interaction U1, Speech in the interface: 6. Human communication

intentions and effects • Sender: intention to convey a message  message needs to be encoded • Receiver: decoding  effect Ideal communication: intention = effect Distortions underway: intention  effect’ Furthermore: effect is more than intention (sender also communicates unintentional information) • complex messages require sequence of steps  communication as structured action / collaboration SAI User-System Interaction U1, Speech in the interface: 6. Human communication

Contents • basic scheme for communication • the structure of messages • the structure of communication • communication as process • implications for man-machine interaction • key points • human-human communication as model for human-computer interaction SAI User-System Interaction U1, Speech in the interface: 6. Human communication

formal aspects (1) • Simple case: messages containing a single dialogue act (inform, request, confirmation etc) • Sentence/utterance • put the red block before the yellow block next to the ball • I herewith pronounce you man and wife • What time does the next flight to Boston leave? • Sentences made up of word groups • Word groups made up of words • Words made up of syllables and phonemes (~40) or graphemes (~25) / @, e, a, r, … / ; { e, e, a, r, … } SAI User-System Interaction U1, Speech in the interface: 6. Human communication

formal aspects (2) • rules determine well-formedness of representations at different levels: • pragmatics: sentence meaning in context (speech act, dialogue act)  contextual meanings *the king of France is bald (but may be okay in a story) • semantics: sentence meanings  possible meanings *the 50 year old bachelor surprised his wife • syntax: sentence structure  possible sentences • morphology: word structure  possible words • Dutch: loopje; *loopetje; kommetje; *kompje • Phonology: sound structure  possible sound sequences Dutch: herfst; Polish: przewalski SAI User-System Interaction U1, Speech in the interface: 6. Human communication

formal aspects (3) • mapping between levels not fully specified (not one - to - one):  ambiguity • Mapping sounds onto words: /Aiscri:m/ {I scream} or {Ice cream} • Mapping words onto sentence structure: Syntactic ambiguities i saw the girl with the binoculars SAI User-System Interaction U1, Speech in the interface: 6. Human communication

put the red block before the yellow block next to the ball (1) put the red block before the yellow block [ that is ] next to the ball (2) put the red block before the yellow block, next to the ball SAI User-System Interaction U1, Speech in the interface: 6. Human communication

levels of analysis topic … topic … topic transaction exchange exchange exchange turn turn turn move move move SAI User-System Interaction U1, Speech in the interface: 6. Human communication

transaction • (part of) conversation dealing with a single topic • consists of one or more exchanges SAI User-System Interaction U1, Speech in the interface: 6. Human communication

exchange • smallest interactive unit • consists minimally of two turns produced by two different speakers • involves the negotiation of a single piece of information • examples: request - inform - confirm presentation - acceptance SAI User-System Interaction U1, Speech in the interface: 6. Human communication

adjacency pairs: question answer request accept / turn down invitation accept / decline greeting greeting apology smoother • Interactive pressure: opening move induces obligation to answer with closing move (conversational obligation) SAI User-System Interaction U1, Speech in the interface: 6. Human communication

nesting of adjacency pairs: s: question X what time is the next train to amsterdam a: question Y amsterdam central? s: answer Y yes a: answer X in seven minutes at platform 5 mixed initiative vs.system initiative SAI User-System Interaction U1, Speech in the interface: 6. Human communication

turn • Turn: everything the speaker says before the next speaker takes over • Rules for turn-taking behaviour 1. Speaker selects next speaker 2. Next speaker selects self Window of opportunity for turn switching (minimizes overlap and interruption; interruptions only permissible in particular circumstances • Turn consists of one or more moves SAI User-System Interaction U1, Speech in the interface: 6. Human communication

temporal aspects of turn-taking • dialog not as a sequence of well-formed non-overlapping turns: • time window for turn taking • overlap, interruptions SAI User-System Interaction U1, Speech in the interface: 6. Human communication

move/dialog act • no fixed inventory, but dependent on application • Common types: • Primary dialogue acts • request for action • inform • question • Game theory conceives of communication as a sequence of moves Expresses the assumption that communication is governed by a set of rules • Dialogue control acts • acknowledge • check • confirm • greeting • thanks SAI User-System Interaction U1, Speech in the interface: 6. Human communication

Indirect communication • No one-to-one mapping between form and function: indirect communication (politeness forms) • Questions: • Can you tell me what time it is (yes/no question) • I would like to know what time it is (statement) • Do you have a watch (yes/no question with indirect communication ) Direct communication: • What time is it (wh-question) • Please tell me the time (request) SAI User-System Interaction U1, Speech in the interface: 6. Human communication

spoken dialog systems • Identify dialog act performed by user in order to decide what dialog act to produce as an an answer • Machine learning techniques in order to learn best continuation in context x (and to learn best path) SAI User-System Interaction U1, Speech in the interface: 6. Human communication

spoken communication as joint action/cooperation • Speech is volatile • Mechanisms to cope with volatile nature of speech • co-operation receiver must be in receiving mode, i.e., be open to message sender must adjust to channel and receiver characteristics SAI User-System Interaction U1, Speech in the interface: 6. Human communication

spoken communication as joint action/cooperation (2) • Goal achievement is usually accomplished through a number of steps involving both participants: dialogue • Monitoring of transmission success by sender on the basis of conscious and unconscious feedback from receiver • Immediate action in case of problems either by user or sender • Role of eye gaze in staying in touch, monitoring and turn taking SAI User-System Interaction U1, Speech in the interface: 6. Human communication

speech generation • two simultaneous processes: • conceptualisation (“what to say”: content) • formulation (“how to say it”: converting content into utterance) simultaneous, but formulation lagging behind (  spontaneous speech phenomena) • restarts, self-corrections, hesitations (silence, ehm), disfluencies • grammar of spoken language SAI User-System Interaction U1, Speech in the interface: 6. Human communication

message formulation: issues • continuity, coherence: what we say usually links to what was said before: context provides search space for interpretation and allows for efficient expression • pronouns, anaphora • prosody • bridging: establishing a relation between expressions in consecutive utterances can you take the picnic supplies out of the van? the beer is probably warm, so please put it in the fridge. SAI User-System Interaction U1, Speech in the interface: 6. Human communication

grice’s cooperativity principle human communication (supposed to be) governed by grice’s cooperativity principle, consisting of the following maxims (guidelines for communication): • quantity: make your contribution as informative as is required I want a flight; I want a flight from A to B because …. • quality: do not say what you believe is false or for which you lack adequate evidence • relation: be relevant • manner: be perspicuous, brief and orderly; avoid obscurity and ambiguity SAI User-System Interaction U1, Speech in the interface: 6. Human communication

speech comprehension • Stages in comprehension: • Perception: perceiving sounds and mapping sounds onto words • Interpretation • Evaluation • From a processing point of view perception and interpretation are closely interlinked keywords: on-line processing, hypothesis testing and integrating SAI User-System Interaction U1, Speech in the interface: 6. Human communication

backchannelling • “utterances” and other communicative signals guiding the progress of the communication, i.e., serving a control function • nodding • yes, hm-hm • “madrid?” SAI User-System Interaction U1, Speech in the interface: 6. Human communication

error recovery • on-line property allows fast and efficient recovery: • problems noticed immediately • cause of problem usually easily identified “what did you say?” “did you say …” “what do you mean” SAI User-System Interaction U1, Speech in the interface: 6. Human communication

common ground • Grounding agreeing about current state of the dialogue Important role of feedback • Shared knowledge + knowledge that speaker knows is available to hearer + world knowledge (“living memory”) + knowledge concerning context of communication and what has been established by previous part of discourse SAI User-System Interaction U1, Speech in the interface: 6. Human communication

human communication no neat succession of turns: overlap in time: provision for speech recognition needed • Indirect communication: message function and form no 1:1 relation • interpretation dependent on domain knowledge: requires domain modelling: successful for restricted domains • interpretation dependent on context: requires discourse model SAI User-System Interaction U1, Speech in the interface: 6. Human communication

supporting the interaction • Prompting: system initiative • Zooming: increase amount of feedforward in system prompt • How may I help you • Which connection • Are you interested in connections or prices • Please respond with yes or no • Cooperative replies • User: “Do you have flights between 7:15 and 7:45?” • S: “No, but there is a flight at 7:55” SAI User-System Interaction U1, Speech in the interface: 6. Human communication

robustness issue • Lack of robustness creates need for verifcation • Explicit verification: • Did you say “amsterdam”? • Implicit verification • U: “I want to travel from .. to rotterdam” • S: “when do you want to travel to rotterdam?” • Cf: S: “when do you want to travel to amsterdam?” SAI User-System Interaction U1, Speech in the interface: 6. Human communication

Verification is costly when you got it right • Feedback (positive feedback or implicit verification) is misleading when you good it wrong • Use verfication when uncertain • Use feedback when confident SAI User-System Interaction U1, Speech in the interface: 6. Human communication

confidence-based verification • High confidence: just accept or implicit verification • Medium confidence: explicit verification • Low confidence: explicit request for repetition • Precision/recall issue for correctness and confidence measure: • Confidence measure may not be right: either too high or too low SAI User-System Interaction U1, Speech in the interface: 6. Human communication

communication as joint action (cooperation) • taking into consideration listener needs • listening for intention • rule-governed behaviour at many different levels • structure of messages • structure of conversations (e.g. turn-taking behaviour) • grounding • feedback (back-channelling) SAI User-System Interaction U1, Speech in the interface: 6. Human communication

cooperative assistant • Characteristics of cooperative assistant • Interprets messages in terms of intentions not in terms of form (“c.a. is not a politician or lawyer”) • Cooperative replies (not blunt “no” but offers alternatives, e.g. through constraint relaxation) • Context-sensitive replies and help No global help manual • Proactive, triggered by implicit cues Does not await explicit request for help but anticipates SAI User-System Interaction U1, Speech in the interface: 6. Human communication

discussion • Is this a good model for human computer interaction? • Considerations: • Schneiderman c.s.: user should stay in control • Maes/Sidner: • computers should be able to understand our intentions so that we don’t have to specify all the actions explicitly ourselves • Computers should monitor our actions so that they can learn how to anticipate our needs and preferences SAI User-System Interaction U1, Speech in the interface: 6. Human communication

Module u1: Speech in the Interface 6: Human Communication