Exploring Error Handling in Spoken Dialogue Systems: A Research Overview

1. error handling – Higgins / Galatea Dialogs on Dialogs Group July 2005

2. work by … • Gabriel Skantzeph.d. studentKTH, Stockholm “I am doing research on spoken dialogue systems. More specifically, I am interested in studying miscommunication and error handling, but also in the representation and modelling of utterances and dialogue, as well as conducting experiments with users.“ • and co-authors: J. Edlund, D. House, R. Carlson

3. 3 papers • Higgins Higgins – a spoken dialogue system for investigating error handling techniques, Edlund, Skantze, Carlson [2004] • Galatea GALATEA: A Discourse Modeller Supporting Concept-Level Error Handling in Spoken Dialog Systems, Skantze [2005] • Prosody & Clarifications The Effects of Prosodic Features on the Interpretation of Clarification Ellipses, Edlund, House, Skantze [2004]

4. 1st paper • Higgins Higgins – a spoken dialogue system for investigating error handling techniques, Edlund, Skantze, Carlson [2004] • Galatea GALATEA: A Discourse Modeller Supporting Concept-Level Error Handling in Spoken Dialog Systems, Skantze [2005] • Prosody & Clarifications The Effects of Prosodic Features on the Interpretation of Clarification Ellipses, Edlund, House, Skantze [2004]

5. Higgins • practical goal of Higgins project • build a collaborative dialog system in which error handling ideas can be tested empirically • error handling issues, plus • incremental dialogue processing • on-line prosodic feature extraction • robust interpretation • flexible generation and output

6. domain • pedestrian city navigation and guiding • user gives system a destination • system guides user by giving verbal instructions • complex • large variety of error types • semantic structures can be quite complex • reference resolution • domain can be extended even further

7. architecture • follow-up from Adapt • everything is XML • domain objects • utterance semantics • discourse model • database content • system output (before surface) • 3D city model

8. research issues • early detection and correction • late detection • incrementality • error recovery

9. early detection and correction • KTH LVCSR – output likely to contain errors  • robust interpretation Pickering: • some syntactic analysis is needed • e.g. relations between objects • but handles insertions and non-agreement phrases • humans - good at early detection (woz)

10. late detection and correction • discourse modeller (GALATEA) • joins several results from Pickering into a discourse model • adds grounding information • can be manipulated later • remove concepts which turn out not to be grounded

11. incrementality • end-pointers cause trouble • even more so in this domain better:

12. incrementality [2] • all components support incremental processing • several issues • when to barge in? (semantic content and prosody) • longer-than-utterance units: interpreter or dialog manager? • rapid and unobtrusive feedback: challenge for synthesis

13. error recovery • signaling non-understandings • decreased experience of task success • slower recovery • ask other task-related question

14. 2nd paper • Higgins Higgins – a spoken dialogue system for investigating error handling techniques, Edlund, Skantze, Carlson [2004] • Galatea GALATEA: A Discourse Modeller Supporting Concept-Level Error Handling in Spoken Dialog Systems, Skantze [2005] • Prosody & Clarifications The Effects of Prosodic Features on the Interpretation of Clarification Ellipses, Edlund, House, Skantze [2004]

15. GALATEA • a discourse modeller for conversational spoken dialog systems • builds a discourse model (what has been said during the discourse) • resolution of ellipses & anaphora • tracks the grounding status • who said what when (plus confidence information) • can be used for concept-level error handling

16. should do grounding at concept level • explicit and implicit verification on whole utterance can be tedious and unnatural • 45% of clarifications in BNC are fragmentary / elliptical

17. should do grounding at concept level • Traum (1994) – utterance level computational model of grounding • Larsson (2002) – issue-level computational model of grounding in Issue-Based DM • Rieser (2004), Schlangen (2004):systems capable of fragmentary clarification requests, but models do not handle user reactions • systems should keep grounding information at the concept level • like RavenClaw? 

18. semantic representation • rooted unordered trees of semantic concepts • nodes: attr-value pairs, objects, relations, properties

19. semantic representation • enhanced with “meta”-information • confidence • communicative acts • info is new / given

20. ellipsis resolution • transforms ellipsis into full propositions • rule based • ~10 rules • domain-specific

21. anaphora resolution • keeps a list of entities (talked about) • assigns ids • when given entities are added to the discourse, look up the antecedent • if found, unification (and move to the top of the entity list) • unification also allows entities to be referred to in new ways • how does this fare and compare?

22. grounding status • who added the concept? • in which turn? • how confident? • may be used by the action manager • for instance remove all items with high grounding status when referring to an entity

23. updating grounding status

24. late error detection • discover inconsistencies in discourse model • look at grounding status to see where error may be • concept can be removed

25. future • methods for automatic tuning of strategy selection • extend to track confidence and grounding status at different levels • evaluate • how people respond to incorrect confirmations, and how can that information be used to update grounding status • error recovery after non-understandings • other domains

26. 3rd paper • Higgins Higgins – a spoken dialogue system for investigating error handling techniques, Edlund, Skantze, Carlson [2004] • Galatea GALATEA: A Discourse Modeller Supporting Concept-Level Error Handling in Spoken Dialog Systems, Skantze [2005] • Prosody & Clarifications The Effects of Prosodic Features on the Interpretation of Clarification Ellipses, Edlund, House, Skantze [2004]

27. prosody in clarifications • effects of prosodic features on interpretation of elliptical clarifications • U: Further ahead on the right I see a red building… • S: Red (?) • vary prosodic features • study impact on user’s understanding of the system’s intention

28. motivation • long (whole utterance) confirmations are not good • tedious, unnatural • BNC corpus: 45% of clarifications are elliptical • short confirmations • make dialog more efficient by focusing on the actual problematic fragments • however • interpretation depends on context and prosody

29. 3 readings • U: Further ahead on the right I see a red building… • S: Red (?) • Ok, red [all positive] • Do you really mean red? What do you mean by red? [positive perception, negative understanding] • Did you say red? [positive contact, negative perception]

30. stimuli • 3 test words [red, blue, yellow] • di-phone voice (MBROLA) • manipulated • peak position [mid, early, late / 100ms] • peak height [130Hz / 160 Hz] • vowel duration [normal, long / +100ms]

31. subjects + design • 8 speakers: 2f / 6m, 2nn / 6n • introduced to Higgins • listen to all 42 (only once); random order • 3 options • Okay, X • Did you really mean X? • Did you say X?

32. results • no effects for • color, subject, duration • significant effects for • peak position, peak height, & their interaction

33. results • Statement: early, low peak • Question: late, high peak • Clear division between “did you mean” and “did you say”

34. food for thought • how about English? • red • red? • red!? • how many ways can you say it?

35. conclusion • strong relationship between intonation and meaning • statement: early, low peak • question: late, high peak • clear division between “did you mean” and “did you say”

36. the end