Error Awareness and Recovery in Task-Oriented Spoken Dialogue Systems

1. Error Awareness and Recovery in Task-Oriented Spoken Dialogue Systems Thesis Proposal Dan Bohus Carnegie Mellon University, January 2004 Thesis Committee Alex Rudnicky (Chair) Roni Rosenfeld Jeff Schneider Eric Horvitz (Microsoft Research)

2. Problem • Spans most domains and interaction types • Has a significant impact on performance Lack of robustness when faced with understanding errors

3. An example S: Are you a registered user ? U: No I'm not. No [NO I'M NOT NO] S: What is your full name ? U: Adam Schumacher [ATHENS IN AKRON] S: There is an Athens in Georgia and in Greece. Which destination did you want ? U: Neither [NEITHER] S: I'm sorry, I didn't catch that. There is an Athens in Georgia and in Greece. Which destination did you want ? U: Georgia [GEORGIA] S: A flight from Athens... Where do you want to go ? U: Start over [START OVER] S: Do you really want to start over ? U: Yes [YES] S: What city are you leaving from ? U: Hamilton, Ontario [HILTON ONTARIO] S: Sorry, I'm not sure I understood what you said. Where are you leaving from ? U: Hamilton [HILTON] S: I'm still having trouble understanding you. To go on, I need you to answer the following question. What city are you leaving from ? U: Toronto [TORONTO]

4. Some Statistics … • Corrections [Krahmer, Swerts, Litman, Levow] • 30% of utterances correct system mistakes • 2-3 times more likely to be misrecognized • Semantic error rates

5. Significant Impact on Interaction • CMU Communicator 40% Failed 26% 33% sessions Contain understanding errors • Multi-site Communicator Corpus [Shin et al] Failed 37% 63% sessions

6. Outline • Problem • Approach • Infrastructure • Research Program • Timeline & Summary problem : approach : infrastructure : indicators : strategies : decision process : summary

7. Increasing Robustness … • Increase the accuracy of speech recognition • Assume recognition is unreliable, and create the mechanisms for acting robustly at the dialogue management level • ASR performance increases / demands increase • More general problem : approach : infrastructure : indicators : strategies : decision process : summary

8. Snapshot of Existing Work: Slide 1 • Theoretical models of grounding • Contribution Model [Clark], Grounding Acts [Traum] • Practice: heuristic rules • Misunderstandings • Threshold(s) on confidence scores • Non-understandings Analytical/Descriptive, not decision oriented Ad-hoc, lack generality, not easy to extend problem : approach : infrastructure : indicators : strategies : decision process : summary

9. Snapshot of Existing Work: Slide 2 • Conversation as Action under Uncertainty [Paek and Horvitz] • Belief networks to model uncertainties • Decisions based on expected utility, VOI-analysis • Reinforcement learning for dialogue control policies[Singh, Kearns, Litman, Walker, Levin, Pieraccini, Young, Scheffler, etc] • Formulate dialogue control as an MDP • Learn optimal control policy from data Do not scale up to complex, real-world domains problem : approach : infrastructure : indicators : strategies : decision process : summary

10. Research Program: Goals & Approach • Decision making under uncertainty A task-independent, adaptive and scalable framework for error recovery in task-oriented spoken dialogue systems A task-independent, adaptive and scalable framework for error recovery in task-oriented spoken dialogue systems Approach: problem : approach : infrastructure : indicators : strategies : decision process : summary

11. Three Components 0. Infrastructure 1. Error awareness 2. Error recovery strategies 3. Error handling decision process Develop indicators that … Assess reliability of information Assess how well the dialogue is advancing Develop and investigate an extended set of conversational error handling strategies Develop a scalable reinforcement-learning based approach for error recovery in spoken dialogue systems problem : approach : infrastructure : indicators : strategies : decision process : summary problem : approach : infrastructure : indicators : strategies : decision process : summary

12. Infrastructure • RavenClaw • Modern dialog management framework for complex, task-oriented domains • RavenClaw spoken dialogue systems • Test-bed for evaluation Completed Completed problem : approach : infrastructure : indicators : strategies : decision process : summary

13. RoomLine user_name results query registered Login GetQuery GetResults DiscussResults Welcome GreetUser DateTime Location Properties AskRegistered AskName Network Projector Whiteboard registered: [No]-> false, [Yes] -> true registered: [No]-> false, [Yes] -> true user_name: [UserName] ExplicitConfirm Error Handling Decision Process registered: [No]-> false, [Yes] -> true user_name: [UserName] query.date_time: [DateTime] query.location: [Location] query.network: [Network] AskRegistered Indicators Login RoomLine Strategies Dialogue Stack Expectation Agenda RavenClaw Dialogue Task (Specification) Domain-Independent Dialogue Engine problem : approach : infrastructure : indicators : strategies : decision process : summary

14. RavenClaw-based Systems • RoomLine • CMU Let’s Go!! Bus Information System • LARRI [Symphony] • TeamTalk [11-741] • Eureka [11-743] problem : approach : infrastructure : indicators : strategies : decision process : summary

15. Three Components 0. Infrastructure 1. Error awareness 2. Error recovery strategies 3. Error handling decision process Develop indicators that … Assess reliability of information Assess how well the dialogue is advancing Develop and investigate an extended set of conversational error handling strategies Develop a scalable reinforcement-learning based approach for error recovery in spoken dialogue systems problem : approach : infrastructure : indicators : strategies : decision process : summary

16. Existing Work • Confidence Annotation • Traditionally focused on speech recognizer[Bansal, Chase, Cox, and others] • Recently, multiple sources of knowledge[San-Segundo, Walker, Bosch, Bohus, and others] • Recognition, parsing, dialogue management • Detect misunderstandings: ~ 80-90% accuracy • Correction and Aware Site Detection[Swerts, Litman, Levow and others] • Multiple sources of knowledge • Detect corrections: ~ 80-90% accuracy problem : approach : infrastructure : indicators : strategies : decision process : summary

17. Proposed: Belief Updating • Continuously assess beliefs in light of initial confidence and subsequent events • An example: S: Where are you flying from? U: [CityName={Aspen/0.6; Austin/0.2}] S: Did you say you wanted to fly out of Aspen? U: [No] [CityName={Boston/0.8}] initial belief + system action + user response updated belief [CityName={Aspen/?; Austin/?; Boston/?}] problem : approach : infrastructure : indicators : strategies : decision process : summary

18. contents confidence contents confidence correction correction Belief Updating: Approach • Model the update in a dynamic belief network t t + 1 C C C C initial belief updated belief system action system action User response features CurrentTop Current2nd Current3rd Confidence Yes No Utterance Length Positive Markers Negative Markers problem : approach : infrastructure : indicators : strategies : decision process : summary

19. Three Components 0. Infrastructure 1. Error awareness 2. Error recovery strategies 3. Error handling decision process Develop indicators that … Assess reliability of information Assess how well the dialogue is advancing Develop and investigate an extended set of conversational error handling strategies Develop a scalable reinforcement-learning based approach for error recovery in spoken dialogue systems problem : approach : infrastructure : indicators : strategies : decision process : summary

20. Is the Dialogue Advancing Normally? Locally, turn-level: • Non-understanding indicators • Non-understanding flag directly available • Develop additional indicators • Recognition, Understanding, Interpretation Globally, discourse-level: • Dialogue-on-track indicators • Summary statistics of non-understanding indicators • Rate of dialogue advance problem : approach : infrastructure : indicators : strategies : decision process : summary

21. Three Components 0. Infrastructure 1. Error awareness 2. Error recovery strategies 3. Error handling decision process Develop indicators that … Assess reliability of information  Assess how well the dialogue is advancing Develop and investigate an extended set of conversational error handling strategies Develop a scalable reinforcement-learning based approach for error recovery in spoken dialogue systems problem : approach : infrastructure : indicators : strategies : decision process : summary

22. Error Recovery Strategies • Identify • Identify and define an extended set of error handling strategies • Implement • Construct task-decoupled implementations of a large number of strategies • Evaluate • Evaluate performance and bring further refinements

23. User Initiated System Initiated Help Ensure that the system has reliable information (misunderstandings) Ensure that the dialogue on track Where are we? Start over Scratch concept value Go back Channel establishment Explicit confirmation Global problems (compounded, discourse-level problems) Local problems (non-understandings) Suspend/Resume Implicit confirmation Repeat Disambiguation Switch input modality Summarize Ask repeat concept SNR repair Quit Reject concept Restart subtask plan Ask repeat turn Select alternative plan Ask rephrase turn Start over Notify non-understanding Terminate session / Direct to operator Explicit confirm turn Targeted help WH-reformulation Keep-a-word reformulation Generic help You can say List of Error Recovery Strategies problem : approach : infrastructure : indicators : strategies : decision process : summary

24. User Initiated System Initiated Help Ensure that the system has reliable information (misunderstandings) Ensure that the dialogue on track Where are we? Start over Scratch concept value Go back Channel establishment Explicit confirmation Global problems (compounded, discourse-level problems) Local problems (non-understandings) Suspend/Resume Implicit confirmation Repeat Disambiguation Switch input modality Summarize Ask repeat concept SNR repair Quit Reject concept Restart subtask plan Ask repeat turn Select alternative plan Ask rephrase turn Start over Notify non-understanding Terminate session / Direct to operator Explicit confirm turn Targeted help WH-reformulation Keep-a-word reformulation Generic help You can say List of Error Recovery Strategies problem : approach : infrastructure : indicators : strategies : decision process : summary

25. Error Recovery Strategies: Evaluation • Reusability • Deploy in different spoken dialogue systems • Efficiency of non-understanding strategies • Simple metric: Is the next utterance understood? • Efficiency depends on decision process • Construct upper and lower bounds for efficiency • Lower bound: decision process which chooses uniformly • Upper bound: human performs decision process (WOZ) problem : approach : infrastructure : indicators : strategies : decision process : summary

26. Three Components 0. Infrastructure 1. Error awareness 2. Error recovery strategies 3. Error handling decision process Develop indicators that … Assess reliability of information  Assess how well the dialogue is advancing  Develop and investigate an extended set of conversational error handling strategies Develop a scalable reinforcement-learning based approach for error recovery in spoken dialogue systems problem : approach : infrastructure : indicators : strategies : decision process : summary

27. Previous Reinforcement Learning Work • Dialogue control ~ Markov Decision Process • States • Actions • Rewards • Previous work: successes in small domains • NJFun [Singh, Kearns, Litman, Walker et al] • Problems • Lack of scalability • Once learned, policies are not reusable S2 A S3 S1 problem : approach : infrastructure : indicators : strategies : decision process : summary

28. Proposed Approach Overcome previous shortcomings: • Focus learning only on error handling • Reduces the size of the learning problem • Favors reusability of learned policies • Lessens the system development effort • Use a “divide-and-conquer” approach • Leverage independences in dialogue problem : approach : infrastructure : indicators : strategies : decision process : summary

29. No Action Topic-MDP Explicit Confirmation Topic-MDP No Action user_name registered Topic-MDP No Action Concept-MDP Concept-MDP Explicit Confirm No Action Gated Markov Decision Processes • Small-size models • Parameters can be tied across models • Easy to design initial policies RoomLine Login Welcome GreetUser Gating Mechanism AskRegistered AskName • Decoupling favors reusability of policies • Accommodate dynamic task generation • Independence assumption problem : approach : infrastructure : indicators : strategies : decision process : summary

30. Global, post-gate rewards Local rewards Reward Action Action Gating Mechanism Gating Mechanism Reward Reward Reward MDP MDP MDP MDP MDP MDP Reward structure & learning • Rewards based on any dialogue performance metric • Atypical, multi-agent reinforcement learning setting • Multiple, standard RL problems • Model-based approaches problem : approach : infrastructure : indicators : strategies : decision process : summary

31. Evaluation • Performance • Compare learned policies with initial heuristic policies • Metrics • Task completion • Efficiency • Number and lengths of error segments • User satisfaction • Scalability • Deploy in a system operating with a sizable task • Theoretical analysis problem : approach : infrastructure : indicators : strategies : decision process : summary

32. Outline • Problem • Approach • Infrastructure • Research Program • Summary & Timeline problem : approach : infrastructure : indicators : strategies : decision process : summary

33. Summary of Contributions • Overall Goal:develop a task-independent, adaptive and scalable framework for error recovery in task-oriented spoken dialogue systems • Modern dialogue management framework • Belief updating framework • Investigation of an extended set of error handling strategies • Scalable data-driven approach for learning error handling policies problem : approach : infrastructure : indicators : strategies : decision process : summary

34. Timeline indicators strategies decisions data now proposal Misunderstanding andnon-understandingstrategies Investigatetheoreticalaspects ofproposedreinforcementlearningmodel end ofyear 4 Evaluatenon-understandingstrategies; developremaining strategies Data collection forbelief updating andWOZ study milestone 1 Develop andevaluate thebelief updatingmodels Implementdialogue-on-trackindicators milestone 2 Data collection forRL training Error handling decision process: reinforcement learning experiments Data collection forRL evaluation end ofyear 5 milestone 3 Contingency data collection efforts Additional experiments: extensions or contingency work 5.5 years defense problem : approach : infrastructure : indicators : strategies : decision process : summary

35. Thank You! Questions & Comments committee members, then floor

36. System acquires correct information OK System acquires information System acquires incorrect information Understanding process Misunderstanding System does not acquire information Non-understanding Indicators: Goals • Goal:Increase awareness and capacity to detect problems • Develop indicators which can inform the error handling process about potential problems

37. problem: approach : support work : indicators : strategies : decision process : summary

38. Three Desired Properties • Task-Independence • Reuse the proposed architecture across different spoken dialogue systems with a minimal amount of authoring effort • Adaptability • Learn from experience how to adapt to the characteristics of various domains • Scalability • Applicable in spoken dialogue systems operating with large, practical tasks

39. ExplConf ExplConf ExplConf ImplConf ImplConf ImplConf LC MC HC NoAct NoAct NoAct NoAct 0

40. Belief Updating: Approach • Model the update in a dynamic belief network • Top-N values • Fixed structure • Learn parameters • Data collection • Evaluation • Accuracy • Soft-error t t + 1 C C C C System Action System Action CurrentTop Current2nd Current3rd CurrentTop Current2nd Current3rd Confidence Confidence No No Yes Yes Utterance Length Utterance Length Negative Markers Negative Markers Positive Markers Positive Markers User response features problem : approach : infrastructure : indicators : strategies : decision process : summary

41. No Action Topic-MDP Explicit Confirmation Topic-MDP No Action user_name registered Topic-MDP No Action Concept-MDP Concept-MDP Explicit Confirm No Action Gated Markov Decision Processes Issues: • Structure of individual MDPs • Gating mechanism • Reward structure and learning RoomLine Login Welcome GreetUser Gating Mechanism AskRegistered AskName problem : approach : infrastructure : indicators : strategies : decision process : summary

42. Structure for individual MDPs • State-space: • informative subset of corresponding indicators • Concept-MDPs: confidence / beliefs • Topic-MDPs: non-understanding, dialogue-on-track indicators • Action-space • corresponding system-initiated error handling strategies problem : approach : infrastructure : indicators : strategies : decision process : summary

43. Gating Mechanism • Heuristic derived from domain-independent dialogue principles • Give priority to topics over concept • Give priority to entities closer to the conversational focus problem : approach : infrastructure : indicators : strategies : decision process : summary