Heterogeneous Multi-Robot Dialogues for Search Tasks

1. Heterogeneous Multi-Robot Dialogues for Search Tasks Thomas K Harris, Satanjeev (Bano) Banerjee Alexander Rudnicky AAAI Spring Symposium 2005: Dialogical Robots

2. Communication among autonomous robots and humans • Embodied (not necessarily robotic, or even anthropomorphic) agents will become ubiquitous • NL dialogue will be a useful modality • Agent’s dialogue and task models will remain tightly coupled, and independent from other agents • We will be talking to multiple agents without a (completely) shared dialog system AAAI Spring ’05 Symposium: Dialogical Robots

3. Gedankenexperiment • The problem of many robots: sweeper, furniture mover, baby monitor. • Task: clean up house; don’t wake baby up. • Who do you talk to? • What do you say? • Who talks back? AAAI Spring ’05 Symposium: Dialogical Robots

4. Issues for polylogical systems • What’s a reasonable architecture? • A single system controlling multiple robots • A single DM directing autonomous robots • Shared understanding/generation • Shared microphone and speakers • Robots self-contained: Nothing shared • How to communicate with multiple agents? • Directions to the entire team? (“talk into the air”) • Mediation through a designated team leader? (foreman) • Instructions to each team member? (direct management) • To a proxy agent within/without the team? (personal assistant) AAAI Spring ’05 Symposium: Dialogical Robots

5. What do we have today? • A platform that supports multiple (robot) participants • Sphinx II ASR; Festival TTS • RavenClaw dialogue manager • Galaxy-II message-passing architecture • Several “back-end” interfaces • A basic corridor movement domain • Commands to move robots along in vectors and towards named locations • Mechanisms for describing robot status and location to human interlocutor AAAI Spring ’05 Symposium: Dialogical Robots

6. Current Architecture HUMAN INTERFACE ASR AGENT1 DM1 Robot1 Interpret1 AGENT2 Interpret2 DM2 Robot2 Multi-voice TTS AAAI Spring ’05 Symposium: Dialogical Robots

7. Implementation HUMAN INTERFACE AGENT1 Sphinx II Phoenix Helios Ravenclaw Bashful Bashful Robot Communication System Galaxy AGENT2 Ravenclaw Clyde Clyde Swift Rosetta AAAI Spring ’05 Symposium: Dialogical Robots

8. Human-Robot Treasure Hunt • One Human • Two robots • Segway • Pioneer • Treasures “hidden” in large space • TASK: Retrieve all treasures AAAI Spring ’05 Symposium: Dialogical Robots

9. Scenario for Search We found it! We are at <x,y> AAAI Spring ’05 Symposium: Dialogical Robots

10. Issues in H-R communication • How do people decompose the task into sub-tasks? • What language do people use to get the tasks performed by the robots? • Given a human command, what is the expected robot behavior? • Explore using Wizard of Oz experiments AAAI Spring ’05 Symposium: Dialogical Robots

11. WOZ design • 1 experimenter, 2 robot actors, 1 participant. • Experimenter places treasure, simulates robot treasure sensors. • 1st robot actor is blind, but carries a webcam for the participant’s consumptions. • 2nd robot actor can only move by crawling. Communication takes place by normal speech, walkie-talkie, and through the webcam. AAAI Spring ’05 Symposium: Dialogical Robots

12. Annotation and analysis Data transcribed and annotated Utterances classified into functional categories 394 utterances 20 utterance categories 8 major categories Carnegie Mellon MockBrow annotation tool AAAI Spring ’05 Symposium: Dialogical Robots

13. Controlling team behaviors Grounding Positive/negative feedback Informing robot of it’s state or the world Explanations of commands Orientation Grounding Navigation Simple Navigation commands Spatial Referential Navigation Object Referential Navigation Manipulation Manipulating the environment Manipulating treasure Coverage Manipulating the webcam view Object coverage commands Generic coverage Asking about the robot’s abilities Filler Real-time command modifications Utterance/Task Breakdown AAAI Spring ’05 Symposium: Dialogical Robots

14. Summary / Future Work • Architecture for human-robot teams • WoZ study of language requirements • Different WoZ scenarios • Implement language for robot team • “Field” testing AAAI Spring ’05 Symposium: Dialogical Robots

15. Controlling team behaviors • “you guys get together” • “T- you go first and B- follow” AAAI Spring ’05 Symposium: Dialogical Robots

16. Grounding • Positive/negative feedback • “ok that’s better” • Informing robot of state • “so that’s up” • “I don’t see anything there” • Explanations of commands • “so I can see which direction is up” • Orientation Grounding • “What you’re facing now with the camera – is that the vehicle that you just circumnavigated” • “I can tell you’re going in the wrong direction, stop” AAAI Spring ’05 Symposium: Dialogical Robots

17. Navigation • Simple Navigation commands • “so um T- turn to you left” • “T- I want you to turn right 90 degrees” • “can you go in that general direction” • “can you proceed in that direction” • Spatial Referential Navigation • “go to that open area” • “continue around the periphery of that open area” • “back out of that alley” • “proceed in that direction until you find an opening to turn left” • Object Referential Navigation • “go over by T-” • “can you go on the other side of that vehicle” • “go over by the posters” AAAI Spring ’05 Symposium: Dialogical Robots

18. Manipulation • Manipulating the environment • “T- why don’t you move the trash can” • Manipulating treasure • “T- bring the coin to me” • Manipulating the webcam view • “ok B- look to your left” • “B- can you look around with the camera a little” AAAI Spring ’05 Symposium: Dialogical Robots

19. Coverage • Object coverage commands • “ok so examine the shelf” • “do you see something on that shelf in front of B-” • “can you look over by that table over there” • Generic Coverage • “do you see anything that looks interesting” AAAI Spring ’05 Symposium: Dialogical Robots

20. Asking about the robot’s abilities • “is that possible” AAAI Spring ’05 Symposium: Dialogical Robots

21. Filler • “and now um” • “ok um” AAAI Spring ’05 Symposium: Dialogical Robots

22. Real-time command modifications • “keep going” • “stop” • “a little more” • “change of plans” • “other direction” AAAI Spring ’05 Symposium: Dialogical Robots