Back to the BlocksWorld: Learning New Actions through Situated Human-Robot Dialogue

1. Back to the BlocksWorld: Learning New Actions through Situated Human-Robot Dialogue Presented by Yuqian Jiang 2/27/2019

2. PROBLEM • Learn new actions through situated human-robot dialogue • ...in a simplified blocks world Source: https://goo.gl/images/nS1JgX

3. PROBLEM • How does a robot learn the action stack from a dialogue if it knows primitive actions: open gripper, close gripper, move

5. MOTIVATION • When robots work side-by-side with humans, they can learn new tasks from their human partners through dialogue • Challenges: • Human language: discrete and symbolic, robot representation: continuous • How to represent new knowledge so it can generalize? • How should the human teach new actions?

6. RELATED WORK • Following natural language instructions • Kollar et al., 2010; Tellex et al., 2011; Chen et al., 2010 • Learning by demonstration • Cakmak et al., 2010 • Connecting language with lower level control systems • Kress-Gazit et al., 2008; Siskind, 1999; Matuszek et al., 2012 • Using dialogue for action learning • Cantrell et al., 2012; Mohan et al., 2013

7. METHOD • A dialogue system for action learning

8. Intent Recognizer: • Command or confirmation • Semantic Processor: • Implemented using Combinatory Categorial Grammar (CCG) • Extracts action and object properties

9. “stack the blue block on the red block on your right.”

10. Perception Modules: • From camera image and internal status • A conjunction of predicates representing environment • Reference Solver: • Grounds objects in the semantic representation to the objects in the robot’s perception

11. “stack the blue block on the red block on your right.”

12. Dialogue manager: • A dialogue policy decides the dialogue acts based on the current state • Language Generator: • Pre-defined templates

13. ACTION MODULES • Action knowledge • Action execution • Action learning

15. ACTION LEARNING • If an action is not in the knowledge base, ask for instructions • Follow the instructions • Extract a goal state describing the action effects

17. ACTION LEARNING

18. EXPERIMENTS • Teach five new actions under two strategies • Pickup, Grab, Drop, ClearTop, Stack • step-by-step instructions vs. one-shot instructions (“pick up the blue block and put it on top of the red block”) • Five participants (more will be recruited)

19. EXPERIMENTS

20. RESULTS: Teaching Completion All failed teaching dialogues are one-shot instructions.

21. RESULTS: Teaching Duration Step-by-step dialogues take longer to learn.

22. RESULTS: Execution Step-by-step instructions have better generalization.

23. CONCLUSION • An approach to learn new actions from human-robot dialogue • On top of a layered planning/execution system • Integrated with language and perception modules • Success in generalizing to new situations in blocks world

24. CRITIQUE • Simplified domain with only 3 low-level actions • Cannot learn high-level actions that cannot be sequenced using these low-level actions • Cannot learn actions that involve objects that cannot be grounded • Is it really learning a new action, or just a new word that describes a goal using existing actions?

25. CRITIQUE • Only learns action effects, but no preconditions • Experiments do test situations that violate preconditions, such as picking up a block that has another block on top • Again, only successful because the preconditions of the underlying actions are modeled

26. CRITIQUE • Evaluation • Nothing surprising about the collaborative/non-collaborative results • Prefer to see more details on other modules of the system, and evaluation of their robustness

27. CRITIQUE • Challenges: ✔Human language: discrete and symbolic, robot representation: continuous ?How to represent new knowledge so it can generalize? ?How should the human teach new actions?