Learning to Coordinate Behaviors

1. Learning to Coordinate Behaviors Pattie Maes & Rodney A. Brooks Presented by: Javier Martinez

2. Introduction • Behavior-based system • Learning using positive and negative feedback • Behaviors decide when is time to activate • Distributed algorithm • Test the concept in a robot

3. Motivation • Behavior control is a weak point initial Behavior-based systems • Behavior control has to be prewired • This approach doesn’t scale too well

4. New Ideas • Behavior control is learned through experience • Learning algorithm completely distributed • Each behavior learns when to become active • The solution maximizes positive feedback and minimizes negative feedback

5. The Learning Task What is needed: • Vector of binary perceptual conditions • Set of behaviors • Positive feedback generator • Negative feedback generator

6. The Learning Task The task: • Change the precondition list from each behavior to maximize relevance and reliability

7. The Learning Task Constraints: • Relevance: behavior correlated to positive feedback, not correlated with negative feedback • Reliability: behavior receives consistent feedback

8. The Learning Task More constraints: • Algorithm should deal with noise, • Perform in real time, • Support readaptation

9. The Learning Task Assumptions: • At least one combination of preconditions is bounded • Feedback is immediate • Only combinations of conditions can be learned

10. Algorithm Measure: • Number of times a positive/negative feedback did/didn’t happen when a behavior was/wasn’t active • Calculate the correlation between positive/negative feedback and the status of the behavior

11. Algorithm Measure: • Express relevance and reliability in terms of this correlation • Relevance controls whether a behavior should be active or not • Reliability decides whether the behavior should try to improve itself

12. Algorithm Measure: • Improvement is done by monitoring a perceptual condition • If reliability increases, the behavior is added to the list of preconditions • Keep monitoring in a circle until reaching the threshold

13. Genghis • Six-legged robot that walks forward • 12 behaviors, 6 conditions, 8742 nodes • 4 eight-bit microprocessors, 32 KB memory • The challenge is to learn how to coordinate the legs to produce a forward movement

14. Results Convergence time • Non-intelligent search during the monitoring stage: 10 minutes • Intelligent search: 1min 45sec • A “tripod” gait emerged which is common among six-legged insects

15. Conclusions • A learning algorithm was developed which allows a behavior-based robot to learn when its behaviors should become active using positive and negative feedback

16. Comments • Impressive results • Global behavior (walking) emerges from coordinated Behaviors • Simple idea, powerful consequences. Robot learned how to walk, wasn’t taught

17. Comments • Dead behaviors don’t revive. They might be useful in other situations • How to deal with concurrent actions? (i.e. walking and following a target)