Market-Driven Multi-Agent Collaboration in Robot Soccer Domain

1. Market-Driven Multi-Agent Collaboration in Robot Soccer Domain

2. Today’s Presentation • Multi-Agent Systems • Robot Soccer • The Market Methodology • Market-Driven Approach • Reinforcement-Based Market-Driven Approach • A “New Approach”

3. Multi-Agent Systems

4. Multi-Agent Systems • Why use multi-agent systems? • Multi-agent systems are becoming more popular than complex single agent systems because they eliminate the problem of single point of failure.

5. Multi-Agent Systems • How do they work? • Multi-agent systems work by decomposing a complex task into several low-level actions which can then be assigned to the individual team members.

6. Multi-Agent Systems • How to assign tasks? • This is a key problem, the system must break up the tasks and coordinate the team such that the team collectively completes the overall task.

7. Multi-Agent Systems • How to assign tasks? • The system must keep track of each robot’s capabilities (trivial in a homogeneous team, but more complicated in a heterogeneous team)

8. Robot Soccer

9. Robot Soccer • Problem Domain? • We will look at robot soccer as the problem domain as it provides a very good real world domain for developing multi-agent systems.

10. Robot Soccer Domain • Robot soccer is a well-defined environment which provides a good test-bed for developing multi-agent strategies. Each robot has simple, clearly defined actions available and the overall task easy to understand –Beat the other team.

11. Robot Soccer Domain • Robot soccer provides a good way of comparing two systems/strategies. The two systems can simply be played against each other and see which team wins the most matches.

12. The Problem • We need a way of coordinating the robots to each perform a task/fulfil a role (ie attack, support, defend, goalie etc). • The Market-Driven Approach for coordinating the multi-agent system is based on the way free-markets maximize profits.

13. The Market Methodology

14. The Market Methodology • The main goal in free-markets is the maximization of the overall profit. The theory is that if each participant in the market tries to maximize its profit, the overall profit should increase.

15. Market-Driven Approach

16. The Market-Driven Approach • The Market-Driven Approach splits up the main task into simple tasks and an auction is then held for each task. The robots work out the cost for them to perform a task and then put in their best bid to the auctioneer. The robot which puts in the lowest bid gets the assignment.

17. The Market-Driven Approach

18. The Market-Driven Approach • In Robot Soccer an auction is held for each of the different roles. The robots calculate the cost of fulfilling those roles (based on distance to ball etc) and bid on them. The robots with the best bid on each role will be assigned the role.

19. The Market-Driven Approach • Two (or more) robots may get the same assignment where they must cooperate to perform the task (ie a robot with the ball attacks the goal and another robot supports it by driving close behind)

20. The Market-Driven Approach • An advantage of the Market-Driven Approach is that each robot calculates the cost of performing each role and communicates that cost to the other robots. This cost value is much easier and quicker to communicate rather than sending all of the metrics to the other robots.

21. The Market-Driven Approach • What about how the auction is run? • Centralized • Distributed • Hybrid

22. Centralized • There exists a master agent (auctioneer) that controls the auctions and assigns the roles. • The master agent receives offers from all other agents for each task and sends the auction results back. • Computationally efficient. • Prone to single point failures.

23. Distributed • No master agent. • Every agent broadcasts its offer for every task. • Every agent runs the same auction mechanism and parallely computes the auction results. • Robust against single point failures • Requires more computation in total.

24. Hybrid • There exists a master agent • There is also an auction for the task of being the master • Robust against single point failures • Computationly efficient • Still not implemented, no test results.

25. The Market-Driven Approach • Problem – How to calculate the costs? • Each robot must be able to calculate the cost of filling a particular role. The settings for the cost calculations must be calibrated, the performance of the system depends on the calibrations being correct. • Eg. - Cattacker = M2*distBall + M2*distOppGoal

26. Reinforcement-Based Market-Driven Approach

27. Reinforcement Learning • Reinforcement-Based Market-Driven Approach makes use of Reinforcement Learning (RL) to learn the role assignment process. RL is used when the agent is informed about the consequences of its actions. RL replaces the role assignment as described above.

28. Reinforcement Learning • With the RL system, the robot closest to the ball assigns itself as the attacker and the remaining agents (excluding the static goalie) assign themselves according to a state vector. (see next slide)

29. Reinforcement Learning • The Rules: • Goalie is statically assigned. • The Robot closest to the ball is assigned the role of attacker. • The other robots are assigned roles by a state vector. • State vector metrics – distances to the ball, goals, robots, the cost values and the closest player to the ball.

30. Reinforcement Learning Broadcast Position and Cost Data Calculate Attack Cost Array Calculate Defence Cost Array Closest to Ball Cheapest Yes Yes Shoot No No Role Assigned According to Cost Value Pass To Cheapest

31. New Approach

32. New Approach • The New Approach is effectively a simplified version of the Reinforcement Learning system. However instead of using the exact positions of the robots, the field is divided into a grid.

33. New Approach

34. New Approach • The system can now use this grid to make a decision on what role the robot should be performing. To assign roles, the system uses a state vector with the following metrics: Ball Position (grid number), Ball Possession, Current Role assigned by the Market-Driven strategy, Teammate positions and Opponent positions.

35. New Approach • This approach combines the Market-Driven Approach and the Reinforcement Learning based team with the grid separation of the board to keep the number of variables in the state vector to a minimum.

36. Results

37. Results • The New Approach which combines the Market-Driven, RL and grid system out performs all of the other teams consistently over 90 matches.

38. Questions?

39. References Kose, H., Kaplan, K., Mericli, C., Tatlidede, U. & Akin, L. (2005). Market-Driven Multi-Agent Collaboration in Robot Soccer Domain. Cutting Edge Robotics, 407-416. Kurt, B. (2007). Bogazici University Robotics Server. Retrieved September 09, 2007, from http://robot.cmpe.boun.edu.tr/robsem/ailab_market.ppt