Automated Negotiation Agents

1. Automated Negotiation Agents Sarit Kraus Dept. of Computer Science Bar-Ilan University

2. Negotiation • “A discussion in which interested parties exchange information and come to an agreement.” — Davis and Smith, 1977

3. What is an Agent? PROPERTY MEANING • Situated Sense and act in dynamic/uncertain environments • Flexible Reactive (responds to changes in the environment) Pro-active (acting ahead of time) • Autonomous Exercises control over its own actions • Goal-oriented Purposeful • Persistent Continuously running process • Social Interacts with other agents/people • Learning Adaptive • Mobile Able to transport itself • Personality Character, Emotional state

4. No Agent is an Island: automated agents negotiate with other automated agents • Monitoring electricity networks (Jennings) • Distributed design and engineering (Petrie et al.) • Distributed meeting scheduling (Sen & Durfee, Tambe) • Teams of robotic systems acting in hostile environments (Balch & Arkin, Tambe, Kaminka) • Electronic commerce (Kraus et al.) • Collaborative Internet-agents (Etzioni & Weld, Weiss) • Collaborative interfaces (Grosz & Ortiz, Andre) • Information agent on the Internet (Klusch, Kraus et al.) • Cooperative transportation scheduling (Fischer) • Supporting hospital patient scheduling (Decker & Jin)

5. Agents negotiate with humans • Training people in negotiations • Trade agents for the Web • Elves agents– representing people

6. Plan of talk: agents negotiate with humans • Automated agent for bilateral negotiations with complete information: the fishing dispute (collaborators: Penina Hoz-Weiss, Jon Wilkenfeld) • Automated agent for multi-party negotiations: the Diplomacy game(collaborators: Daniel Lehmann and Eitan Ephrati) • On going work: learning, incomplete information; mediation(collaborators: Dudi Sarne, Barbara Grosz Lin Raz, Michal Halamish)

7. Fishing Dispute • Negotiators: Canada and Spain • Canada’s stock of flatfish decreases over the years. • Spain has fished this same stock of flatfish for many years, but outside the Canadian exclusive economic zone (EEZ). • Canada would like Spain to restrict its fishing near her EEZ. Spain is dependent on fishing in the area outside the EEZ for employment and trade purposes.

8. Possible Outcomes • An agreement on Total Allowable Catch (TAC). • An agreement on limiting the length of the fishing season. • Canada enforces conservation measures with military forces against Spain. • Spain enforces its right to fish throughout the fishery with military force againstCanada. • If the negotiation has not ended prior to the deadline, then it terminates with a status quo outcome.

9. World State Parameters • World state parameters are also negotiable and affect the utility of players: • Canada subsidizes removal of Spain's ships (0, 5, 10, 15, 20 ships). • Spain reduces the amount of pollution caused by the fishing fleet (0%, 15%, 25%, 50%). • Canada imposes trade sanctions on Spain. • Spain imposes trade sanctions on Canada

10. Fishing Dispute Outcomes TAC Limit Season Opt Out Status Quo World State Parameters Canada subsidizes Spain reduces Canada imposes Spain imposes ships Pollution Trade Sanctions Trade Sanctions

11. Negotiation Process • Each of the parties can make requests, threats, offers, conditional offers and counteroffers, as well as to comment on the negotiation. • The utility of each ending is affected by the period when the negotiation ended. • Canada loses over time since Spain continues to fish while negotiating. Spain gains over time for the same reason. • Spain  Thule Canada Ultima

12. Spain offers to set TAC at 44 thousand tons. S Canada offers to set TAC at 18 thousand tons. S C E Negotiations in the Fishing Dispute Spain asks that Canada compensate Spain for Spain’s restricted fishing practices by replacing the income of twenty ships.

21. Other Negotiations Games • Team Games (SPIRE); negotiations on coordination; exchange of information; finding solutions is complex • Competitive games: when agents can benefit from reaching an agreement (also in bilateral games). • Trade games: Monopoly, Traders of Genoa, Kohle, Kies & Knete,Treasure game • War games: Diplomacy, Risk • Crisis games: Hostage Crisis. • Semi-cooperative games: Color Trail, Majority Game

22. Chess • Programs play chess as well as people • Programs play chess in a way much different than people: they mainly search the game tree

23. A . . . B . . . A . . . . . . Final states -1 0 +1 Evaluation Search tree for Tic-Tac-Toe . . .

24. Fishing dispute vs. Chess • Type of game: crisis game vs. war game.Coordination game vs. zero sum game • Number of players: 2 • Moves: simultaneously+ negotiations vs. sequentially– need to reach an agreement. • Number of pieces to move: no pieces vs. one piece at a time • Information: Complete information. • Needed capabilities: Negotiation skills vs. strategic skills.

25. NEGOTIATIONS Game theory techniques: formalize the game; find an equilibrium; follow the equilibrium strategy. Market techniques. Appropriate for games of many players that can exchange similar items. Heuristics: domain specific; “advice” books; human like strategies Markov Decision Processes. Modeling the opponent Learning from DB Learning from experience CHESS Heuristic Search Playing Techniques

26. The Automated Negotiator Agent (fishing dispute) • The agent plays the role of one of the countries. • During the negotiation the agent receives messages, analyzes them and responds. It also initiates a discussion on one or more parameters of the agreement. • It takes actions when needed.

27. Nash Equilibrium • An action profile is an order set a=(a1,…,aN) of one action for each of the N players in the game. • An action profile a is a Nash Equilibrium(Nash 53) of a strategic game, if each agent j does not have a different action yielding an outcome that it prefers to that generated when chooses aj, given that every other player i chooses ai.

28. Strategy of Negotiation Formal strategic negotiation theory: The agent is based on the a bargaining model. By backward induction the agent builds the strategy to be reached at each time period according to the sequential equilibrium (Kraus, Strategic Negotiation in Multiagent Environments, MIT Press 2001). When the agent plays against humans Not Enough Heuristics

29. Automated agent: Using equilibrium strategy when playing against humans • Human negotiators do not use equilibrium strategies even though game is not complex and the automated agent finds equilibrium fast. • Not surprising: Kahneman & Tversky showed that humans do not use decision theory. • The agent using the equilibrium did not reach beneficial agreements.

30. Heuristics • Negotiation tactics • Attributes • Risk Attitude • Opting out • Fine tuning

31. Attributes • Number of points lower than the equilibrium utility value that the agent will agree to. • The number of fish ton (TAC) the agent will increase/decrease in his offer. • Sending the first message / waiting to receive a message. • Full offer message or not.

33. Modeling the risk attitude of theopponent • The agent is always neutral toward risk, but is sensitive to the risk level of its human opponent and will change its view of the human’s utility function accordingly. • Risk attitude influences the agreement an opponent is willing to accept. • The agent begins with the assumption that its opponent is risk neutral. It uses a heuristic method to decide whether to change the estimation of the risk attitude of the opponent. • When the agent decides that its opponent is risk prone, it changes the opponent’s utility function. This leads the agent to a recalculation of his strategy.

34. Experiments Results

35. Fishing Dispute: Conclusions • We developed an agent that can play well against a human player. • The agent was tested on students in their third year of computer science studies. • The results of the experiments implied that the agent plays well and fair. • It raised the sum of the utilities in the simulation it was involved in. • The agent played as Spain significantly better than a human did, and just as good as a human Canada player.

36. Diplomacy’s Rules • Each player represents one of seven European powers: England, Germany, Russia, Turkey, Austria-Hungary, Italy and France.

37. Diplomacy’s Map

39. Diplomacy’s Rules (Cont.) • Winner: The power that gains control over the majority of the board. • Beginning: 1901; two seasons a year. • A season: consists of a negotiations stage and a move stage. • Moves: All players secretly write the orders for all of their units simultaneously. • Negotiations: Coalitions and agreements among the players reached in the negotiations stage significantly affect the course of the game. The rules of the game do not bind a player to anything she says. Deciding who to trust as situations arise is part of the game.

40. If you support my attack on Vienna I will support your attack on Rumania F I know that Italy is going to attack Trieste R G E Negotiations in Diplomacy If you will not help me I will attack you Don’t trust Germany

41. Moves in Diplomacy • Only one unit may be in any space at one time. • A unit can be ordered to: move, support, hold (convoy). • An army or a fleet may support the move of another unit of that country or any other country in making a move. • Support can also be given on a defensive basis. • Opposing units with equal support do not move. An advantage of only one support is sufficient to win.

42. Moves in Diplomacy

43. The Need for Negotiations in Diplomacy • Moves require close cooperation between various allied powers. • Incomplete information: communications between players are done secretly. • The game is complex: 834 possible moves in each step of the game (without negotiation moves) . Negotiation is used to obtain information about the goals of the other players. • Others negotiate.

44. Diplomacy vs. Chess • Type of game: war games. • Number of players: 7 vs. 2 • Moves: simultaneous vs. sequential. • Number of pieces to move: all pieces vs. one piece. • Information: uncertainty about messages exchanged between other players vs. full information • Needed capabilities: negotiation skills vs. strategic skills.

45. NEGOTIATIONS Game theory techniques: formalize the game; find an equilibrium; follow the equilibrium strategy.Impossible in Diplomacy because of complexity. Market techniques. Appropriate for games of many players that can exchange similar items. Heuristics: domain specific; “advice” books; human like strategies Markov Decision Processes. Learning from DB Learning from experience CHESS Heuristic Search Playing Techniques

46. Diplomat: an Automated Diplomacy player Previous Agreements Board Status Beliefs on other players Analysis Others Moves Analysis & Strategies Finder Analysis & Strategies Finder Moves Detailed plans and their estimated value for possible coalitions Negotiations Agreements

47. Diplomacy Structure Secretary Prime Minister Front 2 Ministry Of Defense Front 1 Strategies Finder Front 3 Foreign Office Intelligence Military Headquarters Analyzer 13 Analyzer 14 Write orders 15 Desk 10 Desk 11 Desk 12 Write orders 15

48. Strategies Finder (SF) • Front: possible enemies and possible allies, e.g., Russia and Italy against Austria and Germany. • Diplomat’s strategy for a given front includes: • A list of orders associated with their purpose. • The expected average profit from carrying out the strategy for each power who participates in the strategy and the common expected profit for all of the powers. • A Venice (I) moves to Triests in order to attack Triest • A Vienna (R) supports A Venice to Trieste in order to attack Trieste ……… • Expected outcome: Aver: 10617 Min: 5002 Max: 20862 Russia: 3358 Italy 18117

49. Strategies Finder (SF) (Cont) • Diplomat identifies possible front based on on-going agreements, beliefs about other agents and their relations. • SF finds some strategies for each front using domain specific heuristics. The value of each strategy is computed by finding strategies for the enemies of the front. • The negotiation is done based on the identified strategies. • Question: What is the best strategy?

50. Diplomat’s negotiation Exchange information; Decide what kind of agreement to try to achieve. Find common enemies. Negotiating about the general purposes of an agreement: spaces on the board to attack, to defend, to leave or to enter. Signing the final Agreement; Deciding if to keep it. Deciding onthe specific movements in order to achieve the purposes From previous stage