Decision Making AI

1. John See 20 Dec 2010 Decision Making AI

2. Decision Making AI • Ability of a game character to decide what to do • Decision Making in Millington’s Model

3. Decision Making AI • Decision Trees • Finite State Machines (FSM) • Rule-based Systems • Fuzzy Logic & Neural Networks • Blackboard Architecture

4. Decision Trees • Fast, easy to understand • Simplest technique to implement, but extensions to the basic algorithm can be sophisticated • Typically used to control characters, animation or other in-game decision making • Can be learned, and learning is relatively fast (compared to fuzzy logic/NN)

5. Decision Trees – Problem Statement • Given a set of knowledge, we need to generate a corresponding action from a set of actions • Map input and output – typically, a same action is used for many different sets of input • Need a method to easily group lots of inputs together under one particular action, allowing the input values that are significant to control the output

6. Decision Trees – Problem Statement • Example: Grouping a set of inputs under an action Enemy is visible Enemy is now < 10m away Attack Enemy is visible Enemy is still far (> 10m), but not at flank Attack Enemy is visible Enemy is still far (> 10m), at flank Move Enemy is not visible, but audible Creep

7. Decision Trees – Algorithm Overview • Made up of connected decision points • Tree has starting decision, its root • For each decision, starting from the root, one of a set of ongoing options is chosen. • Choice is made based on character’s knowledge (internal/external)  Fast! No prior representation! • Continues along the tree, making choices at each decision node until no more decisions to consider • At each leaf of the tree, an action is attached • Action is carried out immediately

8. Decision Trees • Check a single value and don’t contain any Boolean logic (AND, OR) • Representative set • Boolean – Value is true • Enumeration – Matches one of the given set of values • Numeric value – Value is within given range • 3D Vector – Vector has a length within given range • Examples?

9. Decision Trees – Combining Decisions • AND two decisions – place in series in the tree • OR two decisions – also use decisions in series, but the two actions are swapped over from the AND

10. Decision Complexity • Number of decisions that need to be considered is usually much smaller than number of decisions in the tree. • Imagine using IF-ELSE statements to test each decision? • Method of building DTs: Start with simple tree, as AI is tested in game, additional decisions can be added to trap special cases or add new behaviors

11. Decision Making - Branching • So far, we have considered only binary trees – decisions choose between 2 options. • It is possible to build DT with any number of options, or different decisions with different number of branches

12. Decision Making - Branching • Deep DTs may result in a same alert being checked numerous times before a decision is found • Flat DTs are more efficient, requires less decision checking

13. Decision Making - Branching • Still common to find DTs using only binary decisions • Why? • Underlying code for multiple branches simplifies down to a series of binary tests (IF-ELSE statements) • Binary DTs are easier to optimize. Some learning algorithms that work with DTs require them to be binary

14. Decision Making – Performance • Takes no memory, performance is linear with number of nodes visited • Assume each decision takes constant amount of time, and tree is balanced, performance: O(log2 n), where n is number of decision nodes in tree • This DOES NOT consider the execution time of the different checks required in the DT, which can vary a lot!

15. Decision Making – Balancing the Tree • DTs can run the fastest when atree is balanced • A balanced tree keeps the height of its branches approximately equal (within 1 to be considered balanced). In our context, it will have about the same number of decision making levels on each branch

16. Decision Making – Balancing the Tree • 8 behaviors, 7 decisions • 1st tree – extremely unbalanced, 2nd tree – balanced • To get to H, 1st tree needs 8 decisions, 2nd tree needs 3 only

17. Decision Making – Balancing the Tree • If all behaviors are equally likely, what is the average number of decisions needed for both trees?

18. Decision Making – Balancing the Tree • If we were likely to end up at decision A majority of time, which is more efficient? • How do we treat decisions that are time-consuming to run? Let’s say A is the most time-consuming decision…

19. Decision Making – Merging Patterns • DTs can be extended to allow multiple branches to merge into a new decision – efficient, but some care is needed! • A decision/action can be reached in more than one way • Avoid causing loops, can never find an action leaf

20. Random Decision Trees • To provide some unpredictability and variation to making decisions in DTs • Simplest way: Generate a random number and choose a branch based on its value • DTs are normally intended to run frequently, reacting to the game state, random decisions can be a problem • What is a potential problem with the following DT?

21. Random Decision Trees Possible considerations… • Allow random decision to keep track of what it did last time. • When a decision is considered, a choice is made at random, and that choice is stored. Next time the decision is considered, no more randomness, previous choice is maintained. • If something in the world changes, a different decision was arrived, the stored choice should be removed. • Timing Out: Allow the AI to “time out” after a set time, and a random choice is to be made again. Gives variety and realism.

22. State Machines • Often, characters in a game act in one of a limited set of ways • Carry on doing the same thing until some event or influence makes them change • Can use decision trees, but it is easier to model this behavior using state machines (or finite state machines, FSM) • State machines take into consideration • the world around them • their internal state

23. State Machines – Basics • Each AI character occupies one state at each instance • Actions or behaviors are associated with each state • So long as the character remains in that state, it will continue carrying out the same actions/behavior • States are connected by transitions • Each transitions leads from one state to another, the target state, and each has a set of associated conditions • Changing states: when the game determines that conditions of a transition are met, the conditions trigger and a new state is fired

24. State Machines – Simple Example • State machine to model a soldier – 3 states • Each state has its own transitions • The solid circle (with a transition w/o trigger condition) points to the initial state that will be entered when the state machine is first run

25. State Machines vs. Decision Trees • Now, name some obvious differences in making decisions using decision trees and state machines?

26. Finite State Machines (FSM) • In game AI, a state machine with this kind of structure (as seen earlier) is usually called a finite state machine (FSM) • An FSM has a finite number of states and transitions • It has finite internal memory to store its states and transitions

27. FSM – Generic Implementation • Use a generic state interface that keeps track of a set of possible states and records the current state it is in • With each state, a series of transitions are maintained. Each transition is also a generic interface with conditions • At each iteration (game loop), an update function is called to check if any of the transitions from the current state is triggered. • If a transition is triggered, then the transition will be fired • The separation of triggering and firing of transitions allows the transitions to have their own actions

28. FSM – Generic Implementation • Refer to textbook or other references for a more in-depth code-level implementation of FSMs

29. FSM – Complexity • State machines only requires memory to hold a triggered transition and the current state • O(1) in memory and O(m) in time, where m is the (average) number of transitions per state • The algorithm calls other supporting functions to perform action and etc. These probably account for most of the time spent in the algorithm. • Hard-coded FSM – inflexible, does not allow level designers the control over building the FSM logic

30. Hard-coded FSM • Hard-coded FSM – • Consists of an enumerated value, indicating which state is currently occupied, and a function that checks if a transition is followed • States are HARD-CODED, and limited to what was HARD-CODED • Pros – Easy and quick implementation, useful for small FSMs • Cons: • Inflexible, does not allow level designers the control over building the FSM logic • Difficult to maintain (alter) – Large FSMs, messy code • Every character needs to be coded its own AI behaviors…

31. Hierarchical State Machines • One state machine is a powerful tool, but will still face difficulty expressing some behaviors • Also if you wish to model somewhat different behaviors from more than one state machines for a single AI character • Example: Modeling alarm behaviors with hierarchical s/m (using a basic cleaning robot state machine)

32. Hierarchical State Machines • If the robot needs to get power if it runs out of power & resume its original duties after recharging, these transition behaviors must be added to ALL existing states to ensure robot acts correctly

33. Hierarchical State Machines • This is not exactly very efficient. Imagine if you had to add many more concurrent behaviors into your primary state machine?

34. Hierarchical State Machines • A hierarchical state machine for the cleaning robot • Nested states – could be in more than one state at a time • States are arranged in a hierarchy  next state machine down is only considered when the higher level state machine is not responding to its alarm

35. Hierarchical State Machines • H* - “history state” node • When the composite state (lower hierarchy) is first entered, the H* node indicates which sub-state should be entered • If composite state already entered, then previous sub-state is restored using the H* node

36. Hierarchical State Machines • Hierarchical state machine with cross hierarchy transition • Most hierarchical s/m support transitions between levels of the hierarchy • Let’s say we want the robot to go back to refuel when it does not find any more trash to collect…

37. Hierarchical State Machines • Refer to textbook for more details on its implementation • Performance: • O(n) in memory (n is number of layers in hierarchy) • O(nt) in time, where t is number is number of transitions per state

38. DT + SM • Combining decision trees and state machines • One approach: Replace transitions from a state with a decision tree • Leaves of DT (rather than straightforward conditions/actions) are now transitions to other states

39. DT + SM • To implement state machine without decision tree transitions… • We may need to model complex conditions that require more checking per transition • May be time-consuming as need to check all the time

40. Fuzzy Logic • Founded by LotfiZadeh (1965) • “the essence of fuzzy logic is that everything is a matter of degree” • Imprecision in data… • and uncertainty in solving problems • Fuzzy logic vs. Boolean logic • 50%-80% less rules than traditional rule-based systems, to accomplish identical tasks • Examples: Air-conditioner thermostat or washing machine

41. Fuzzy Logic in Games • Example of uses: • To control movement of bots/NPCs (to smooth out movements based on imprecise target areas) • To assess threats posed by players (to make further strategic decisions) • To classify player and NPCs in terms of some useful game information (such as combat or defensive prowess)

42. Crisp data & Fuzzy data • Crisp data (real numbers, value) • Fuzzy data (a predicate or description, with degree value) • Fuzzy logic gives a predicate a degree value. Instead of belonging to a set of being excluded (1 or 0, Boolean logic), everything can partially belong to a set, and some things more belong than others

43. Fuzzy sets • Fuzzy sets – the numeric value is called the degree of membership (these values are NOT probability values!) • For each set, a degree of membership of 1 given to something completely in the set. Degree membership of 0 given to something completely outside the fuzzy set • Typical to use integers in implementation instead of floating-point values (between 0 and 1), for fast computation in game • Note: Anything can be a member of multiple sets at the same time

44. Fuzzy Control / Inference Process • 3 basic steps in a fuzzy control or fuzzy inference process

45. Step 1 - Fuzzification • Mapping process – converts crisp data (real numbers) to fuzzy data (degree of membership) • E.g.: Given a person’s weight, find the degree to which a person is underweight, overweight or at ideal weight

46. Membership Functions • Membership functions map input variables to a degree of membership, in a fuzzy set between 0 and 1 • Any function can be used, and the shape usually is governed by desired accuracy, the nature of problem, or ease of implementation. • Boolean logic m/f

47. Membership Functions • Grade m/f • Reverse grade m/f • Triangular m/f • Trapezoid m/f

48. Membership Functions • Earlier example of using a set of membership functions to represent a person’s weight

49. Step 2 – Fuzzy rule base • Once all inputs are expressed in fuzzy set membership, combine them using logical fuzzy rules to determine degree to which each rule is true • E.g. • Given a person’s weight and activity level as input, define rules to make a health decision • If overweight AND NOT active then frequent exercise • If overweight AND active then moderate diet • But having a fuzzy output such as “frequency exercise” is not enough – need to quantify the amount of exercise (e.g. 3 hours per week)

50. Fuzzy rules • Usually uses IF-THEN style rules • If A then B • A  antecedent / premise • B  consequent / conclusion • To apply usual logical operators to fuzzy input, we need the following fuzzy axioms: • A OR B = MAX(A, B) • A AND B = MIN(A, B) • NOT A = 1 – A