Learning & User Modeling

1. Learning & User Modeling Presentation By Brigette Swan

2. What Is Good AI? • Display of realistic behavior when reacting to the player. • Able to challenge the player both tactically and strategically.

3. But What About… • Adaptation to the quirks and habits of a particular player over time. • At the moment, many games only implement difficulty sliders.

4. Standards? • This is a coarse level of control at best. It forces the player to estimate his or her own abilities, which can often be incorrect. • What happens if one mode is too easy, but the next is too hard?

6. Why? • Learning algorithms are computationally expensive, enough to make game programmers run screaming. • Is there a method that is not so daunting, and yet still gives a realistic feel to player-computer interaction? • Enter….

7. The Player Model • Something like a profile, the player model is a set of demographics on a particular individual’s skills, preferences, weaknesses, and other traits. • The model can be easily updated as the game progresses, whenever the AI interacts with the player.

8. The Player Model • The game AI can then query the user model, allowing it to select reactions and tactics that would best challenge the player’s personal style. • The profile could be for a single play, or even be updated over multiple sessions.

9. Model Design • The model is a collection of numerical attributes, representing the traits of the player. • Each attribute is an aspect of player behavior, which can be associated with strategies, maneuvers, or skills.

10. Model Example

11. Model Example • This particular trait does not take into account the player’s behavior relative to the game context. • Models can hold detail like this, but the more detailed the information, the longer the profile takes to design, and the more computationally expensive the updates become.

12. At the most basic level, the player model is a statistical record of the frequency of some subset of player actions. Class PlayerModel { public: enum ETrait { kUsesSmokeGrenades, kAlwaysRuns, kCanDoTrickyJumps }; void Initialize(); void UpdateTrait(ETrait trait, float observe value); float GetTrait(ETrait trait); private: vector<float> _traitValues; }; Model Implementation

13. Model Implementation • Each trait is represented by a floating-point value between 0 and 1, where 0 roughly means “the player never does this” and 1 means “the player always does this.” • Each trait is initialized to 0.5 to reflect the lack of knowledge.

14. The update method is based on the least mean squares (LMS) training rule, often used in machine learning. float PlayerModel::UpdateTrait(ETrait trait, float observeValue) { float currValue = _traitValues[trait]; float delta = observeValue – currValue; float weightedDelta = kLEARNINGRATE * delta; traitValues[trait] += weightedDelta; } Model Implementation traitValue = α · observedValue + (1- α) · traitValue

15. 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.55 0.45 0.5 0.5 0.5 0.62 0.38 0.54 0.56 0.5 0.62 0.38 • New Game User Model Set To Default UsesSmokeGrenades: FleesFromBattle: UsesTrickyJumps: UsesRangedAttacks: UsesCloseAttacks: Ninja Vanish!

16. 1.0 0.56 0.3 1.0 0.25 UsesSmokeGrenades: FleesFromBattle: UsesTrickyJumps: UsesRangedAttacks: UsesCloseAttacks: + =

17. Model Updates • In order for the user model to be effective, the game must be able to update the profile. This requires two steps. • The game must detect when an update should be made. • The game must then tell the model to update itself.

18. Model Updates • While the latter is easy, with the function just displayed, the former can prove a more daunting task. • The game must be able to recognize that the player has taken, or failed to take, an action, or sequence of actions, which correspond to a certain trait.

19. Model Updates • For example, the “CanDoTrickyJumps” trait, might be triggered if the player can navigate across rooftops. Thus, the game must have a mechanism to determine that the player had made, or failed, such a jump. • This detection can be hardcoded into the jump routines.

20. Model Updates • What about a trait like “DetectsHidingEnemies”? • In a game where enemies hide in shadows and around corners, the game must compute the visibility of every nearby enemy. This also must include the processing of scene geometry.

21. Model Updates • The game might be able to make use of the AI as it does all these calculations for interaction, or the keep a cache of past computations to help reduce the cost of the algorithm. • A queue for player actions allows background processing as a solution.

22. Model Updates • Once the game has detected a need for an update, a value is assigned to the action and placed into the profile. • For example, a player that successfully makes the difficult jump, as previously described, would receive an update like this. model.UpdateTrait(PlayerModel::kCanDoTrickyJumps,1.0);

23. Alternatives? • While hard-coded trait detection routines are straightforward, it leaves the modeling code scattered throughout the source, which would make maintenance a headache and a half. • Also, modifying the structure of the player model in any way requires changing the code, and therefore, rebuilding the game.

24. FSM Updates • Trait detection routines can also be implemented as a collection of finite state machines. Each concrete trait is associated with a single FSM which contains the entire detection algorithm. • The FSMs monitor the stream of AI and player actions, and update the model whenever necessary.

25. FSM Updates

26. FSM Updates • The FSM system is designed to be data-driven, and thus, can be separated from the source code and loaded into the game dynamically. • Removing the player model from the source code removes it from the build process, allowing it to be altered without a time-consuming recompilation.

27. Hierarchy • User models can also be designed in a hierarchy, using a tree. Each leaf node represents one of the concrete traits that we have been discussing. • These nodes are clustered under particular abstract traits, which allows the AI to generalize when selecting a strategy against a player.

29. Player OnePrimary Abstract Trait: MeleeAttacker Enemy AI ? Player TwoPrimary Abstract Trait: RangedAttacker Secondary Abstract Trait: Healer

30. Hierarchy • The hierarchical model is slightly more complex to implement, but it not difficult, resembling a binary tree in execution. • This design also makes it simple to maintain a hashtable to the various leaf traits, speeding up the search for the GetTrait() process.

31. Using The Model • The player model exists to make the AI less episodic and more aware of past interactions. • From the player’s perspective, it appears as though his enemies have gotten smarter.

32. Using The Model • The profile can also be used by the AI to exploit weaknesses in a player, to give them more of a challenge. • In a friendly game, such demographics could be used to provide helpful advice to a novice player.

33. In Games Today… • Rare are the games that implement user models well, when they do at all. At the moment, it is difficult to say when player modeling is being used, and when it is not. • But we have a few guesses.

34. In Games Today… • Sports: Recording play styles of users is important in sports style games. The NFL 2k series actually experimented with player models, but according to most players, they were implemented rather poorly.

35. In Games Today… • FPS: Adaptive enemies would be an asset to games like Splinter Cell: Pandora Tomorrow. Developers and designers always seem to have grand plans, but reviews indicate mediocre execution at best.

36. In Games Today… • RTS: While Homeworld does not implement the exact type of player model described here, it does compensate based on the user’s skills. Since the player keeps his units from the previous encounter, the enemy force is based on the number and type of units that the player has at the beginning of the next mission.

37. In Games Today… • Fighters: This game style does some minor profiling of its players. In tournaments, the AI creates a single session model, allowing the opponents to learn and counter the player’s favored movement patterns. The difficulty slider is used to determine the learning rate of the AI.