Capturing Expert Knowledge for BDI-Based Behaviour Modelling

1. Capturing Expert Knowledgefor BDI-Based Behaviour Modelling Emma Norling Centre for Policy Modelling Based on a portion of my PhD work @ the University of Melbourne“Modelling Human Behaviour with BDI Agents”

2. Outline • Motivation • Background: BDI and models of expert behaviour • Some approaches to knowledge acquisition • A little more depth on a couple of approaches • Tailoring techniques for BDI-based modelling • Partial worked example • Knowledge acquisition for other frameworks

3. Motivation • Military simulation • Detailed models of individual behaviour • Subjects acting as experts in highly specialised domains • Range of purposes for models, including tactics analysis, equipment analysis, logistics planning • BDI-based models of behaviour already used extensively

4. Goals Plans Reasoner Beliefs Intentions What is BDI? • Beliefs – Desires – Intentions model of agency • Based in folk psychology, the way we think we think Goals Sensors AGENT Actions

5. Expert Behaviour • Knowledge acquisition and encoding is a difficult task • Modelling expert behaviour has additional complications: • Much of the expert’s knowledge is “second nature” to the expert • Person performing knowledge acquisition often has little or no experience of the task

6. Sources of Inspiration • Expert Systems • Participatory Modelling • Cognitive Task Analysis • Other subject-based techniques: • Personality profiling (Barry Silverman et al, U. Penn) • Activity traces (Georgeon et al, INRETS France) • Few published details

7. Applied Cognitive Task Analysis (ACTA) • Originally developed as a means of gathering task knowledge for the development of training programs and for task redesign • Series of semi-structured interviews • A preliminary interview, to develop a task diagram. • A knowledge audit, contrasting expert and novice behaviours. • One or more simulation interviews, resulting in mental maps of the incidents. • Purpose is to develop a picture of the whys behind the task, as opposed to a procedural description

8. Critical Decision Method (CDM) • Aim is to capture expertise by exploring an incident that displays expertise in action • Four “sweeps”: • Incident verification and selection. • Time line verification and decision point identification. • Deepening, and “the story behind the story.” • “What if...” queries, expert/novice differences, decision errors, etc.

9. A Tailored Approach • Direct observation, serving three purposes: • Pre-filtering the subjects to ensure they are suitable for the modelling, • Preparation of the interviewer with respect to the domain, and • Providing reference points for discussion in later interviews. • Development of a task diagram, as in ACTA • An iterative process of: • Expansion of each of the elements in the diagram through a combination of • probes as per the knowledge audit process of ACTA, • presentation of hypothetical situations, and • CDM sweeps using scenarios proposed by the subjects. • Analysing the data and designing the models.

10. Example • Work was motivated by modelling human behaviour in the military domain • Access to subjects and scenarios in this domain is difficult • Quake 2 provided a readily-accessible domain with an abundance of willing subjects for modelling

11. Developing a Task Diagram • When you play the game, do you perceive any distinct phases? • What are your main goals in each of these phases? • What are the relative priorities of these main goals? • Say you enter a new game, where you don’t know the world map, and you may know some, but not all, of the players. What are the first things that you do? • Do you make an effort to get to know the style of the other players? How do you use that knowledge?

12. Expanding the Task Diagram • You say the first stage of the game is when you don’t know the map. When do you consider that you do know the map? Do you explore every nook and cranny? • What makes a good sniping spot? • If you’d just respawned and you could hear but not see a fight nearby, what would you do? • How important are the sounds in the game to you? What sorts of things do you listen for? • What sort of things most clearly differentiate novice players from expert players? • Say you’d identified a particular opponent as being a better player than you. Would you make an attempt to actively avoid him/her? • When you were talking about exploring the map, you said you explore cautiously. What do you mean by cautiously?

13. Analysing the data • Aim was to identify • The goals of the subject, • The plans used by the subject, and • The beliefs of the subject, or more accurately, the things about which the subject had beliefs.

14. Interview Excerpt Subject: ...hearing doors open is a good one, or hearing lifts activate. Like for instance if you’re up the top on that level you can hear the lift coming before it gets there, so...Interviewer: Right, so you’re ready to shoot at them? Subject: As soon as you hear the lift activate, you can start lobbing grenades into it....Subject: One of the things that it’s common to do on this level... and... a lot of other levels, is to run across the lift, activate it so it comes up... umm... people will start... lobbing grenades in. Then if you’re quick enough you can actually run around the corner and umm... beat the lift to the top. And... shoot them as they’re lobbing grenades into the lift.

15. Leading to…

16. Further Iterations • “You said you used sounds like doors opening and lifts activating. Which other sounds do you pay attention to?” • “With your trick of activating a lift and then trying to run around and surprise a person lobbing grenades in, do you just assume that there will be someone there, or do you use other information to decide when/if to do this?” • “If there was more than one route that would get you to the top of the lift in time, how would you decide which one to use?”

17. More Generally… • This approach was designed particularly to complement the folk-psychological underpinnings of BDI • Also, it was designed to look at detailed models of individual behaviour • Could it be used more generally? • Yes, but probably still suited to detailed models • There is a difficulty translating human knowledge into code, caution must be used!