Modeling and Measuring Situation Awareness in Individuals and Teams

1. Modeling and Measuring Situation Awareness in Individuals and Teams Cleotilde Gonzalez In Collaboration with: Lelyn Saner, Octavio Juarez, Mica Endsley, Cheryl Bolstad, Haydee Cuevas, and Laura Strater

2. Computational Models of SA • Individual aspects of SA • Design aspects of SA • Organizational aspects of SA Measures of SA • Individual SA • Shared SA Conclusions Agenda

3. Situation Awareness • the Perception of the Elements in the Environment within a Volume of Time and Space, • the Comprehension of their Meaning, and • the Projection of their Status in the Near Future. • Formation of SA influenced by: • Individual abilities • Interactions with others • Environment

4. Integrated theory of mind: ACT-R (Anderson & Lebiere, 1998) • Shared attention (Juarez & Gonzalez, 2003, 2004) • Learning theory (Gonzalez, Lerch & Lebiere, 2003; Gonzalez & Lebiere, 2005) • Representation of Recognition (Gonzalez & Quesada, 2003) • Learning and decision making in dynamic systems (Gonzalez et al., 2003; Martin, Gonzalez & Lebiere, 2004) Micro and Macro Cognition: Convergence and Constraints Revealed in a Qualitative Model Comparison (Lebiere, Gonzalez & Warwick, 2009) Computational Cognitive Models

5. Computational Cognitive Models

6. A SA meta-architecture provided a full set of cognitive models interacting with OTB, and resulting in the “commander’s SA” (Gonzalez et al., 2004; Juarez & Gonzalez, 2003)

7. Computational Models of Design Aspects of SA (Juarez & Gonzalez, 2006)

8. Computational Models of SA • Individual aspects of SA • Design aspects of SA • Organizational aspects of SA Measures of SA • Individual SA • Shared SA Conclusions Agenda

9. Individual Measures of SA: SAGAT • Situation Awareness Global Assessment Technique (SAGAT) • Human-in-the-loop simulation exercises • Use of SAGAT queries (from GDTA) • Stop at random times and query the user • Compare response with reality of the situation • Examples: What is the aircraft altitude? • What is the aircraft activity in this sector (en route, inbound to airport, outbound to airport) • Which aircraft will need a new clearance to achieve landing requirements? • SAGAT score: accuracy of the responses

10. Individual SA measures, learning and working memory • Can we learn to be aware? Effects of task practice and working memory influence situation awareness (SA) - Gonzalez & Wimisberg, 2007 • How do we measure individual SA • Queries may be answered while the simulation display is not visible or covered (Endsley, 1995) or while the display is visible, uncovered (Durso et al., 1995).

11. Methods • The design was a 2 x 18 mixed design. Participants were randomly assigned to one of two conditions (covered or uncovered display) and they were asked to run the simulation 18 times (trials). • Individuals were asked to answer SA queries while the simulation was paused • Participants took the Visual Span Test (VSPAN) (Shah & Miyake, 1996).

14. Results

18. The correlation between SA scores and VSPAN decreased over time SA scores were higher in the uncovered condition than in the covered condition • This is due mostly to perception The effect of practice was significant only in the covered condition, but not in the uncovered condition Summary of results relevant for individual measures of SA

19. Measures of Shared SA (Saner, Bolstad, Gonzalez & Cuevas, in press; Saner, Bolstad, Gonzalez & Cuevas, in preparation) • Ground • TRUTH • X1 • X2 • X3 • X4 • X5 • X6 • X7 • Person 1 • X1 • X3 • X4 • X6 • X7 • X8 • X9 • Person 2 • X1 • X2 • X4 • X6 • X8 Shared SA-the degree to which team members possess the same SA on shared SA requirements (i.e. on the information that they both need to know) (Endsley,1995, 1995b; Endsley & Jones, 2001) A good measure of shared SA needs to account for the ACCURACY

20. SimQ1 SimQ2 SimQ3 SimQ4 SimQ5 SimQ6 SimQ7 Shared SA Person 1 Q1 Q2 Q3 Q4 Q5 Q6 Q7 Situation Awareness Global Assessment Technique (SAGAT) - Seven queries while task is stopped - Objective knowledge of situation • Person 2 • Q1 • Q2 • Q3 • Q4 • Q5 • Q6 • Q7 Score Similarity = 1-absolute value of [(p1-p2)/(p1+p2)] Range from 0 to 1 A good measure of shared SA needs to account for the SIMILARITY

21. Method • Training at Joint Personnel Recovery Agency (JPRA) - JFCOM • 16 servicemen, 3 DoD contractors; Age M=33.85 • Randomly assigned to one of four Teams: • Navy, Army, Special Operations, or Joint Service • Utilized Cross-Training • Five scenarios over 3 days • Each scenario had 3 to 12 incidents • Scenarios randomly stopped 3 times for SAGAT, Communication, and Workload measures • Received training prior to the exercise

22. Special Operations Cell (p13, p14, p15, p16, p17) Joint Service Cell (p1, p2, p3, p4) Army Cell (p5, p6, p7, p8) Navy Cell (p9, p10, p11, p12) Methods and Procedure • Joint Personnel Recovery Agency (JPRA) training exercise • Four team groups (i.e. cells) • Five Predictors of Shared SA • Experience Similarity- years in real service • Shared JPRA Knowledge- prior experience with recovery operations • Shared Cognitive Workload- subjective ratings, five NASA-TLX scales • Communication Distance- inverse frequency of communication • Organizational Hub Distance- degree of dissociation from Joint Service Cell

23. Possible Models Expected Classic Hierarchy

24. Results

25. Conclusions – Measures of Shared SA • Development of a Shared SA measure must account for both, accuracy and similarity of SA between members of an organization • As shared knowledge increased, so did shared SA. • Organizational Hub Distance (OHD) is key predictor • Physical Distance and JointCell Membership • Unexpected Role of OHD • Participants processed new information directly

26. Possible Models We observed that being in branch cells was associated with higher SSA rather than being in the joint cell  Observed Expected

27. The success of Computational Models of SA, depends on appropriate and robust measures of individual and shared SA • Although individual measures and procedures exist, there is a huge need for defining the methods and procedures for measuring SA at the team level We investigated measures of SA at both, the individual and team levels • We created a shared SA measure that builds on individual SA Computational models of both, SA and SSA can incorporate these measures. Conclusions