680 likes | 967 Vues
Cognitive Task Analysis for Teams. Nancy J. Cooke New Mexico State University CTA Resource On-line Seminar October 11, 2002. Acknowledgements. NMSU Faculty : Peter Foltz NMSU Post Doc: Brian Bell
E N D
Cognitive Task Analysis for Teams Nancy J. Cooke New Mexico State University CTA Resource On-line Seminar October 11, 2002
Acknowledgements • NMSU Faculty: Peter Foltz • NMSU Post Doc: Brian Bell • NMSU Graduate Students: Janie DeJoode, Jamie Gorman, Preston Kiekel, Rebecca Keith, Melanie Martin, Harry Pedersen • US Positioning, LLC: Steven Shope • UCF: Eduardo Salas, Clint Bowers • Sponsors: Air Force Office of Scientific Research, Office of Naval Research, NASA Ames Research Center, Army Research Laboratory Nancy Cooke
Overview • What is team cognition? • Q&A • “Shared” mental models • Q&A • Holistic CTA for teams • Conclusions • Q&A Nancy Cooke
What is Team Cognition? Nancy Cooke
Team Cognition in Practice Nancy Cooke
Experimental Context CERTT (Cognitive Engineering Research on Team Tasks) Lab A Synthetic Task Environment for the Study of Team Cognition Five Participant Consoles Experimenter Console Nancy Cooke
Defining Team “…a distinguishable set of two or more people who interact dynamically, interdependently, and adaptively toward a common and valued goal/object/mission, who have each been assigned specific roles or functions to perform, and who have a limited life span of membership” Salas, Dickinson, Converse, and Tannenbaum (1992) Nancy Cooke
Defining Team Cognition • It is more than the sum of the cognition of individual team members. • It emerges from the interplay of the individual cognition of each team member and team process behaviors Nancy Cooke
Team Cognition Framework Individual knowledge Team Process Behaviors Team Knowledge Team Performance Nancy Cooke
Team Cognition Framework Collective level + + Individual knowledge Team Process Behaviors Holistic Level Team Knowledge Team Performance Nancy Cooke
Team Knowledge • Long-term knowledge • Taskwork • Teamwork • Fleeting Knowledge (i.e., momentary understanding, situation model) • Taskwork • Teamwork Nancy Cooke
Measurement Limitations • Measures tend to assume homogeneous teams • Measures tend to target collective level • Aggregation methods are limited • Measures are needed that target the more dynamic and fleeting knowledge • Measures are needed that target different types of long-term team knowledge • A broader range of knowledge elicitation methods is needed • A need for streamlined and embedded measures • Newly developed measures require validation Nancy Cooke
Other Related Work • Group Think (Janis, 1972) • Distributed Cognition (Hutchins, 1991) • Common Ground in Discourse (Clark & Schaefer, 1987; Wilkes-Gibbs & Clark 1992 ) • Group Decision Support (Fulk, Schmitz, & Ryu, 1995) • Social Decision Schemes (Davis, 1973; Hinsz, 1999) • Transactive Memory (Wegner, 1986) • Shared Mental Models (Cannon-Bowers, Salas, & Converse, 1993) Nancy Cooke
Why Do We Care? • Outcome measures of team performance do not reveal why performance is effective or ineffective • Team cognition is assumed to contribute to team performance • Understanding the team cognition behind team performance should facilitate interventions (design, training, selection) to improve that performance Nancy Cooke
Team Cognition and Functions of Cognitive Task Analysis • Elicitation: Interviews, observations, think aloud used to make knowledge explicit • Assessment: Judgments are made regarding specific elicited knowledge (e.g., accuracy, intrateam similarity) • Diagnosis: Patterns in elicited knowledge (i.e. symptoms associated with dysfunctional or exceptional performance) are tied to a diagnosis Nancy Cooke
Questions or Comments? Nancy Cooke
Shared Mental Models Nancy Cooke
“Shared Mental Models” Shared Mental Models Shared Knowledge Nancy Cooke
“Shared” Sharing = to have compatible knowledge Sharing = to have the same knowledge “Shared beliefs” “Share the pie” vs. To hold in common To distribute Nancy Cooke
The “Apples and Oranges” Problem Measures to assess team knowledge often assume knowledge homogeneity among team members. • Shared knowledge = similar knowledge • Accuracy is relative to single referent Person A Person B Referent Nancy Cooke
Teams, by Definition, Consist of “Apples and Oranges” Airport Incident Command Center Telemedicine Nancy Cooke
“Shared” Knowledge Knowledge Base Person A Person B Shared = Common Nancy Cooke
“Shared” Knowledge Shared = Complementary Nancy Cooke
“Shared” Knowledge Shared = Common and Complementary Nancy Cooke
“Shared” Knowledge Common and Complementary Knowledge and Shared Perspectives/Varied Granularity Nancy Cooke
“Shared” Knowledge Conflicting Knowledge Irrelevant Knowledge No Coverage Common and Complementary Knowledge and Shared Perspectives Nancy Cooke
An Approach to the Apples and Oranges Problem Measures of team knowledge with heterogeneous accuracy metrics Nancy Cooke
Experimental Context • Five studies: Two different 3-person tasks: UAV (Uninhabited Air Vehicle) and Navy helicopter rescue-and-relief • Procedure: Training, several missions, knowledge measurement sessions • Manipulate: co-located vs. distributed environments, training regime, knowledge sharing capabilities, workload Nancy Cooke
Experimental Context MEASURES • Team performance: composite measure • Team process: observer ratings and critical incident checklist • Other: Communication (flow and audio records), video, computer events, leadership, demographic questions, working memory • Taskwork & Teamwork Knowledge, Situation Awareness Nancy Cooke
Long-term Taskwork Knowledge • Factual Tests • Psychological scaling The camera settings are determined by a) altitude, b) airspeed, c) light conditions, d) all of the above. How related is airspeed to restricted operating zone? Nancy Cooke
Long-term Teamwork Knowledge Given a specific task scenario, who passes what information to whom? Teamwork Checklist ___AVO gives airspeed info to PLO ___DEMPC gives waypoint restrictions to AVO ___PLO gives current position to AVO AVO= Air Vehicle Operator PLO = Payload Operator DEMPC = Navigator Nancy Cooke
Team Situation Awareness • Assess accuracy and similarity of situation models of team members • SPAM (Situation Present Assessment Method) queries--display not interrupted • Queries about future events • Team members queried in random order at designated point in scenario within a 5-minute interval Durso, et al., 1998 How many targets are left to photograph? Nancy Cooke
Traditional Accuracy Metrics Team Referent .50 Team Member: Air Vehicle Operator 50% ACCURACY Nancy Cooke
Heterogeneous Accuracy Metrics AVO Referent DEMPC Referent PLO Referent Team Referent .33 1.0 0 .50 ACCURACY Overall: .50 Positional: 1.0 Interpositional: .17 Team Member: AVO AVO= Air Vehicle Operator PLO = Payload Operator DEMPC = Navigator Nancy Cooke
Results Across Studies • Taskwork knowledge is predictive of team performance But… • True for psychological scaling, not factual tests • Timing of knowledge test is critical Nancy Cooke
Knowledge Profiles of Two Tasks Knowledge profile characterizing effective teams depends on task (UAV vs. Navy) Nancy Cooke
Knowledge Profiles of Two Tasks Complementary Common UAV Task Command-and-Control Interdependent Knowledge sharing Navy Helicopter Task Planning and execution Less interdependent Face-to-Face Nancy Cooke
Knowledge Acquisition Training Mission Experience Procedure: Taskwork Knowledge Knowledge Acquired: Teamwork Knowledge Teamwork knowledge is acquired through mission experience and its acquisition seems dependent on a foundation of taskwork knowledge acquired in training. Nancy Cooke
Results: Team Situation Awareness • Team SA mirrors the performance acquisition function and generally improves with mission experience • Team SA is generally good predictor of team performance (especially a repeated query) SA and Performance data from first UAV study. Nancy Cooke
Implications of Heterogeneous Metrics • Can deal with “apples and oranges” issue • Can assess knowledge underlying task performance • Knowledge profiles of tasks can inform training and design interventions Nancy Cooke
Future Directions on Apples and Oranges Problem • Apply metrics to fleeting knowledge • Embed knowledge measures in task • Need a taxonomy of tasks and additional profile work • Need to connect the knowledge profile (symptoms) to diagnosis of team dysfunction or excellence Nancy Cooke
Questions or Comments? Nancy Cooke
Holistic CTA for Teams Nancy Cooke
Team Cognition Framework Collective level + + Individual knowledge Team Process Behaviors Holistic Level Team Knowledge Team Performance Nancy Cooke
The “Sum of All Team Members” Problem Team Process Behaviors Team Knowledge Team Performance Individual knowledge Collective level + + The Problem: Measures are taken at the individual level and aggregated, as opposed to being taken at the holistic level. Holistic Level Nancy Cooke
The Sum of All Team Members Problem • Aggregating individual data is problematic given the apples and oranges problem • Team process behavior is missing from collective measures • Cognition at the holistic level should be more directly related to team performance Nancy Cooke
Our Approach to the Sum of All Team Members Problem • Consensus assessment tasks • Consensus concept ratings • Consensus teamwork checklist • Consensus SA queries • Communication as a measure of team cognition Nancy Cooke
Consensus Assessment Tasks An Example: Concept Ratings • Step One: Individual Concept Ratings collected • Present to each individual: • airspeed – altitude (1=related, 5=unrelated) • Responses: • AVO=4, PLO=1, DEMPC=5 • 2) Consensus Ratings Collected • Present to the team: • airspeed – altitude (1=related, 5=unrelated) • Prior responses: AVO=4, PLO=1, DEMPC=5 • Team discussion: PLO: “Well I said related since my camera settings for shutter speed and focus are dependent on each of these values” DEMPC: “OK, let’s go with that 1 it is” AVO= Air Vehicle Operator PLO = Payload Operator DEMPC = Navigator Nancy Cooke
Consensus Assessment Tasks Results • Consensus measures correlate moderately with performance compared to collective measures • Perhaps consensus does not adequately tap in-mission process behavior • Although collective measures and process behaviors predict team performance for co-located teams better than holistic measures, this is not true for distributed teams Nancy Cooke
Communication as a Window to Team Cognition The “Good” • Observable • Team behavior diagnostic of team performance • Think aloud “in the wild” • Reflects team cognition at the holistic level • Rich, multidimensional (amount, flow, speech acts, content) Nancy Cooke