Collaborative Time Sensitive Targeting aka Technologies for Team Supervision

1. Collaborative Time Sensitive TargetingakaTechnologies for Team Supervision March 31, 2008 Missy Cummings Humans & Automation Lab MIT Aeronautics and Astronautics http://halab.mit.edu

2. Review of Original Proposal Project Goal • Develop technological support for futuristic unmanned aerial vehicle (UAV) teams engaged in TST operations (but really any time critical resource allocation task) Technology to Support UAV Operators: Activity Awareness Displays Objective Performance Measures Technology to Support the Team Supervisor: Activity Awareness Displays Interruption Recovery Tools

3. Vehicle(s) Teamwork in UAV Operations UAV Operators Intelligence Consumers (e.g., Ground Troops) Ground Crew Operations Center

4. Generalizable to Other Domains 3 “Confederates”as UAV Operators

5. Team Interface Issues in Time-Critical Operations • Increasing reliance on collaboration tools • e.g., email, instant messaging (chat) • Increasedcommunication overhead • Need interfaces that intelligently share activity information to improve planning & coordination in networked teams • Activity awareness • Research for technology aids to assist supervisors of teams virtually non-existent • Measurement of technology interventions very difficult • Objective team performance metrics are elusive

6. Boeing Team Test Facility 3 “Confederates”as UAV Operators

7. Simulated Task Environment Surveillance support for a ground convoy through a hostile region • UAV team consists of: • 1 Mission Commander • 3 UAV operators, each controlling multiple UAVs • UAVs have camera sensors only • Team must coordinate with external strike team to destroy identified threats Situation Map Display Mission Status Display Remote Assistance Display

8. Team Supervision Decision Aiding UAV Operations Team Interruption recovery tools for the team supervisor

9. Interruption Recovery Assistance (IRA) tool • Interactive event timeline, containing event “bookmarks” • Increased recovery time but positive impact on decision accuracy, especially in complex task situations. • IRA tool tended to provide greater benefits to participants without military experience • Some experimental/interface issues so additional study needed • Summer 2008 w/ Waterloo

10. Team Current Work: Predictive Tool for Team Supervision UAV Team Supervisor Technologies for assessing: • How well is the team doing? • When & who should received attention?

11. Current Work: Team Performance Prediction • The need for more objective team performance metrics • Our focus is supervisory control in team settings with embedded layers of automation • Predicting team states versus actual performance • Artificial intelligence, behavioral pattern detection and performance correlations • Alert team supervising agent of sub-optimal team behavior/cognitive strategies • Relying on human knowledge-based reasoning to determine if an anomalous team state exists • Two stage approach • Predict an individual operator’s state transition as a proof of concept • Adapt model to UAV team setting

12. Team Prediction Model Characteristics • Evaluation and predictions of team behaviors: • Automatic, continuous, and in real-time • Only relies on easily observable data • User interaction with machine • User communication • Eye tracking? • Use Bayesian pattern prediction methods applied to individual/team behaviors • Hidden Markov Model • Not predicting performance per se • Anomalous conditions • Value of human judgment

13. Hidden Markov Models • Probabilistic state transitions • Observables vs. Hidden States • Hidden states are not directly visible, but variables influenced by them are. • Cognitive states vs. behaviors • HMM uses the observables to infer: • Most likely hidden state sequence • Based on future sequences, the most likely observables • Machine learns the clustering of behaviors (need lots of data!) Target Monitoring Modify Flight Parameters H&S Monitoring Camera Control Comms Waypoint Update Mission Replan

14. Predicting Individual Behavior

15. Human State Estimation - TRACS

16. Building the Model Grammar Pattern Recognizer/ Predictor ILDs Low Level Input Future Team Behavior

17. 4 State HMM for Mission Planning Task • Unsupervised vs. supervised model training for state recognition Leveraging Automation Manual Planning Exploration

18. Predicting an Operator’s Strategies

19. Live Demo!

20. Validating our HMM Approach • Currently only behaviors that entail mouse clicks are observed • A limit, particularly for monitoring interfaces • Eye tracking arguably gives us additional data about the information operators access • Noisy • Cost-benefit analysis? • Study will be conducted this summer comparing HMM results with and without eye tracking input

21. Extending our Approach to UV Teams Sector A Sector C Visualization task a) Replan Path b) Replan Target Sector B Evaluation/ Choice of UV + engage Extending to task to teams: UV hand-off between Sectors Choice of UV

22. Develop HMM using experimental data from individual experiments Investigate supervised vs. unsupervised learning Develop a team version of the interface Update HMM with team experiments to be conducted this summer The Plan

23. Additional Future Work: Designing for Team Activity Awareness UAV Operations Team

24. Preliminary UAV Operator Display Designs CommunicationsDisplay MapDisplay Tasking Display:TargetID & rerouting, reassigning UAVs

25. Deliverables • 6 Conference papers • ASNE, HSIS, ICCRTS, HFES • 4+ Technical reports • 3 Undergraduate theses • 2 pending • 1 Masters thesis • 1 PhD dissertation (pending) • 3 Journal articles in various stages • 1 workshop (CSCW 2006) • IP disclosure pending HMM validation • The Holy Grail of team research???

26. Future Plans • Near Term • Eye tracking study • HMM-Multi UV experiment • Extension for supervisor IRA tool • Mid Term • Activity awareness displays for operators • Decision support display for HMM • Multi-UV connection to predictive model system architecture research (ONR) • Development of CE metric • Project slated to end December 2008 • Requested no-cost extension to JUN 09 • Additional funding

27. Backup

28. Simulated Task Environment Display Detail Map Display Mission Status Display

29. Simulated Task Software Architecture Large-Screen Wall Displays Situation Map Display Mission Status Display Simulation & Collaboration Server(Grouplab SharedDictionary) TabletPC Display Mission Commander Display

30. Designing to Promote Activity Awareness • UAV status & tasking • Current & expected convoy safety • Current & expected operator task performance, relative to convoy safety 1 (reviewing ATR imagery) 4 (nominal) Situation Map Display 8 (down) Mission Status Display

31. Designing to Promote Activity Awareness • UAV status & tasking • Current & expected convoy safety • Current & expected operator task performance, relative to convoy safety Situation Map Display Potential threat envelope Target strike indicators Known threat envelope Mission Status Display

32. Designing to Promote Activity Awareness • UAV status & tasking • Current & expected convoy safety • Current & expected operator task performance, relative to convoy safety Situation Map Display Mission Status Display

33. Developing Methodology for Deriving Collaborative System Requirements Project Goal: • To develop techniques to identify dependencies in operator decision making to understand how to assist coordination of team member tasking