Improving Collaboration in Unmanned Aerial Vehicle Operations

1. Improving Collaboration in Unmanned Aerial Vehicle Operations March 27, 2007 Stacey D. Scott Humans & Automation Lab MIT Aeronautics and Astronautics http://halab.mit.edu

2. Shadow UAV Teamwork in Unmanned Aerial Vehicle (UAV) Operations UAV Operators Intelligence Consumers (e.g., Ground Troops) Ground Crew Lawyers & Politicians

3. Supporting Futuristic UAV Teams Engaged in Time-Critical Operations Future UAV Teams

4. Current Collaboration Tools Insufficient for Time-Critical Operations • Increasing reliance on collaboration tools • e.g., email, instant messaging (chat) • These tools introduce communication overhead, hindering management of tasking & team coordination (Scott et al., 2006) • Trend in corporate and educational collaboration systems towards intelligently sharing activity information to improve planning & coordination in networked teams • Activity awareness design approach (Carroll et al., 2006) • Promoting activity awareness in UAV operator interfaces may: • help UAV teams coordinate their actions • reduce the costs of obtaining activity-related information

5. Research Goals: Supporting Activity Awareness in Collaborative UAV Operations • To develop and evaluate activity awareness tools for future teams of UAV operators engaged in time critical missions • To investigate the software automation and interface requirements of UAV operations team members • what type of information is useful to know about one’s team members and their task activities? • what information do team members need to know about the UAVs’ activities? • how does one’s job duties / team role impact activity awareness requirements (e.g. mission commander, UAV operator, etc.)?

6. Initial Research Focus Future UAV Teams Technological Support for the Team Supervisor

7. Assisting Team Supervisors of UAV Teams Research Approach: • Develop representative collaborative, time-critical C2 task scenario • Conduct cognitive task analysis (CTA) to generate system requirements for supervisory decision support and activity awareness displays • Develop supervisory displays in a simulated task environment • Conduct exploratory experiment

8. Representative Collaborative UAV Task Scenario Ground Force Protection: • Futuristic UAV Operations Team performing ISR in support of a time-sensitive targeting for a ground convoy traversing a hostile region Task Details: • UAV Operations Team: • 1 Mission Commander • 3 UAV operators, each controlling multiple semi-autonomous UAVs • UAVs have camera sensors only • Team must coordinate with external strike team • UAV team can communicate with convoy & external intelligence sources

9. Hybrid Cognitive Task Analysis (Nehme et al., 2006): 1 2 4 3 Generating Requirements for Assisting Supervision Step 1: Scenario Task Overview Step 2: Event Flow Diagram Step 3: Situation Awareness (SA) Requirements Produces Functional & Information Requirements Step 4: Decision Ladders & Display Requirements

10. Supervisory Displays & Simulated Task Environment Large-Screen Wall Displays Situation Map Display Mission Status Display Simulation & Collaboration Server(Grouplab SharedDictionary) TabletPC Display Mission Commander Display

11. Designing to Promote Activity Awareness • UAV status & tasking • Current & expected convoy safety • Current & expected operator task performance, relative to convoy safety 1 (reviewing ATR imagery) 4 (nominal) Situation Map Display 8 (down) Mission Status Display

12. Designing to Promote Activity Awareness • UAV status & tasking • Current & expected convoy safety • Current & expected operator task performance, relative to convoy safety Situation Map Display Potential threat envelope Target strike indicators Known threat envelope Mission Status Display

13. Designing to Promote Activity Awareness • UAV status & tasking • Current & expected convoy safety • Current & expected operator task performance, relative to convoy safety Situation Map Display Mission Status Display

14. Exploratory Study Subjects: 8 participants (including 6 NROTC) Protocol: • 1 subject + 3 “confederates” per experiment • 2 hrs training + 2 x 15 min trials • Retrospective interview Situation Map Display Mission Status Display 1 Subject as mCdr 3 “Confederates”as UAV Operators

15. Some Preliminary Results • Convoy threat summary intuitive, useful for timing command decisions • Particularly, visualizing relationship between strike team activity and convoy’s progress • UAV status & tasking useful for deciding when to assist UAV operators • However, lack of time-on-task cues required mental effort • Operator performance displays rarely used, since surveillance progress was easily obtained from the Map 1 4 8 (reviewing ATR imagery) (nominal) (down)

16. Ongoing Work: Test-bed for Exploring Other Collaborative TST Issues UAV Operations Team 1. Interruption recovery tools for the team supervisor

17. Ongoing Work: Test-bed for Exploring Other Collaborative TST Issues 2. Activity awareness interfaces for the UAV operators 3. New methods for deriving collaborative system requirements UAV Operations Team 1. Interruption recovery tools for the team supervisor

18. Expanded Simulated Task Environment Large-Screen Wall Displays Situation Map Display Mission Status Display Remote Assistance Display Simulation & Collaboration Server(GroupLab SharedDictionary) Mission CommanderDisplay (TabletPC)

19. Interruption Recovery Assistance for Team Supervisor Interactive event timeline, containing event “bookmarks” Mission Commander Display(TabletPC)

20. Interruption Recovery Assistance for Team Supervisor (cont’d) Clicking on an event bookmarkhighlights the location of the corresponding event on the map display, e.g., a convoy attack

21. Preliminary UAV Operator Display Designs CommunicationsDisplay MapDisplay Tasking Display:TargetID & rerouting, reassigning UAVs

22. Improved Methodology for Deriving Collaborative System Requirements Project Goal: • To develop techniques to identify dependencies in operator decision making to understand how to assist coordination of team member tasking

23. Is the team doing well? Is Operator using “good” strategies? Operator Supervisor Future Work: Team Performance Prediction • Behavioral pattern detection and performance predictions • Predictive algorithms (NN, HMM, Hybrid) • Alert team supervising agent of sub-optimal cognitive strategies • Stress & time pressure • Rapidly evolving situation Excessive workload

24. Is the team doing well? Is Operator using “good” strategies? Team Operator Supervisor Future Work: Team Performance Prediction • Behavioral pattern detection and performance predictions • Predictive algorithms (NN, HMM, Hybrid) • Alert team supervising agent of sub-optimal cognitive strategies • Stress & time pressure • Rapidly evolving situation Excessive workload

25. Conclusions • Initial activity awareness designs show potential for facilitating supervisory-level decision making, and supporting teamwork in general • Particularly, designs aimed at conveying current & expected status of interdependent activities • Activity awareness approach (i.e. intelligent sharing of activity data) appears to be an appropriate design technique for large-screen displays in time-critical C2 environments • Further work is needed to understand what activity information should be shared (and at what level of detail) during different UAV mission types and mission phases

26. Contact and Sponsor Information Contacts: • Dr. Stacey Scott, sdscott@mit.edu • Prof. Mary (Missy) Cummings, missyc@mit.edu • Humans and Automation Lab (HAL) website: http://halab.mit.edu Thanks to project sponsor: • Boeing Phantom Works