MURI: Computer-aided Human Centric Cyber Situation Awareness

1. MURI: Computer-aided Human Centric Cyber Situation Awareness Peng Liu Professor & Director, Lions Center Pennsylvania State University ARO Cyber Situation Awareness MURI

2. Team • Peng Liu, Professor and Director, Penn State Center for Cyber-Security, Information Privacy and Trust • Massimiliano Albanese, Assistant Professor, GMU • Nancy Cooke, Professor and Science Director, Arizona State Cognitive Engineering Research Institute • Coty González, Associate Research Professor and Director, CMU Dynamic Decision Making Lab • Dave Hall, Professor and Dean, Penn State College of IST • Christopher Healey, Professor, NC State • Sushil Jajodia, University Professor and Director, George Mason Univ. Center for Secure Information Systems • Mike McNeese, Professor and Associate Dean, Penn State College of IST • Peng Ning (on leave), Professor, NCSU • Douglas Reeves, Professor and Interim Assistant Dean for COE Graduate Programs, NCSU • VS Subrahmanian, Professor and past Director, U. of Maryland Institute for Advanced Computer Studies • John Yen, University Professor and Director, Intelligent Agents Lab # of graduate students: 16 # of post docs: 3

3. ARO MURI: Computer-aided Human Centric Cyber Situation Awareness PSU, ASU, CMU, GMU, NCSU, UMD Contact: Peng Liu, Tel. 814-863-0641, E-Mail: pliu@ist.psu.edu Objectives: Improve Cyber SA through: • Cyber SA specific cognition models • Cognition-friendly tools and analytics that fill the gap between the sensor side and the analyst side of cyber SA • Cross-layer situation knowledge integration DoD Benefit: • Significantly improved capabilities in gaining cyber SA in face of cyber attacks • Significantly improved job performance of analysts Accomplishments • Year 5: See slide 5 Challenges • Understanding the mental processes of analysts • Team integration Scientific/Technical Approach • Take a holistic approach to integrate the “human cognition” aspects and the “cyber tools” aspects of cyber SA • Leverage cognition models to develop human cognition-friendly SA techniques, tools, and analytics

4. Cognitive Models & Decision Aids • Instance Based Learning Models • Simulation • Measures of SA & Shared SA • Information Aggregation & Fusion • Transaction Graph methods • Damage assessment • Automated • Reasoning • Tools • R-CAST • Plan-based narratives • Graphical models • Uncertainty analysis Data Conditioning Association & Correlation Multi-Sensory Human Computer Interaction Computer network • Enterprise Model • Activity Logs • IDS reports • Vulnerabilities Real World Computer network Security Analysts Test-bed

5. Year 5 accomplishments Research: -- Major achievements made -- See individual presentations Technology transitions: -- See slides later on Pub: -- 37 (1 book, 10 journals, 20 conf., 6 chapters) (Y1-Y5 total: 170+) -- 2 PhD thesis -- 7 presentations Tools: -- ARSCA  Shift Transition -- MetaSymploit -- NETS simulator -- DEXTAR -- Patrol  X-Ray -- Switchwall -- NSDMiner -- CyberCog -- PASS  PADUA -- CAULDRON -- etc. Deep collaboration with ARL: -- 11 ARL security analysts -- 5 researchers at ARL -- 4 joint publications Awards: -- Max Albanese received the 2014 Mason Emerging Researcher/Scholar/Creator Award

6. Cyber Operations for Mission Assurance • What has happened? • What is the impact? • Why did it happen? • What should I do? Sensors, probes Computer networks (e.g., GIG) Security Analysts

7. Cyber Situation Awareness What has happened? What is the impact? Why did it happen? What should I do? Enabler Core Cyber SA

8. Cyber SA Info Processing Box Attacks Depicted Situation The Network Compare Data Sources (feeds) Ground Truth (estimates) Job Performance

9. Why Research is Needed? 20+ CNDSPs*, whose operations are relying on human analysts, face critical challenges: • Job performance is unstable • Hard to get the big picture: walls between functional domains • Better analytics and tools are needed to improve job performance * In the commercial world, similar issues exist.

10. State of the Art: Big Gap Exists Current tools: Desired cyber SA capabilities: Vulnerability scan Event logging Traffic classifying Intrusion detection Alert correlation Signature gen. Taint analysis Back tracking Integrity check Static analysis Bug finding Attack graphs Symbolic execution Sandbox VM monitors … • Ability to create problem-solving workflows • To see big picture • To manage uncertainty • To reason albeit incomplete/noisy knowledge • To quickly locate needles in haystacks • To do strategic planning • To predict • … … BIG GAP

11. Scientific Objectives Develop a deep understanding on: • Why the job performance between expert and rookie analysts is so different? How to bridge the job performance gap? • Why many tools cannot effectively improve job performance? • What models, tools and analytics are needed to effectively boost job performance? Develop a new paradigm of cyber SA system design, implementation, and evaluation.

12. Scientific Barriers • Massive amounts of sensed info vs. poorly used by analysts • Silicon-speed info sensing vs. neuron-speed human cognition • Stovepiped sensing vs. the need for "big picture awareness" • Knowledge of “us” • Lack of ground-truth vs. the need for scientifically sound models • Unknown adversary intent vs. publicly-known vulnerability categories

13. Potential Scientific Advances Understand the nature of human analysts’ cyber SA cognition and decision making. Let this nature inspire innovative designs of SA systems. Break both vertical stovepipes (between compartments) and horizontal stovepipes (between abstraction layers). “Stitched together” awareness enables advanced mission assurance analytics (e.g., asset map, damage, impact, mitigation, recovery). Discover blind spot situation knowledge. Make adversary intent an inherent part of SA analytics.

14. Scientific Principles Cybersecurity research shows a new trend: moving from qualitative to quantitative science; from data-insufficient science to data-abundant science. The availability of sea of sensed information opens up fascinating opportunities to understand both mission and adversary activity through modeling and analytics. This will require creative mission-aware analysis of heterogeneous data with cross-compartment and cross-abstraction-layer dependencies in the presence of significant uncertainty and untrustworthiness. SA tools should incorporate human cognition and decision making characteristics at the design phase.

15. Why a Multidisciplinary Approach? Several fundamentally important research questions cannot be systematically answered by a single-disciplinary approach. See next slide.

16. Q1: What are the differences between expert analysts and rookies? Computer and Information Science of Cyber SA Q2: What analytics and tools are needed to effectively boost job performance? Q3: How to develop the better tools? Our focus Cognitive Science of Cyber SA Decision Making and Learning Science of Cyber SA

17. Technical Approach Draw inspirations from cognitive task analysis, simulations, modeling of analysts’ decision making, and human subject research findings. Use these inspirations to develop a new paradigm of computer-aided cyber SA Develop new analytics and better tools Let tools and analysts work in concert “Green the desert” between the sensor side and the human side Develop an end-to-end, holistic solution: In contrast, prior work treated the three vertices of the “triangle” as disjoint research areas

18. The proposed cyber SA framework It is a ‘coin’ with two sides: • The life-cycle side • Shows the SA tasks in each stage of cyber SA • Vision pushes us to “think out-of-the-box” in performing these tasks • The computer-aided cognition side • Build the right cognition models • Build cognition-friendly SA tools

19. Perception Comprehension Projection • Cognitive Models & Decision Aids • Instance Based Learning Models • Simulation • Measures of SA & Shared SA • Information Aggregation & Fusion • Transaction Graph methods • Damage assessment • Automated • Reasoning • Tools • R-CAST • Plan-based narratives • Graphical models • Uncertainty analysis Data Conditioning Association & Correlation Multi-Sensory Human Computer Interaction Computer network • Enterprise Model • Activity Logs • IDS reports • Vulnerabilities Real World Computer network Security Analysts Test-bed

20. Situation Knowledge Abstraction Perspective Mission Workflows App, Net Services Reeves Jajodia, Albanese Subrahmanian VulnerabilityExploits Alerts Gonzalez, Cooke Yen, Healey OS Liu: integration McNeese & Hall: multi-level cognition and fusion CPU

21. Impact on DoD Significantly enhance mission assurance through: • Significantly improving the job performance of CNDSPs 2. Developing cognition-friendly SA tools to effectively improve job performance • Situation knowledge integration • Situation knowledge discovery & elicitation • Reasoning assistants, decision aids • Better interfaces, better shift transitions

22. Y5 Team Integration Within each theme: • Collaboration is pervasive • Collaboration is further deepened • Joint research tasks • Co-authored papers • Tool-level integration in progress Between themes: • Integration along the functional perspective • Integration along the knowledge abstraction perspective • E.g., Jajodia & Cooke, Coty & Cooke, Hall & McNeese & Liu, Healey & Hutchinson, Yen & Cam & Erbacher & Glodek & Hutchinson & Liu, Jajodia & Albanese & Cam & Yen & Liu

23. Technology Transfer (1) Partner: Contact: Focus: Status: ARL Rob Erbacher, Bill Glodek, Steve Hutchinson, Hasan Cam, Renee Etoty, Chris Garneau Collect the cognitive traces of CNDSP analysts -- Over two years -- Over 30 traces collected -- ARSCA tool is being used at ARL -- Weekly teleconferences -- In discussion: directly operate on ARL datasets

24. Technology Transfer (2) Partner: Contact: Focus: Status: ARL Rob Erbacher, Bill Glodek, Steve Hutchinson Shift transitions -- A user study on shift transition fully designed -- IRB developed and approved -- ARSCA-shift-transition tool developed -- Shipped to ARL site and tested there -- Pilot study is being scheduled

25. Technology Transfer (3) Partner: Contact: Focus: Status: ARL Hasan Cam Enhance the ARL petri-net model for impact assessment -- feed outputs of CAULDRON and ARSCA into petri-net -- Proposal developed and approved -- Just started (Nov 2014) -- First experiment sketched

26. Technology Transfer (4) Partner: Contact: Focus: Status: ARL Rob Erbacher, Christopher Garneau (a) Investigate how the current practice of training professional CNDSP security analysts can be enhanced by leveraging ARSCA. (b) A pilot study for investigating the feasibility of using ARSCA-facilitated training procedures for supporting the training of analysts about their analytical reasoning process. -- Proposal developed and approved -- Just started (Nov 2014) -- Weekly teleconferences

27. Technology Transfer (5) Partner: Contact: Focus: Status: ARL Christopher Garneau, Rob Erbacher Human subject experiments on the cognitive effects of different (visualization) views -- IRB developed and approved -- User study fully designed -- Pilot study being scheduled at Penn State

28. Tech Transfer (6) Phase II STTR: Cooke group has been working with Sushil Jajodia and Max Albanese (George Mason and fellow MURI PIs) on an STTR that involves a higher fidelity version of CyberCog, DEXTAR, in which we will integrate CAULDRON. - Phase II STTR through Sandia Research Corporation - AFRL has shown interest in the test-bed we have been developing on the Phase II STTR Cooke group has been working on SBIR for AFRL with Charles River Associates that involves team sensors for cyber analysts.

29. Technology Transfer (7) Partner: Contact: Focus: Partners: Contacts: Focus: Partner: Contact: Focus: AFRL – Human Effectiveness Directorate 711th Human Performance Wing, Wright-Patterson AFB, OH Benjamin Knott and Vince Mancuso Human performance and measurement of cognition Deloitte, Ernst and Young, KPMG, Price Waterhouse Coopers J.B. O’Kane (Vigilant by Deloitte), Jenna McAuley (EY-ASC) and others Observe practicing analysts, test visualization toolkits and fusion tools, measure human cognition and performance MIT Lincoln LaboratoriesCyber Security Information Sciences Division Stephen Rejto and Tony Pensa Conduct human-in-the-loop experiments; evaluate MIT-LL/PSU analyst tools

30. Technology Transfer (8) Partner: Contact: Focus: Status: Partner: Contact: Focus: Status: Partner: Contact: Focus: Status: NIST Anoop Singhal Gain awareness of stealthy info bridges in a cloud -- One research work done -- One NIST technical report produced -- Paper published NEC Labs America, Inc. Z. Qian, Z. Li Discover long-running Idling processes in enterprise systems -- One research work done -- A real enterprise environment (on 24 hosts) -- In-depth measurement study -- Paper submitted IAI, Inc. Jason Li System call level enterprise cyber SA -- A new research work done -- One PhD dissertation

31. Tech Transfer (9) Ethnographic studies/knowledge elicitation with network analysts working in education, military, government, and industry domains. Briefings provided to several companies including: Deloitte, Lockheed Martin, Raytheon Corporation, MITRE, Computer Sciences Corporation, and MIT Lincoln Laboratory. Briefings to NSA, DTRA, ONR, DHS, and DoDII. Neville Stanton, University of Southampton is the developer of EAST modeling and is collaborating with Buchanan and Cooke on this form of modeling applied to cyber.

32. No-Cost Extension Plan Each PI has a research plan from their perspectives: see the individual presentations Set-aside project 1 with ARL Set-aside project 2 with ARL Team integration exercises will be held

33. Q & A Thank you.