An experiment in Security Decision Making

1. An experiment in Security Decision Making Adrian Baldwin, Yolanta Beres, Marco Casassa Mont, Simon Shiu (all HP Labs) Geoff Duggan, Hilary Johnson (University of Bath) Chris Middup (Open University)

2. Context • TSB funded trust economics project: • We developed an approach (using economic and mathematical modelling) to help enterprises make “better” security decisions • A series of case studies providing good feedback and anecdotal evidence that were on a good path • Challenge – can we do better than that? • This paper: • An in depth study of a small group of security professionals (one stakeholder type), on how our approach to security decision making affects them

3. Preferences Problem System Model Utility A rigorous approach to Security Decision making Problem Architecture components of utility things to measure problem refinement consequences of preferences

4. SDM Hypotheses Our methods will positively influence: • the conclusions or decisions made, • the thought process followed, • the justifications given, and • the confidence the stakeholder has in the final conclusions or decisions made.

5. SDM experiment scope • Measure effect on security professionals/experts (i.e. not our effect on other stakeholders nor groups/organisations) • Qualitative in depth study of decision making process (of twelve professionals) • Bundled economic framing and system modelling as a “single” intervention • Controlled experiment, i.e. two groups one intervened using our methods, one left as a control

6. The SDM problem • Chose a problem on the security of client infrastructure • Why – we had several similar case studies that meant we knew: • it was a representative current and challenging business security problem • we had decent/realistic empirical data relating to the problem • there are interesting “trade-offs” that meant the answer is subjective and contextual and likely to be different for different stakeholders • We had 4 decision options that represented different trade-offs • We had to iterate a number of times before we had sufficient supporting material and a problem we could control, and that was rich enough!

7. Experiment design 1. Session Introduction 2. Problem Description 3. Question & Answers 5a. Preference/ Economic Framing 4. Decision Options 5b. Modelling & Results 5. Question & Answers 6. Choice & Justification 7. Introspection

8. Phases Session introduction Problem description Q&A Decision options (a) Preference Elicitation(b) Model analysis Choice & Justification Introspection Experiment phases Options • Invest in patching • Invest in Host based intrusion prevention (HIPS) technology • Change policy to lock down (remove admin privileges) from users • Do nothing

9. Phases Session introduction Problem description Q&A Decision options (a) Preference Elicitation(b) Model analysis Choice & Justification Introspection Experiment phases (intervene phase only) Identify major outcomes (components of utility) Identify appropriate proxy metrics for each outcome Prioritise outcomes

10. Phases Session introduction Problem description Q&A Decision options (a) Preference Elicitation(b) Model analysis Choice & Justification Introspection Experiment phases Describe model of concurrent processes, and how options are explored Show (chosen proxy measure) results in 3*3 results tool

11. Data Analysis • 173 questions before intervention (from all twelve participants) • 152 justifications (from all twelve participants) • 6 ordered prioritised outcomes • 12 decision options • 48 Likert scores on confidence (four from each participant)

12. The choices • In the control group: 3 selected Lockdown, 2 selected HIPS and 1 selected Patching • In the intervention group: 3 selected Lockdown and 3 selected HIPS • A very security oriented set of options!

13. Categorization of questions Similar balance between groups

14. Categorization of justifications More balanced business justification for the intervened group

15. SDM Hypotheses Our methods will positively influence: the conclusions or decisions made, the thought process followed, the justifications given, and the confidence the stakeholder has in the final conclusions or decisions made. SDM results What do the data results say in relation to our original hypothesis Not sufficient evidence that we influenced conclusions or decisions made There is evidence we influenced the justifications given • Which in turn suggests we affected their thought processes There was a slight (but not significant) increase in confidence in decisions made

16. Some further analysis potential theoretical explanations NB on study style: smaller qualitative studies often fertile for early theoretical development Security priority in questions (and control group’s justifications) suggest presence of confirmation bias The intervened group’s broader justifications suggest our methods managed to counter some of this bias The intervened group did not value the economic framing • “i’d made those trade offs already” is at odds with this result - suggests cognitive dissonance

17. Conclusions & Next Steps • Encouragement that economic framing improves analysis • Assume that a study of group decision support would make this results stronger • Encouragement to use tools to support simultaneous comparison of multiple outcomes and choices • More cognitive science should be done to complement security economics • Future analysis • Study ‘question’ data to see methods/structure followed by security profession (compared with ISO27k, hunting for low hanging fruit, ...) • Future studies • To test the suggested theories • To explore the effect on multi-stakeholder decision making

18. Questions

19. Phases Session introduction Problem description Q&A Decision options (a) Preference Elicitation(b) Model analysis Choice & Justification Introspection Experiment phases (intervene phase only) Identify major outcomes (components of utility) Identify appropriate proxy metrics for each outcome Prioritise outcomes

20. Phases Session introduction Problem description Q&A Decision options (a) Preference Elicitation(b) Model analysis Choice & Justification Introspection Experiment phases Prioritise outcomes

21. Phases Session introduction Problem description Q&A Decision options (a) Preference Elicitation(b) Model analysis Choice & Justification Introspection Experiment phases Describe model of concurrent processes, empirical studies, and how options are explored

22. Phases Session introduction Problem description Q&A Decision options (a) Preference Elicitation(b) Model analysis Choice & Justification Introspection Experiment phases Show results in 3*3 (option to proxy measure) results tool

23. Phases Session introduction Problem description Q&A Decision options (a) Preference Elicitation(b) Model analysis Choice & Justification Introspection Experiment phases 10 minutes to ask any questions they deem relevant Scripted answers (e.g. on history, culture, processes, architecture, business, regulations etc…) Answers to “new” questions were added to the script for future sessions After 10 minutes we provided “essential” information that had not been asked about This allowed us to collect data on what questions were asked and in what order

24. Phases Session introduction Problem description Q&A Decision options (a) Preference Elicitation(b) Model analysis Choice & Justification Introspection Experiment phases Choose preferred option For each option: • Pro’s – reasons why option would be good • Con’s – reasons why option would be bad • Likert scale 1-7 confidence in the option

25. Phases Session introduction Problem description Q&A Decision options (a) Preference Elicitation(b) Model analysis Choice & Justification Introspection Experiment phases For intervened group • What difference the interventions and tools made What information they used to reach their conclusion Any strategies they used when asking questions

26. Phases Session introduction Problem description Q&A Decision options (a) Preference Elicitation(b) Model analysis Choice & Justification Introspection Experiment phases 3 Roles: interviewer, expert and observer Interviewer explained and gathered: • Structure of session • Incentives for trying hard • Experience of participant

27. Phases Session introduction Problem description Q&A Decision options (a) Preference Elicitation(b) Model analysis Choice & Justification Introspection Experiment phases Verbally scripted, web based and written material introducing them to the security role they are being asked to play and the client infrastructure security problem the CISO has. Whether/how to deal with rising risk from malware on client infrastructure

28. Data Analysis • All questions and justifications were transcribed and put in ‘random’ order • 3 experts categorised these – differences resolved through discussion • Relation to ISO 27000 • Relation to main business outcomes (compliance, productivity, cost, security risk)