Analyzing the Risks of Information Security Investments with Monte-Carlo Simulations

1. Analyzing the Risks of Information Security Investments with Monte-Carlo Simulations WEIS05 Workshop on the Economics of Information Security James R. Conrad, University of Idaho Department of Computer Science conr2286@uidaho.edu

2. Contents Introduction to the Problem The Monte-Carlo Solution Overview of Monte-Carlo Simulations Example Analysis and Critique Conclusions

3. Introduction to the Problem • An information security investment may need to compete for resources with other business opportunities • But many information security models rely upon experts’ estimates • And the experts’ estimates may include significant uncertainty • How can the analyst communicate an opportunity when so much is uncertain?

4. Monte-Carlo Solution • Apply the Monte-Carlo technique to simulate and express uncertainty in information security models • This is not a new model --- this is an enhancement of existing models • While less common in the Computer Science discipline, many financial decision makers are already familiar with the Monte-Carlo approach

5. Monte-Carlo Simulations • Specify uncertainty in probability distributions • Monte-Carlo engine samples distributions • Engine executes the security model once for each of several thousand iterations • Monte-Carlo engine captures and collects the result of each iteration distributions engine model results

6. Monte-Carlo Simulations • Engine simulates uncertainty in the model parameters • Model continues to operate with discrete values • Extra complexity largely confined to the Monte-Carlo engine • Results can be charted as probability distributions distributions engine model results

7. Monte-Carlo Example • Based upon Longstaff et al’s example appearing in “Are we Forgetting the Risks of Information Technology?” of IEEE Computer, December 2000 • Simulates the benefit/cost ratio of a proposed infosec investment for a financial enterprise • Modeling parameters are similar to Longstaff’s example with an added complication… • …The experts don’t agree!

8. intrusion rates Original (pre-Monte-Carlo) Parameters & Model other parameters Intrusion Rate Parameters r1 2 Simulated annual intrusion count w/o investment e 5.00E-01 Effectiveness of investment r2 =r1*e Annual intrusion count with investment Other Parameters p1 =r1/365 Daily probability of intrusion w/o investment p2 =r2/365 Daily probability of intrusion with investment X $20,000,000,000,000 Asset value y1 $100,000,000 Cost of software assurance w/o investment y2 $200,000,000 Cost of software assurance with investment z1 1.00% Losses w/o investment z2 0.50% Losses with investment Model Calculations d1 =p1*z1 Calc damage w/o investment d2 =p2*z2 Calc damage with investment D =y2-y1 Calc cost to provide software assurance with investment d =d1-d2 Calc percentage of losses prevented by investment b =d*X-D Calc net benefit of investment bcr =b/D Calc benefit/cost ratio for investment (bcr=7.22) model benefit/cost ratio, bcr

9. Uncertainty in the Revised Example • Consider a case in which the experts don’t agree upon an single value estimate for the annual intrusion rate (fixed at r1=2 events/year in the original problem) • The hypothetical disagreement stems from uncertainty in anticipated business practices • Experts do agree there exists a 20% chance that business practices will change in a way that will raise the intrusion rate to 20 events/year and an 80% chance that those practices will remain unchanged

10. Uncertainty in the Revised Parameters • Model variability of optimistic intrusion rate as a Poisson process (for purposes of this example), ro=randpoisson(2) • Model variability of pessimistic intrusion rate as a Poisson process, rp=randpoisson(20) • Model uncertainty of anticipated business conditions by choosing the optimistic rate 80% of the time and the pessimistic rate 20% of the time using randdiscrete(0.80,0.20,ro,rp) • Variability refers to a truly random process • Uncertainty refers to the experts’ inability to anticipate future business conditions

11. intrusion rates Revised Params & Model other parameters Intrusion Rate Parameters ro =randpoisson(2) Optimistic annual intrusion count w/o investment rp =randpoisson(20) Pessimistic annual intrusion count w/o investment r1 =randdiscrete(0.8,0.2,ro,rp) 80% Chance of ro. 20% Chance of rp. e 5.00E-01 Effectiveness of investment r2 =r1*e Annual intrusion count with investment Other Parameters p1 =r1/365 Daily probability of intrusion w/o investment p2 =r2/365 Daily probability of intrusion with investment X $20,000,000,000,000 Asset value y1 $100,000,000 Cost of software assurance w/o investment y2 $200,000,000 Cost of software assurance with investment z1 1.00% Losses w/o investment z2 0.50% Losses with investment Model Calculations d1 =p1*z1 Calc damage w/o investment d2 =p2*z2 Calc damage with investment D =y2-y1 Calc cost to provide software assurance with investment d =d1-d2 Calc percentage of losses prevented by investment b =d*X-D Calc net benefit of investment bcr =b/D Calc benefit/cost ratio for investment model benefit/cost ratio, bcr

12. Simulation of Revised Example • randpoisson() and randdiscrete() sample the probability distributions in each iteration of the simulation • The Monte-Carlo engine recalculates the model for each iteration and captures the results (bcr) • The Monte-Carlo engine charts the captured simulation results (next slide)

13. Simulation Results

14. Why not use a weighted average of r1 and r2? • Why doesn’t the revised model simply compute a weighted average of the two possible intrusion rates? • r1 = randpoisson(2)*0.8+randpoisson(20)*0.2 • The randdiscrete() simulation preserves the bimodal nature of the experts’ disagreement. • Any attempt to “average away” that uncertainty conceals the truth: The experts don’t agree.

15. Analysis • The results reflect the experts’ strong preference for the optimistic intrusion rate in which the benefit/cost ratio remains unchanged at 7.22. Risk-tolerant decision makers might manage to this value. • The mean value lies at 22 between the two modes. • The results also reflect a second mode at about 81 along with a 10% chance of the benefit/cost ratio exceeding 81. Risk-adverse decision makers might manage to this value to avoid a catastrophe “on their watch.”

16. Critique • But are real experts willing to provide even more estimates? • The author’s industry experience with Monte-Carlo models is that many experts are relieved to disclose the uncertainty they know to be in their estimates • What real experts truly dislike is being held accountable to an expected value they know is merely representative of the possibilities

17. Additional Critique • Given a tool to express uncertainty as probability distributions, which distributions closely model the empirical evidence? • How to extend the Monte-Carlo approach to graphical models?

18. Conclusions • Monte-Carlo techniques offer an approach to simulate uncertainty in expert estimates • Enables the use of probability distributions for model parameters and forecast results • The Monte-Carlo engine simulates random variables, allowing a security model to continue to manipulate discrete values with only minimal changes • May be particularly useful for visualizing the potential of an extreme event, the unlikely possibility of a catastrophic outcome

19. Questions and Optional Slides

20. Why Poisson Distribution? • The example problem uses a Poisson process to approximate intrusion attempts • If and/or when the Poisson process usefully reflects empirical intrusion attempts is an open question • Review: Models the number of events occurring during a specified time interval for a Poisson process • Review: Continuous opportunity for independent events to occur • Review: Long-term rate is constant • Review: Used to model lightening strikes in a storm

21. Correlated Parameters • “Every iteration of a… model must be a scenario that could physically occur.” -- Vose. • The parameters must “make sense” to the security model! • One correlated parameter can usually be expressed as a function (relation) of another. • Consider r1 and r2 in the example. These are likely related which is why r2 is calculated as a function of r1. • If the relationship (e) between r1 and r2 is also uncertain, this too can be simulated.

22. Variability and Uncertainty • Yes, this example lumped (simulated) variability and uncertainty together for simplicity • Vose (Risk Analysis, 2000) offers an excellent treatment of this subject for those who need to keep them separated

23. Partitioning • Yes, partitioning is an alternative technique • The Monte-Carlo technique might be viewed as an automated approach to partitioning • …and the Monte-Carlo technique avoids the subjective choice of partition boundaries • …and the Monte-Carlo technique has commercial tool support for systems-level models.

24. Commercial Tools • Yes, commercial off-the-shelf tools are available • They are most useful for systems-level security models. • They are less useful for low-level combinatorics security models • Search for “monte carlo simulation” and pay particular attention to the “Sponsored Links”

25. Performance • The author’s industry experience includes Monte-Carlo simulations using “hundreds” of random distribution parameters • Yes, they required several hours to run… • In 1997! • My computer is more than 10X faster today. • Simulation multiplies model complexity by n, the number of iterations. A simulation of an O(m2) model becomes n*O(m2). • Opportunities for parallel approaches when n cannot be ignored.