Enhancing Error Control in Probabilistic Model Checking Algorithms

1. Error Control forProbabilistic Model Checking Håkan L. S. Younes Carnegie Mellon University

2. Contributions • Framework for expressing correctness guarantees of model-checking algorithms • Enables comparison of different algorithms • Improves understanding of sampling-based algorithms • New sampling-based algorithm for probabilistic model checking • Better error control through undecided results Error Control for Probabilistic Model Checing

3. Probabilistic Model Checking • Given a model M, a state s, and a property , does  hold in s for M? • Model: stochastic discrete event system • Property: probabilistic temporal logic formula arrival departure q “The probability is at least 0.1 that the queuebecomes full within 5 minutes” Error Control for Probabilistic Model Checing

4. Temporal Stochastic Logic (CSL) • Standard logic operators: ,  , … • Probabilistic operator: ≥ [] • Holds in state s iff probability is at least  for paths satisfying  and starting in s • Until: ≤T  • Holds over path iff  becomes true along  within time T, and  is true until then Error Control for Probabilistic Model Checing

5. Property Example • “The probability is at least 0.1 that the queue becomes full within 5 minutes” • ≥0.1[≤5full] Error Control for Probabilistic Model Checing

6. Possible Results ofModel Checking • Given a state s and a formula , a model-checking algorithm A can: • Accept as true in s (s) • Reject as false in s (s) • Return an undecided result (sI) • An error occurs if: • A rejects  when  is true (false negative) • A accepts  when  is false (false positive) Error Control for Probabilistic Model Checing

7. Ideal Error Control • Bound on false negatives:  • Pr[s|s ]   • Bound on false positives:  • Pr[s|s ]   • Bound on undecided results:  • Pr[sI]   Error Control for Probabilistic Model Checing

8. Unrealistic Expectations 1 –  –  s≥ [] s≥ [] Probability of acceptingP≥[] as true in s  p  Actual probability of  holding Error Control for Probabilistic Model Checing

9. Temporal Stochastic Logic with Indifference Regions (CSL) • Indifference region of width 2 centered around probability thresholds • Probabilistic operator: ≥ [] • Holds in state s if probability is at least + for paths satisfying  and starting in s • Does not hold if probability is at most − • “Too close to call” if probability is within  distance of  Error Control for Probabilistic Model Checing

10. Error Control forCurrent Solution Methods • Bound on false negatives:  • Pr[s|s]   • Bound on false positives:  • Pr[s|s]   • No undecided results: = 0 • Pr[sI] = 0   Error Control for Probabilistic Model Checing

11. Probabilistic Model Checkingwith Indifference Regions 1 –  s≥ [] s≥ [] Probability of acceptingP≥[] as true in s s≥ []  s≥ []   p  −   + Actual probability of  holding Error Control for Probabilistic Model Checing

12. Hypothesis TestingYounes & Simmons (CAV’02) • Single sampling plan: n, c • Generate n sample execution paths • Accept ≥ [] iff more than c paths satisfy  • Probability of accepting ≥ [] as true: • Sequential acceptance sampling Error Control for Probabilistic Model Checing

13. Statistical EstimationHérault et al. (VMCAI’04) • Estimate p using sample of size n: • Choosing n: • Acceptance condition for ≥ []: Same as single sampling plan n, n + 1! Error Control for Probabilistic Model Checing

14. Statistical Estimation vs.Hypothesis Testing Error Control for Probabilistic Model Checing

15. Numerical Transient AnalysisBaier et al. (CAV’00) • Estimate p with truncation error : • Acceptance condition for ≥ []: • Pr[s|s] = 0 • Pr[s|s] = 0   Error Control for Probabilistic Model Checing

16. Alternative Error Control • Bound on false negatives:  • Pr[s|s ]   • Bound on false positives:  • Pr[s|s ]   • Bound on undecided results:  • Pr[sI|(s)  (s)]     Error Control for Probabilistic Model Checing

17. Probabilistic Model Checkingwith Undecided Results Acceptance probability 1 –  Undecided result withprobability at least 1 –  –   Rejection probability  p  −   + Actual probability of  holding Error Control for Probabilistic Model Checing

18. Statistical Solution Method • Simultaneous acceptance sampling plans • H0: p against H1: p –  • H0: p +  against H1: p • Combining the results • Accept ≥ [] if H0 and H0 are accepted • Reject ≥ [] if H1 and H1 are accepted • Undecided result otherwise         Error Control for Probabilistic Model Checing

19. 20 = 10–2 = 0 15 Verification time (seconds) 10 5 0 14 14.1 14.2 14.3 14.4 14.5 Formula time bound (T) Empirical Evaluation(Symmetric Polling System) serv1  P≥0.5[U ≤Tpoll1] Error Control for Probabilistic Model Checing

20. Empirical Evaluation(Symmetric Polling System)  =  =  = 10–2 Error Control for Probabilistic Model Checing

21. Summary • Statistical estimation is never more efficient than hypothesis testing • Statistical methods are randomized algorithms for CSL model checking • Numerical methods are exact algorithms for CSL model checking • New statistical solution method with finer error control ( parameter) Error Control for Probabilistic Model Checing