Ymer: A Statistical Model Checker

1. Ymer:A Statistical Model Checker Håkan L. S. Younes Carnegie Mellon University

2. Probabilistic Model Checking • Given a model M, a state s, and a property , does  hold in s for M? • Model: stochastic discrete event system • Property: probabilistic temporal logic formula • Example: ≥0.1[≤5full] Ymer: A Statistical Model Checker

3. Statistical Solution Method • Use acceptance sampling to verify probabilistic properties • Hypothesis: P≥ [] • Observation: verify  over a sample path • Bounds on probability of verification error • Probability of false negative: ≤  • Probability of false positive: ≤  Ymer: A Statistical Model Checker

4. Indifference region p1 p0 Error Bounds 2 Probability of errorwhen verifying P≥[]    Actual probability of  holding Ymer: A Statistical Model Checker

5. Ymer at a Glance • Supports time-homogeneous generalized semi-Markov processes • Limited to time-bounded properties • Distributed acceptance sampling (even with sequential acceptance sampling) • Purely statistical approach for verifying nested probabilistic statements Ymer: A Statistical Model Checker

6. DistributedAcceptance Sampling Slave Master Master Acceptance Sampling register model & property observation  Slave simulation Slave simulation observation  done Ymer: A Statistical Model Checker

7. Avoiding Sample Bias • Process observations as they come in? • No, bias against observations that take a long time to generate (long sample paths) • Process observations according to a predetermined schedule  Schedule: 1 2 1 1 2 Received: 1 1 2 Ymer: A Statistical Model Checker

8. … Server Case Study:Symmetric Polling System • Single server, n polling stations • Stations are attended in cyclic order • Each station can hold one message • State space of size O(n·2n)     Polling stations Ymer: A Statistical Model Checker

9. Results 100 Machine 1: 733 MHz Pentium III Machine 2: 500 MHz Pentium III 90 80 Percent of single machine 70 60 50 102 104 106 108 1010 1012 1014 Size of state space Ymer: A Statistical Model Checker

10. Nested Probabilistic Statements: Robot Grid World • Probability is at least 0.9 that goal is reached within 100 seconds while periodically communicating • ≥0.9[≥0.5[ ≤9 comm]≤100goal] Ymer: A Statistical Model Checker

11. Statistical Verification ofNested Probabilistic Statements • Cannot verify path formula without some probability of error • Probability of false negative: ≤ ′ • Probability of false positive: ≤ ′ Observation error Ymer: A Statistical Model Checker

12. Performance Considerations • Verification error is independent of observation error • Pick observation error to minimize effort • The same state may be visited along multiple sample paths • Memoize verification results to avoid repeated effort Ymer: A Statistical Model Checker

13. Robot Grid World (results) ≥0.9[≥0.5[ ≤9 comm]≤100goal] 104 103 = 0.025 = = 10−2 102 = 0.05 Verification time (seconds) 101 100 10−1 10−2 102 104 106 108 1010 1012 Size of state space Ymer: A Statistical Model Checker

14. Robot Grid World:Effect of Memoization 1.0 0.9 103 0.8 0.7 0.6 Unique/visited states Sample size 102 0.5 0.4 0.3 0.2 101 0.1 102 104 106 102 104 106 Size of state space Size of state space Ymer: A Statistical Model Checker

15. Availability • Source code is released under GPL • http://sweden.autonomy.ri.cmu.edu/ymer/ Ymer: A Statistical Model Checker