1 / 40

Comparing Performance of Probabilistic Model Checkers: A Comprehensive Study

This study evaluates the performance of various probabilistic model checkers with a focus on tool selection. Four examples are presented along with an overall evaluation of the tools based on selected benchmarks. The experiments cover qualitative properties, bounded reachability, steady-state properties, and nested formulas, showcasing the tools' capabilities and limitations. The analysis considers timing, memory usage, and simulation accuracy, providing insights for potential users in fields like distributed systems, security, biology, and quantum computing.

zander
Télécharger la présentation

Comparing Performance of Probabilistic Model Checkers: A Comprehensive Study

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. How Fast and Fat IsYour Probabilistic Model Checker? an experimental performance comparison David N. Jansen3,1, Joost-Pieter Katoen1,2, Marcel Oldenkamp2,Mariëlle Stoelinga2, Ivan Zapreev1,2 1 MOVES Group, RWTH Aachen University 2 FMT Group, University of Twente, Enschede 3 ICIS, Radboud University, Nijmegen

  2. ProbabilisticModel Checking Probabilistic System Probabilistic Requirement PRISM (hybrid) PRISM (sparse) MRMC VESTA ETMCC Probabilistic Model Probabilistic Formula YMER ≤ Probabilistic Model Checker Yes No Probability

  3. Why This Work? • Used more often • applications: distributed systems, security, biology, quantum computing... • Powerful tools • Problem: Which tool to choose?

  4. ProbabilisticModel Checkers Probabilistic System Probabilistic Requirement Choices made Four examples Probabilistic Model Probabilistic Formula Overall evaluation Probabilistic Model Checker Yes No Probability

  5. Tools

  6. Experiment Relevance • Repeatable • Verifiable • Significant • Encapsulated

  7. Selected Benchmarks

  8. SynchronousLeader Election • nodes in a ring elect a leader • each node selects random number as id • passes it around the ring (synchronously) • if  unique id,node with maximum unique id is leader • [Itai & Rodeh, 1990]

  9. 1 4 2 2 5 3 1 5 SynchronousLeader Election

  10. 1 4 4 5 1 5 2 5 1 4 5 1 2 2 1 1 3 5 3 5 2 2 2 3 SynchronousLeader Election 1 4 2 2 5 3 1 5

  11. RandomizedDining Philosophers • Dining Philosophers • pick up chopsticks in random order • Deadlocks resolved • if there is no second chopstick,give up eating • [Pnueli & Zuck, 1986]

  12. Birth–Death Process • Models a waiting queue • Standard modelin performance evaluation • Limit queue size to get finite model

  13. Tandem Queueing Network • Two queues after each other • [Hermanns, Meyer-Kayser & Siegle, 1999] checkin counter two-phase security check exponential

  14. Cyclic Polling System • server cycles over n stationsand serves each one in turn • e.g. teacher walks through class,each pupil may ask a question • [Ibe & Trivedi, 1990]

  15. Modelling informal description PRISM model adapt syntax VESTA model .tra format model YMER model ETMCC MRMC PRISM YMER VESTA

  16. Experiment 1Qualitative Properties • unbounded reachability with prob 1 • Cyclic Polling System: busy1P≥1(trueUpoll1)If station 1 is busy,the server will poll it eventually

  17. PRISM: MTBDD Size • Multi-Terminal BDD =data structure for transition matrix • size heavily depends on model • large MTBDD  slow

  18. CPS versus SLE runtime 458.847 states 1.131.806 MTBDD nodes 7.077.888 states 2.745 MTBDD nodes

  19. VESTA:simulation problem • actual probability close to bound P≥p(...) • estimate is almost always in [p–,p+] • some irregularity stops the simulation • 0.95 Prob(yes  actual Prob≥p) Prob(actual Prob≥p  yes)

  20. P≥1(... U ...)Timing Overview

  21. Analysis

  22. Result Overview: Timing depends heavily on MTBDD size depends heavily on MTBDD size depends heavily on MTBDD size

  23. Result Overview: Memory MTBDD size varies heavily almost independent from model size

  24. Experiment 2Bounded Reachability • Tandem Queueing Network:P<0.01(trueU ≤2full)Is the probabilitythat the system gets full in 2 time unitssmall?

  25. Analysis

  26. Result Overview: Timing

  27. Result Overview: Memory

  28. Experiment 3Steady State Property • Tandem Queuing NetworkS>0.2( P>0.1(X2nd queue full) )In equilibrium,the probability to satisfy is > 0.2 P>0.1(X2nd queue full) P>0.1(X ...)

  29. Simulating Steady State? • simulation of bounded reachabilityhas clear stopping criterion • simulation of unbounded reachability reachability with very large bound • simulation of steady state? never stops

  30. Analysis

  31. Result Overview: Timing

  32. Result Overview: Memory

  33. Nested Formulas • Birth–Death Process:P≥0.8(P≥0.9(trueU ≤100n70) Un50)The probability to reach n50 (while the probability to reach n70in 100 steps never drops <0.9)is ≥0.8

  34. Result Overview: Timing did not terminate

  35. Result Overview: Memory

  36. Conclusions

More Related