Multiple Shooting, CEGAR-based Falsification for Hybrid Systems

1. Multiple Shooting, CEGAR-based Falsification for Hybrid Systems  Jyotirmoy Deshmukh James Kapinski Aditya Zutshi Sriram Sankaranarayanan

2. Hybrid Systems Discrete Controller Sense Actuate Safety Critical ! Physical System (plant)

3. Falsification Error? System Description ErrorStates Initial States t Is there a trajectory from an initial state to an error state?

4. System Description Most systems do not have Hybrid Automaton models! Mode 1 Mode 2 Simulink/Stateflow X, t X’ SIM(X,t) X’ X t Hybrid Automaton Model [Alur, Henzinger, Lygeros, Sastry, Tomlin,…]

5. Single Shooting SIM(X,t) System Description Inefficient in the presence of non-linearitiesand discrete updates Error States Initial States S-Taliro: [Fainekos, et al.] BREACH: [Donze’] RRT: [Bhatia et al., …]

6. Multiple Shooting • Explore trajectory space • Narrow gaps iteratively Proposed Solution CEGAR Error States Gaps Initial States

7. Contributions Multiple Shooting CEGAR (Counter Example Guided Refinement) Trajectory segment Abstract path B Narrowing of gaps Refinement A • Grid based Abstractions Verification of Hybrid Systems Based on Counterexample-Guided Abstraction Refinement [Clarke, Fehnker, et al.]

8. Scatter and Simulate • Grid based Abstractions • Induced by norm Fundamental question in abstractions: A  B ? Scatter & Simulate B • Explicit Abstractions • Black Box: No system dynamics • Complex dynamics • Curse of Dimensionality A

9. Multiple Shooting & CEGAR Compute Explore it using scatter & simulate • Search Error Paths • Trade soundness for efficiency. • Find a subset of paths. Assume implicit abstraction Enumerate error paths Check for concrete paths Error Paths done Refineabstraction using CEGAR Assume a finer abstraction Compute

10. Multiple Shooting & CEGAR… Compute Explore it using scatter & simulate • Refine by CEGAR • Examine abstract error paths • Entire path • Initial cell Assume implicit abstraction Enumerate error paths Check for concrete paths Error Paths done CEGAR Finer grid size Assume a finer abstraction Compute

11. Scatter and Simulate Compute Get cell from Q Sample cell Error States Cell Queue Simulate for Initial States Identify reached cells If new, add cell to Q Error Paths Enumerate error paths

12. Refinement CEGAR Refine Grid Error Paths Compute Scatter & Simulate New Error Paths Enumerate error Paths

13. Concretization • Described procedure can run forever • Only comes up with segmented trajectories • No termination guarantee due to numerical errors • Solution • interleave Concretization: Use random testing on refined initial cells Scatter &Simulate Done!! Concretize CEGAR

14. DemoVan der Pol – iteration 1 Plot of Scatter & Simulate Intial Set with initial cells

15. DemoVan der Pol – iteration 2 Plot of Scatter & Simulate Intial Set with initial cells

16. DemoVan der Pol – iteration 3 Plot of Scatter & Simulate Intial Set with initial cells

17. DemoVan der Pol – iteration 4 Plot of Scatter & Simulate Intial Set with initial cells

18. DemoVan der Pol – iteration 5 Plot of Scatter & Simulate Intial Set with initial cells

19. 14 Cont. States 625 Modes Experiments • Van Der Pol • Lorenz • Brusselator • Bouncing Ball • Bouncing Ball + SHM • Constrained Pendulum • Navigation 30(mod.) • Idle Speed Controller • MPC • Glucose Insulin • Quadcopter(mod.) • Cardiac Academic Examples • Cont. States: 2-14 • Modes: 0-625 Complex Benchmarks

20. Comparison Random Testing • Van Der Pol • Lorenz • Brusselator • Bouncing Ball • Bouncing Ball + SHM • Constrained Pendulum • Navigation 30(mod.) • Idle Speed Controller • MPC • Glucose Insulin • Quadcopter(mod.) • Cardiac Light-weight Scatter and Simulate S-Taliro dReach Exhaustive S-Taliro: [Fainekos, et. Al.]dReach: [Gao, et. Al. ]

21. Times are hard to compare! Experimental Setup Random Testing S-Taliro Scatter & Sim. • Random Testing • Use random testing to synthesize safety properties when they don’t exist • Run 100,000 simulations and find number of violations • S-Talirovs Scatter & Sim. • Run 10 times • Run terminates if • Violation found • Timeout: 1hr • Tools can restart during a run • Time taken is hard to compare • S-Taliro has a single threaded impl.

22. Results - Van Der Pol Highly non-linear! 2 continuous States Random Testing S-Taliro Scatter & Sim. Vs

23. Results - Bouncing Ball Hybrid! 4 continuous States 1mode Random Testing S-Taliro Scatter & Sim. Vs

24. Results - Navigation30 625 Modes! 4 continuous States 625 modes Random Testing S-Taliro Scatter & Sim. Vs Becnhmarks for Hybrid Systems Verification: [Fehnker and Ivancic]

25. Results - Idle Speed Controller Inputs! 9 continuous States 4 modes 1 input Random Testing S-Taliro Scatter & Sim. Vs A new algorithm for reachability analysis of hybrid automata : [A. Casagrande, et al.]

26. In Summary… • Falsification technique for Hybrid Systems. • No explicit model required! • Simulations are cheap and parallelizable! • Generalizable in many direction. But… • Can not find non-robust trajectories • Convergence is not guaranteed • Best effort search • Can provide asymptotic guarantees

27. Extra Slides…

28. Falsification Approaches: Shooting • Single Shooting • Random testing • S-Taliro • BREACH • Systematic Sim. • RRTs • … • Multiple Shooting • Proposed approach: • Scatter & Simulate

29. Single Shooting: Random Testing SIM(X,T) System Description • Naïve: needs guidance • Curse of dimensionality: Scales poorly with increasing states Error States Initial States

30. Single Shooting:Guided Testing • S-Taliro: [Fainekos, et. Al] • BREACH: [Donze] Inefficient in the presence of non-linearities and discrete updates Error States Initial States

31. Multiple Shooting Distribute non -linearity Solution…? Use mature NLP Solvers Translate the problem as an optimization problem with equality constraints Error States Proposed Solution Use Abstractions and CEGAR Initial States Undesirable Gaps A Trajectory Splicing Approach to Concretizing Counterexamples for Hybrid Systems: [Zutshi, et al.]

32. Abstractions and CEGAR How to effectively use Multiple Shooting? Use Discrete Abstractions and a refinement procedure CEGAR: Counter Example Guided Refinement • Induced by norm • Grid Based Implicit Abstraction • Partitions the state space into rectangular Cells • Discovers relations using simulation Verification of Hybrid Systems Based on Counterexample-Guided Abstraction Refinement [Clarke, Fehnker, et al.]

33. Grid Based Abstraction • Discretizes concrete states • Relations induced by Dynamics Abstract State: Concrete States: HSolver: [Ratschan, et al.]

34. Explicit Abstractions Curse of Dimensionality • Explicit abstraction construction • Used by verification approaches • Sound procedure finds relations between adjacent cells • Enumerate all abstract error paths Predicate Abstraction for reachability analysis of HS [Alur, Dang, Ivancic]

35. Exploring Implicit Abstractions Mitigate curse of dimensionality! • Implicit Abstractions • Use simulations in a multiple shooting fashion • Sample relations • Efficiently discover a subset of abstract error paths