Lecture 6

1. Lecture 6 Model-based Testing and Automated Test Case Generation

2. Automated Test-Case Generation (TCG) • By now we have seen that manual construction of test cases is a difficult, time-consuming and error-prone activity that requires expertise. • Automated Test-Case Generation (TCG) has been the “holy grail” of testing for some time. • Is this even possible? If so, how? • Black/white-box testing • Want algorithms which generate test cases (preferably with a known level of coverage)

3. Constraint Solving • A constraint is a logical formula with one or more free variables • Examples: x >= 0, x+1 > y, x2 = 2.0, etc. • A constraint solver is an algorithm which finds values for the free variables, which make a formula true, aka. solutions • Examples: • x=1 I= x >= 0 ( I= is the satisfaction symbol ) • x=1.41421356 I= x2= 2.0 • Exist many different kinds of constraint solvers

4. Specification-based Black-box Testing • System requirement (Sys-Req) • System under Test (SUT) • Test verdict pass/fail (Oracle step) Sys-Req pass/fail Test case Output TCG SUT Oracle Constraint solver Language runtime Constraintchecker

5. Example: ATCG for numerical codeNewton’s Square root algorithm Postcondition Ι y*y – x Ι ≤ ε preconditionx≥ 0.0 TCG SUT Oracle x=4.0 y=2.0 Input Output Constraint solver Constraint checker Newton Code x=4.0 satisfiesx ≥ 0.0 Verdict x=4.0, y=2.0 satisfies Ι y*y – x Ι ≤ ε

6. TCG for MBT of Embedded Systems • We can make a simple model of an embedded system as a deterministic Kripke structure (or Moore machine) • M = ( Q, Σ, q0, δ, λ ) • Where • q0 Q, - initial state • δ : Q x Σ Q - state transition function • λ: Q  Bn- n-bit output function

7. Example: 2-bit shift register • Q = A, B, C, D, Σ= 0,1, q0=A B, λ(B) =10 1 1 A, λ(A) =00 D, λ(D) =11 0 1 0 0 C, λ(C) =01 0 1

8. Another Representation • Q = A, B, C, D, Σ = 0,1, q0=A Bit1 & !Bit2 1 1 !Bit1 & !Bit2 Bit1 & Bit2 0 1 0 0 !Bit1 & Bit2 0 1

9. 2-Bit Shift Reg in nuSMV format MODULE main VAR Bit1 : boolean; -- Boolean var Bit2 : boolean; input : boolean; state : {A, B, C, D}; -- scalar var variable ASSIGN init(state) := A; init(Bit1) := 0; init(Bit2) := 0; -- var “input” is uninitialised!

10. next(state) := case state = A & input = 1: B; state = B & input = 0 : C; state = B & input = 1 : D; state = C & input = 0 : A; state = C & input = 1 : B; state = D & input = 0 : C; TRUE : state; -- why? esac; next(Bit1) := case ?? esac; next(Bit2) := case ?? esac;

11. Propositional Linear Temporal Logic PLTL in nuSMV syntax • Simple temporal logic to express automaton requirements • Requirements are statements about all possible simulations • Boolean variables • A, B, …, X, Y, .. MyVar, etc. • Boolean operators • !ϕ, (ϕ & φ), (ϕ ϕ) , (ϕ-> ϕ) ,… • LTL operators • Fϕ(sometime in the future ϕ) • Gϕ(always in the future ϕ) • ϕ U φ(φ holds until ϕholds) • Xϕ (next step ϕ holds) • Write Xnϕ for X X … Xϕ(n times)

12. Examples Today is Wednesday Wednesday Tomorrow is Wednesday X Wednesday (A) Thursday (always) immediately follows (a) Wednesday G( Wednesday -> Thursday ) (A) Saturday (always) follows (a) Wednesday G( Wednesday -> F( Saturday ) ) Yesterday was (a) Wednesday G( Wednesday -> X Thursday) & Thursday • Exercise: define the sequence of days precisely, i.e. just one solution • Question: are there any English statements you can’t make in LTL? • Question: what use cases can you express in LTL?

13. Useful Logical Identities for LTL • Boolean identities !!φ φ, !(ϕ φ) (!ϕ& !φ), (ϕ-> φ)(!ϕ φ)etc. • LTL identities !G(!ϕ )  F(ϕ) !X(ϕ)  X(!ϕ) G(φ& ϕ)  G(φ)& G(ϕ) G(φ)  φ& X G(φ) • Exercise: using these identities, prove: !F(!φ)  G(φ) F(φ ϕ)  F(φ)  F(ϕ) F(φ)  φX F(φ) • Remark TCG usually involves negating formulas, so its useful to understand what a negation means

14. Specifications in nuSMV files LTLSPEC G( Bit1 <-> X Bit2 ) –- the value of Bit1 now always equals the -- value of Bit2 in the next time step -- This is obviously TRUE! G( Bit1 <-> X Bit1 ) –- the value of Bit1 now always equals the -- value of Bit2 in the next time step -- This is obviously FALSE!

15. Model Checking • nuSMV is a temporal logic model checker • A model checker is a tool which takes as input • An automaton model M • A temporal logic formula φ • If φ is a true statement about all possible behaviors of M then the model checker confirms it (proof) • If φ is a false statement about M the model checker constructs a counterexample(a simulation sequence) • A counterexample to φsatisfies !φ • A simulation sequence can be executed as a test case

16. nuSMV Output Example -- specification G( Bit1 <-> X Bit2 ) -- is true -- specification G( Bit1 <-> X Bit1 ) -- is false -- as demonstrated by the following execution sequence Trace Description: LTL Counterexample Trace Type: Counterexample -> State: 1.1 <- state = A Bit1 = false Bit2 = false -> Input: 1.2 <- input = 1 -> State 1.2 <- state = A Bit1 = true Bit2 = false

17. Automated White-box TCG • We can use a model checker to generate counterexamples to formulas (i.e. test cases) with specific structural properties. • This is done by inspecting the graph structure of the automaton • i.e. white/glass box testing • “Most” use of model checkers concerns this.

18. Test Requirements/ Trap Properties • Recall a test requirement is a requirement that can be satisfied by one or more test cases • Basic idea is to capture each test requirement as an LTL formula known as a “trap property” Example Suppose the test requirement is “cover state D” Trap property is G!( Bit1 & Bit2 ) This formula is False and any counterexample must be a path that goes through state D.

19. Case Study: Car Controller Velocity = slow !accelerate accelerate Velocity = stop brake accelerate !accelerate brake Velocity = fast Basic idea (incomplete)

20. Module main Var accelerate: boolean; brake: boolean; velocity: {stop, slow, fast} ASSIGN init(velocity) := stop next(velocity := case accelerate & !brake & velocity = stop : slow; accelerate & !brake & velocity = slow : fast; !accelerate & !brake & velocity = fast : slow; !accelerate & !brake & velocity = slow: stop; Brake: stop; TRUE: velocity; esac;

21. Trap properties for Structural Coverage Models • Let us use nuSMV to automatically construct test suites that satisfy the different structural coverage models introduced in Lecture 2. • Examples: • Node coverage NC • Edge coverage EC • Condition coverage PC • How to interpret these concepts?

22. Node Coverage for CC • Want a path that visits each node • Simple approach: 1 trap property per node • General form: • G(!“unique_state_property”) • Counterexamples satisfy: • F(“unique_state_property” ) • Examples: G(!velocity = stop) G(!velocity = slow) G(!velocity = fast) • Clearly this will give redundant test cases, but method is still linear in state-space size.

23. Edge coverage for CC • Want to be sure we traverse each edge between any pair of nodes • General form G( α & ϕ -> X !β ) • Counterexample satisfies F(α & ϕ& X β ) • Example (5 more cases): G(velocity=stop & accelerate -> X velocity=slow) • Exercise: define the remaining 5 trap formulas α φ β

24. Condition Coverage for CC • For each transition, and each Boolean guard v there exist two test cases for the transition: (1) guard is true. (2) guard is false α β Boolean v; G!(α & v) -> X(!β) G!(α & !v) -> X(!β)

25. Requirements-based TCG • Can we also perform requirements-based test case generation? • Want to test the requirements are fulfilled rather than explore structure of the code. • Can look at negation of a requirement

26. Car Controller LTL requirements 1. Whenever the brake is activated, car has to stop G(brake -> X velocity=stop) 2. When accelerating and not braking, velocity has to increase gradually G(!brake& accelerate&velocity=stop -> X velocity=slow ) G(!brake& accelerate&velocity=slow -> X velocity=fast ) 3. When not accelerating and not braking, velocity has to decrease gradually G(!brake& !accelerate&velocity=fast -> X velocity=slow) G(!brake& accelerate&velocity=slow -> X velocity=stop)

27. Safety Requirements • A safety requirementdescribes a behavior that may not occur on any path. • “Something bad may not happen” • To verify, all execution paths must be checked exhaustively • Safety properties usually have the form G!φwhere φ is the “bad thing” • Counterexamples (test cases) are finite

28. Liveness Requirements • A livenessrequirementdescribes a behavior that must hold on all execution paths • “Something good eventually happens” • Safety does not imply liveness • Liveness does not imply safety • Liveness properties may have the form F(φ) orG(ϕ-> Xnφ ) orG(ϕ-> Fφ) where φdescribesthe “good” thing and ϕ is some preconditionneeded for it to occur. • Counterexamples may be finite orinfinite (why?)

29. Infinite Counterexamples • nuSMV describes infinite counterexamples as a “loop and handle” • a,b,c,d,(x,y,z)ωor handle(loop)ω • a, …, d, x, …, z are the individual states • We can generate these sequences, but can we execute them? • Are they test cases at all? • We call a liveness requirement finite if all counterexamples are finite • Use cases are examples of finite liveness requirements

30. TCG for Finite Liveness Requirements • A common form of liveness requirement is G( φ -> Xnϕ ) (1) • Negating (1) and rewriting gives F( φ & Xn (!ϕ ) ) (2) • Every counterexample to (2) is a solution to (1) i.e. an intended behavior. • We can ask nuSMV for these intended behavior, and compare them against the actual behavior. • Edit the sequence to delete all internal and output variables • Agreement = pass, disagreement = fail

31. Car Controller Examples • F(brake &X !velocity=stop) (2) F(!brake & accelerate & velocity=stop -> X velocity=slow)

32. Conclusions • TCG is certainly possible using various existing algorithms. • Generally requirements MBT is trickier than structural MBT using this approach. • There are other approaches to requirements testing. See next lecture.