1. Software Testing – Lecture #2 Thomas Ball with material from M. Young, A. Memon and MSR’s FSE group

2. Testing - The Story So Far

3. Two Threads • Specification-based test generation • Foundations of Software Engineering • Implementation-based test generation • Testing, Verification and Measurement • Unifying Themes • create finite-state systems from infinite-state systems via predicates • use finite-state algorithms to guide test generation

4. Specification-based Testing • What does this API do? • Executable spec prescribes potential behavior • Where do concrete tests come from? • Explore behavior of spec, generate test strategies • Does a test execution succeed or fail? • Use spec as oracle for runtime verification • How do we know when we’re done testing? • Cover spec (and implementation) • What do we know when we’re done testing? • Model and implementation agree!

5. 4 Steps to Testing Heaven • Modeling • define infinite transition system • Exploration • reduce to finite test graph • Gaming • generate test strategies • Monitoring • verify conformance

6. Example: Alternating Bit Protocol • This protocol works on the producer-consumer model. • Producer wants to send messages in a reliable manner to a consumer through unreliable channel

7. Describe (transition system in Spec# of) all possible runs of ABP s0 Send s1 ?LoseMsg ?Receive s2 s0 ?Timeout ?LoseAck s4 s3 Send Send s5 ?LoseMsg s6 ?Receive s3 s2 1st Step: Model the Infinite Transition System Types • Msg, Ack State • Bitstatus, … Controllable action • Send Observable events: • Receive • Lose Msg • Lose Ack

8. ABP: Data Structures class DataMsg { // Message readonly Data data; readonlybool bit; } class Ack { // Acknowledgement readonlybool bit; }

9. bool SenderBit = true; int SenderRecordNr = 0; Ack SenderInbox = null; DataMsg ReceiverInbox = null; bool ReceiverBit = true; Seq<Data> ReceivedFile=Seq{}; Sends Message Returns Ack ABP: State

10. Enabling Condition State change ABP: Transitions void Send () { require AcknowledgmentArrived() || Timeout(); if(AcknowledgmentArrived()) { if(AcknowledgmentHasTheRightBit()) { ReceiverInbox = DataMsg(SenderRecordNr + 1, !SenderBit); SenderRecordNr += 1; SenderBit = !SenderBit; } SenderInbox = null; } elseif(Timeout()) { ReceiverInbox = DataMsg(SenderRecordNr, SenderBit); }}

11. ABP: Modeling Demo • Show model in SpecExplorer • Simulate as a console application

12. 2nd Step: Finitize the Model Generate “all” possible behaviors (test graph): • At each state, execute any enabled method with any allowed argument values • Nondeterministic choice from enabled method explores possible interleavings Control the search, using • Finite unfolding • Stochastic means • User provided predicate abstractions [ISSTA 2002] This generates a test graph

13. [1,0] [0,0] Top() Push(1) Push(0) Pop() Pop() Push(1) Top() [0] Top() false Push(0) Push(0) Pop() Pop() [] true User provided predicate abstraction • Example: • Generate an FSM from a stack specification. • Observation property: stack.IsEmpty() Tree can be pruned, if an equivalent state has already been visited

14. A test graph G is a directed graph s.t. there are two kinds of vertices in G: states choice points (CP) every edge e has a prob. p(e) s.t. for every CP u∑{p(e): source of e = u}=1, there is a non-negative cost function c defined on edges Test graph state vertex c=1 CP vertex p=0.7c=2 p=0.3c=1

15. ABP: Modeling Demo • Explorer options

16. 3rd Step: Generate Test Strategies • Cover all edges of the test graph (edge coverage) • Reach certain goal states in the test graph (reachability game) [ISSTA 2004] • With maximal probability and minimal cost • With full certainty and minimal cost

17. Edge Coverage Strategy • Produce a tour of the edges in the graph (e.g. Chinese Postman tour) • Divide the tour into segments starting and ending at choice points • At a choice point IUT selects an edge e and TT chooses randomly any segment s starting from e and follows s until the end (s ends in a choice point).

18. Edge Coverage Example • The tour is[e8,e2,e1,e5,e7, e9,e4,e3,e6,e7] • The segments are:[e8,e2,e1,e5,e7][e9,e4,e3,e6,e7] s1 e2 e3 e1 e4 e9 e8 e7 e6 e5

19. 4th Step:Execute Tests and Verify Results • Define conformance notion: Probes • Rewrite the IUT to insert call backs into model • Specify upper bounds on number of repetitions

20. Probes Given an abstract domain X • A probeP m is a function from model states to X • A probeP i is a function from implementation states to X • Model state A and impl state Bconform wrt the pair of probes (P m, P i) if P m(A) = P i(B) • Different probes may be enabled at different states X P1i B4 P1m A3 B5 P2m P2i A1 B2 B3 A2 B1

21. Event Buffering • During conformance testing all events are buffered in an unbounded queue • All enabled probes are checked at every state (P1m,P1i) IUT Model Action Calls t0 s0 a1 a1 a1’ t1 s1 (P2m,P2i) e1 e2 e1’ s2 s3 t2 Events a2 e1 e2 s4

22. ABP Demo • Test sequence generation • Conformance testing

23. Experience • Modeling for Testing • Models are created by testers, not designers • Models are based on informal specs, not impl. • Models are not comprehensive, only issues • Several projects • Web services, Passport, Mediaplayer, Distributed File replication system • Models up to 100 pages • More than 50 people are using it on a daily basis, steep uptake

24. More Info • Public release of Spec# this Summer http://research/microsoft.com/foundations • Contact schulte@microsoft.com if you would like to know more or are interested in a summer internship in 2005! • Consider submitting something for ICFEM 2004 http://research/microsoft.com/conferences/ICFEM2004

25. MSIL Unit Test Toola hybrid helper • Goal capture developer knowledgeASAP via a strong set of unit tests to form a specification of the code’s behavior • How • generate tests based on analysis of MSIL • symbolic execution + constraint satisfaction • runtime analysis to check complicated invariants • Facets • complements specification-based test generation • positive feedback cycle with programmer

26. What criteria should guide unit test generation?

27. Predicate-complete Testing • Predicates • relational expression such as (x<0) • the expression (x<0) || (y>0) has two predicates • predicates come from program and safe runtime semantics • Consider a program with m statements and n predicates • predicates partition input domain • m x 2n possible observable states S • Goal of Predicate-complete Testing: • cover all reachable observable states R  S

28. PCT Coverage L2: if (A || B) S else T L3: if (C || D) U else V • PCT requires covering all logical combinations over {A,B,C,D} at • L2 and L3 • S, T, U and V • Some combinations may not be reachable

29. PCT Coverage does not imply Path Coverage L1: if (x<0) L2: skip; else L3: x = -2; L4: x = x + 1; L5: if (x<0) L6: A;

30. PCT Coverage does not imply Path Coverage L1: if (x<0) L2: skip; else L3: x = -2; L4: x = x + 1; L5: if (x<0) L6: A;

31. PCT Coverage does not imply Path Coverage L1: if (x<0) L2: skip; else L3: x = -2; L4: x = x + 1; L5: if (x<0) L6: A;

32. PCT Coverage does not imply Path Coverage L1: if (x<0) L2: skip; else L3: x = -2; L4: x = x + 1; L5: if (x<0) L6: A;

33. Path Coveragedoes not imply PCT Coverage L1: if (p) L2: if (q) L3: x=0; L4: y=p+q;

34. Path Coverage does not imply PCT Coverage L1: if (p) L2: if (q) L3: x=0; L4: y=p+q;

35. Denominator Problem • Coverage metrics require a denominator • e.g. statements executed / total statements • Easy to define for observable states • executed observable states / (m x 2n) • But (m x 2n) is not a very good denominator! • most observable states will not be reachable • R <<< S

36. m x 2n possible states S Upper bound U Reachable states R Lower bound L Upper and Lower Bounds • Bound reachable observable states • modal transition systems and predicate abstraction • |L| / |U| defines “goodness” of abstraction • Test generation using lower bound L • Refinement to increase |L| / |U| ratio

37. Abstraction Construction MC MC MA MA a a   total may a’ a’   a a   total & onto onto a’ a’  

38. a a may may b b c c Upper Bound: May-Reachability

39. a a may may b b c c Upper Bound: May-Reachability

40. Pessimistic Lower Bound a onto b may c total d

41. Pessimistic Lower Bound a onto b may c total d

42. Pessimistic Lower Bound a onto b may c total d

43. Example void partition(int a[]) { assume(a.length()>2); int pivot = a[0]; int lo = 1; int hi = a.length()-1; while (lo<=hi) { while (a[lo]<=pivot) lo++; while (a[hi]>pivot) hi--; if (lo<hi) swap(a,lo,hi); } } void partition(int a[]) { assume(a.length()>2); int pivot = a[0]; int lo = 1; int hi = a.length()-1; while (lo<=hi) { while (a[lo]<=pivot) lo++; while (a[hi]>pivot) hi--; if (lo<hi) swap(a,lo,hi); } }

44. Observation Vector [ lo<hi, lo<=hi, a[lo]<=pivot, a[hi]>pivot ] • lo<hi  lo<=hi • lo<hi  lo<=hi  (a[lo]<=pivot  a[hi]>pivot)  (a[lo]<=pivot  a[hi]>pivot) Only 10/16 observations possible

45. void partition(int a[]) { assume(a.length()>2); int pivot = a[0]; int lo = 1; int hi = a.length()-1; L0: while (lo<=hi) { L1: ; L2: while (a[lo]<=pivot) { L3: lo++; L4: ;} L5: while (a[hi]>pivot) { L6: hi--; L7: ;} L8: if (lo<hi) { L9: swap(a,lo,hi); LA: ;} LB: ;} LC: ; } 13 labels x 10 observations = 130 observable states But, program constrains reachable observable states greatly.

46. Boolean Program void partition() { decl lt, le, al, ah; enforce ( (lt=>le) & ((!lt&le)=>(al&!ah)|(!al&ah)) ); lt,le,al,ah := T,T,*,*; L0: while (le) { L1: ; L2: while (al) { L3: lt,le,al := (!lt ? F:*), lt, *; L4: ;} L5: while (ah) { L6: lt,le,ah := (!lt ? F:*), lt, *; L7: ;} L8: if (lt) { L9: al,ah := !ah,!al; LA: ;} LB: ;} LC: ; }

47. State Space of Boolean Program [ lo<hi, lo<=hi, a[lo]<=pivot, a[hi]>pivot ] Upper Bound = 49 states

48. plaintext

49. Test Generation • DFS of Lp generates covering set of paths • Symbolically execute paths to generate tests • Run program on tests to find errors and compute coverage of observable states

50. Array bounds violations Generated Inputs void partition(int a[]) { assume(a.length()>2); int pivot = a[0]; int lo = 1; int hi = a.length()-1; L0: while (lo<=hi) { L1: ; L2: while (a[lo]<=pivot) { L3: lo++; L4: ;} L5: while (a[hi]>pivot) { L6: hi--; L7: ;} L8: if (lo<hi) { L9: swap(a,lo,hi); LA: ;} LB: ;} LC: ; } (L0:TTTT,L4:FTFT) { 0,-8,1 } (L0:TTTT,L4:TTFT) { 0,-8,2,1 } (L0:TTTT,L4:TTTT) { 0,-8,-8,1 } (L0:TTTF,L4:TTFF) { 1,-7,3,0 } (L0:TTTF,L4:FTTF) { 0,-7,-8 } (L0:TTTF,L4:TTTF) { 1,-7,-7,0 } (L0:TTFT,L7:TTFF) { 0,2,-8,1 } (L0:TTFT,L7:FTFT) { 0,1,2 } (L0:TTFT,L7:TTFT) { 0,3,1,2 } (L0:TTFF,L0:TTTT) { 1,2,-1,0 }