1. Passive Testing Wissam Mallouli INT-LOR Wissam.mallouli@int-edu.eu Institut National des Télécommunications - Evry 11/30/2007

2. Table of contents • Introduction • Active/Passive Testing • Forward checking technique • The backward checking technique • Signature based technique • Conclusion

3. Table of contents • Introduction • Active/Passive Testing • Forward checking technique • The backward checking technique • Signature based technique • Conclusion

4. Introduction • Fault detection : essential domain but fastidious. Need of automation. • Testing of a system. • Active/Passive testing techniques

5. Table of contents • Introduction • Active/Passive Testing • Forward checking technique • The backward checking technique • Signature based technique • Conclusion

6. Active Testing IUT Active Tester Verdict: PASS,FAIL, INCONC. Functional & Security Formal Specification Test Suites Automatic test generation based on formal descriptions Generation of a : - reasonable test scenarios number (Execution) - Complete (to cover all the system transitions)

7. Conformance Testing(1/2) • Check if the implementation of a system conforms to its specification I O System (S) I O1=O??? System (I)

8. Errors IMPLEMENTATION i1 / o2 1 2 SPECIFICATION output error i1 / o1 1 2 i1 / o1 1 3 transfert error i1 / o2 1 3 mixte error

9. Passive Testing IUT Trace Collection Passive Tester PO Verdict: PASS,FAIL, INCONC. System User Security Properties Specification

10. The two methods of conformity test (1/2) • Active testing : • principe : send messages through the implementation, wait for answer, compare answer with the desired one • need two phases : • test sequences generation • tests application • pros and cons : • + : possibility to focus on a particular area of the specification • - : automatic test generation is difficult • - : can disturb (or crash) the IUT

11. The two methods of conformity test (2/2) • Passive testing : • principe : observation input/output sequences (trace), application of the trace on the spécification • pros and cons : • + : no interference with the IUT • + : no need of tests generation • - : algorithmsefficiency

12. Table of contents • Introduction • Active/Passive Testing • Forward/backward checking technique • Signature based technique • Conclusion

13. Principles • Trace + Formal specification of the system • To map the trace in the specification • 2 differents algorithms : forward and backward checking

14. a/y b/y S1 S2 a/y b/y b/y a/x b/y S0 S3 a/x Example/Exercice a/x b/y a/x b/y b/y a/y b/y a/y b/y (trace 1) a/x b/y a/x b/y b/y a/x b/x a/y b/y (trace 2)

15. EFSM • EFSM (Extended Finite State Machine) : • I/O events with or without parameters • a predicate (to be satisfied) • actions (to be done)

16. EFSM : definitions • EFSM = heptuplet (S, s0, Sf, I, O, x, T) • S : finite non empty set of states • s0 : initial state • Sf : finite set of final states • I : finite set of input symbols (  parameters) • O : finite set of output symbols (  parameters) • x : finite set of variables • T : finite set of transitions

17. EFSM • Transition = sextuplet (st, ft, it, ot, Pt, At) • st : initial state • ft : final state • it : input event • ot : output event • Pt : predicate • At : set of actions

18. EFSM Formalism b/y a/y S0 S2 a/y P(X0) true b/y b/y A(X0) a/x S1 S3 b/y a/x S=(S0,S1,S2,S3) I=(a,b) O=(x,y) 18

19. Variables values checking • Test by value determination • Test by determination of interval of variables

20. Principle value determination according to an event trace (naive approach) Test by value determination

21. Test by value determination • Algorithm in two phases : • phase 1 : homing. Trace exploration until only one candidat state stays • phase 2 : error detection. Trace exploration until the end, or an error

22. Test by value determination • Variables determination : • few possibles transitions giving different values to one variable  UNDEFINED variable • predicate with at least one UNDEFINED variable  accepted predicate (doubt)

23. State of art of passive testing • Test by value determination • Test by determination of interval of variables

24. Test by determination of interval of variables • Principle • Intervals to record values • Assertions = boolean formula of predicates • Uses Candidate Configuration Set (CCS)

25. Le test par détermination d'intervalles de valeurs • Intervalles • objectif : résoudre le problème de la perte d'information (plusieurs valeurs possibles pour une variable) • notation : • a v  b  R(v) = [a; b] • v = a  R(v) = [a; a]

26. Test by determination of interval of variables • Intervals : operations • sum of two intervals : • [a; b] + [c; d] = [a+c; b+d] • Subtraction of two intervals : • [a; b] - [c; d] = [a-c; d-b] • multiplication of an interval by a scalar : • w  [a; b] = [wa; wb] si w  0 • w  [a; b] = [wb; wa] si w  0

27. Test by determination of interval of variables • Consistency and refinement test • refinement of intervals according to constraints (predicats and actions) • if an interval becomes empty then the configuration is false

28. Table of contents • Introduction • Error types and testing methods • State of art of passive testing • The backward checking technique • Conclusion

29. The backward checking technique • Backward checking : principle • Backtracking a transition • Backtracking a trace • Backtracking the past of a trace

30. Backward checking : principle • Idea : « To understand present we must study the past »

31. Backward checking : principle • Backward checking : • backtracking the event trace • backtracking the past of the trace past of the trace FAIL FAIL trace FAIL OK sens of the trace FAIL sens of the algorithm

32. Backward checking : principle • Advantages of this approach : • start with correct information • build artificially a longer trace

33. Backward checking : principle • To validate a trace in the past we must validate special variables : determinants • Definition : a variable v is a determinant for the trace t if v must necessarily be validated before we can validate t

34. The backward checking technique • Backward checking : principle • Backtracking a transition • Backtracking a trace • Backtracking the past of a trace

35. 4 f i Backtracking a transition P : u>=1 A : x=1 y=y+1 z=v+w R = < f ; u [0;3], x [1;1], y [2;8], z [1;2], a [7;9] ; Asrt = - ; D = {u, y, z, a} > R = < i ; u [1;3], a [7;9], cste [1;2], y [1;7] ; Asrt = v+w=cste ; D = {u, a, y, v, w} >

36. The backward checking technique • Backward checking : principle • Backtracking a transition • Backtracking a trace • Backtracking the past of a trace

37. Backtracking a trace • similar to intervals determination algorithm but backward • process the transition backtracking for each I/O couple

38. The backward checking technique • Backward checking : principle • Backtracking a transition • Backtracking a trace • Backtracking the past of a trace

39. Table of contents • Introduction • Active/Passive Testing • Forward checking technique • The backward checking technique • Invariant based technique • Conclusion

40. Test by invariants : principle • Definition : invariant = property that is always true • Two step test : • extraction of invariants from the specification • application of invariants on event traces from implementation • Problem : temporal logic not in the model • Solution : I/O invariants

41. Test by invariants : I/O invariants • An invariant is composed of two parts : • the test (an input or output) • the preambule (I/O sequence) • 3 kind of invariants : • output invariants • input invariants • succession invariants

42. Test by invariants : Output invariant • Definition : invariant in which the test is an output • Meaning : « immediatly after the sequence préambule there is always the output test • Example : (i1 / o1) (i2 / o2) (preambule in blue, test in red)

43. Test by invariants : Input invariant • Definition : invariant in which the test is an input • Meaning : « immediatly before the sequence préambule there is always the input test • Example : (i1 / o1) (i2 / o2) (i3/ (preambule in blue, test in red)

44. Test by invariants : succession invariant • Definition : I/O invariant for complex properties (loops …) • Example : • the 3 invariants below build the property : « only the third i2 we meet is followed by o3 » (i1 / o1) (i2 / o2) (i1 / o1) (i2 / o2) (i2 / o2) (i1 / o1) (i2 / o2) (i2 / o2) (i2 / o3)

45. Test by invariants • + : powerful method (complex properties) • + : fast when invariants are known • - : difficulties to extract interesting invariants (not automated) • - : sensible to non-determinism of specification • - : doesn't detect every errors