1. System testingchapter 14 Csci 565 Spring 2007

2. objectives • System testing • All about threads at system level • How big a thread should be? • How to identify them? • The role of threads in Structural/functional testing • SATM Test Threads

3. System level testing • System testing • Can be considered a final step in integration testing • A check of consistency between the software system and its specification • Encompasses system wide properties against a system specification • System, acceptance, and regression testing are all concerned with the behavior of a entire software system

4. System, acceptance, and regression testing

5. System test design • It is considered a good practice to design system test cases as early possible • Why? • Exposes critical behaviors • Avoids unpleasant surprises • Testing global properties may require extensive simulation of the execution environment

6. Unit, integration, and system testing

7. Threads: some interpretations • Primitives concepts with different meaning at different levels • Think of them as Use-cases/scenarios • Threads and level of abstractions • System level behavior corresponds to a sequence of Atomic System Functions (ASFs) • Integration level behavior corresponds to a sequence of MM-path • Unit level behavior corresponds to a DD-path

8. Threads: some generic examples • Example of threads may include • A scenario of normal usage • A stimulus/response pair • A sequence of source instructions • A sequence of atomic system functions (ASFs) • A sequences of a state machine description of the system • …

9. Threads at different levels: some specific examples (SATM) • Candidate threads for SATM • Not Ok (too concrete) • Digit Entry (concrete) • an ASF works with only one stimuli/response • PIN Entry (abstract) • ASFs work with a set of stimuli/response • max integration, min system level • A transaction (very abstract) • A system level thread because of end-to-end completion • <Card entry, menu selection, perform operation, report the results> • An ATM session or multiple transactions (very very abstract) • A sequence of threads • OK for system level • Good for interactions among threads

10. System testing and requirement specification • System testing w.r.t. requirement specifications • Any system can be specified using • Data (ERD) • Actions (DFD) • Ports (source/destination of input-level system) • Events (occurrence of data or actions on the ports) • Threads (scenarios or paths in control models)

11. Data-centric models • When the focus is on the persistent data, data-centric models must be used (e.g., ERD) • developed using CRUD (create, retrieve, Update, Delete) • In SATM, Data • PAN (different accounts) • PIN (personal identification number)

12. Ports • Ports (devices) • The source and destination of system level inputs and outputs • the points at which an I/O device are connected to the system • Examples in SATM • The digits and cancel keys • Functions keys (buttons) • Display screen • Slots

13. Events • refer to system level input (or output) • the translation point between physical (external) and logical (internal) • Event can be • Discrete (e.g., keystroke) • Continuous (e.g., pressure, temperature)

14. actions • Process, or transformation point • DFD • work with • Inputs/outputs for both control and/or data

15. Relationships among basis concepts

16. Modeling relationship among basis constructs (fig. 14.2)

17. Examples of Models • Models can be • Structural models • Used for development • Express the functional decomposition and decomposition • Contextual models • Emphasize system ports (or external environment) • Behavioral models • Emphasize the control aspects of the system • Identification of models? • Right models for right problems

18. Model selection: Event partitioning view of function F

19. Model selection: FSM of Function F

20. Finding Threads from behavioral models • FSM models of the SATM is the appropriate staring point for the system testing threads • The model can be interpreted as follows • States (processing steps or the abstraction of the history) • Transitions (logical events)

21. Top level SATM in FSM

22. Top level SATM in FSM

23. Low level SATM in FSM (PIN Entry state)

24. Low level SATM in FSM (PIN Entry state)

25. Low level SATM in FSM (PIN Entry state)

26. Table 1: Events in the PIN entry FSM

27. Very low level SATM ( PIN TRY) in SFM

28. Very low level SATM ( PIN TRY) in SFM

29. Very low level SATM ( PIN TRY) in SFM

30. Very low level SATM ( PIN TRY) in SFM

31. Very low level SATM ( PIN TRY) in SFM

32. Very low level SATM ( PIN TRY) in SFM

33. Table 2: Port Events in the PIN try FSM

34. Table 3: An example of a path (or thread) with a correct PIN (1234) Logical event

35. Table 4: An example of a thread (1235, 123C, 1234) Logical events

36. Structural Strategies (or path testing) for thread testing • Generating thread test cases is easy • Too many threads (explosion problem) • Strategies • Bottom-up threads • 6 paths can be tested in the PIN Try FSM to test (see table 5) • Correct recognition of digits • Response to the cancellation • Matching expected and entered PINs

37. Table 5: 6 thread in the PIN Try FSM correct incorrect cancellation

38. Very low level SATM ( PIN TRY) in SFM

39. Bottom-up (next upper level) • Once the lower level is tested, move to the upper level (the PIN Entry FSM) • Four paths tested to achieve these objectives • Correct recognition of PIN • Incorrect PIN • Response to the cancel key • Matching expected and actual PIN entered • `See table 6’

40. Table 6: Thread Paths in the PIN Entry FSM

41. Low level SATM in FSM (PIN Entry state)

42. Low level SATM in FSM (PIN Entry state)

43. Low level SATM in FSM (PIN Entry state)

44. Low level SATM in FSM (PIN Entry state)

45. Coverage Metrics for FSM • FSM are directed graphs • use a graph-based coverage metrics • Hierarchical relationship means • Super node (upper level node) • Sub-node (lower level) • Super nodes treat sub-nodes as a procedure with enter/exist semantics

46. States and transitions Coverage matrices • Two Coverage (metrics) can be applied • State coverage • States correspond to nodes • Transition coverage • Transitions correspond to edges • Path Coverage? • ‘See table 14.7’

47. Table 7: State/Transition coverage of the 3-try thread

48. Table 8: Thread/State incidence

49. Table 9: Thread/Transition Incidence

50. Problems with path testing (FSM) • The main assumption revolves around path traversing to uncover faults • Can be counter-productive • E.g., CCC in table 6 causes a three-try to fail • Some path cannot be executed by themselves because of dependency between lower levels and upper levels • E.g., 1235 in table 5 causes a transition to Second PIN try • What happens if the user does not press any key?