Dynamic Testing

1. Dynamic Testing Test Design Techniques

2. Dynamic Testing • Testing that involves the execution of the software of the component or system • Execution of the test object on a computer • Need a test bed

3. Test Bed Test Driver Test Case 1 Test Case 2 Test Case n Stub1 PoC Test output PoO Test Object Stub 2 comparison Stub k Run time environment, Analysis tools, monitors Test stubs PoC – Point of Control PoO – Point of Observation

4. Incremental approach to execute tests • Steps are: • Determine conditions and preconditions for the test and the goals that are to be achieved • Specify individual test cases • Determine how to execute the tests (usually chaining together several test cases) • Eg: group test cases in such a way that a whole sequence of test cases is executed (test sequence or test scenario) • Document it in test procedure specification • Need test script, e.g., JUnit

5. Techniques for Testing • They are test case design techniques • Different techniques are: • Black Box Testing • White Box or Glass Box or Open Box Testing

6. Black Box Testing Techniques(1) • Test object is treated as a black box • The inner structure and design of the test object is unknown • Test cases are derived/design using the specification or the requirements of the test object • The behavior of the test object is watched from the outside (PoO is outside the test object) • It is not possible to control the operating sequence of the object other than choosing the adequate input test data (PoC is situated outside of test object) • Used for higher levels of testing • Any test design before the code is written (test-first programming, test-driven development) is black box driven

7. Black Box Testing Techniques(2) Poc and PoO “outside” the test object Test input data PoC Test Output data PoO Test Object

8. Black Box Testing Techniques: Equivalence Class Partitioning Method(1) • The domain of possible input data for each input data element is divided in equivalence classes • An equivalence class is a group of data values where the tester assumes that the test object processes them the same way • the behavior of the component or system is assumed to be the same • Test of one representative of the equivalence class is seen as sufficient • test equivalence class for correct input as well as incorrect input

9. Black Box Testing Techniques: Equivalence Class Partitioning Method(2) • Example of correct (valid) equivalence class vEC

10. Black Box Testing Techniques: Equivalence Class Partitioning Method(3) • Example of incorrect (invalid) equivalence class iEC

11. Black Box Testing Techniques: Equivalence Class Partitioning Method(4) Systematic Derivation of the Test Cases • Determine the domain of all possible inputs for every input data element • E.g., function/method parameter at component tests or input screen field at system test • Partition the domain into equivalence classes • First the subdomain of correct inputs • The test object should process these according to specification • Then the values outside this domain as equivalence classes for invalid values • Observe how the test object behaves • Refine the equivalence classes • The equivalence classes should be divided until each different requirement corresponds to an equivalence class • Choose a representative value for every single equivalence class for testing • Define the preconditions and the expected result for every test case

12. Black Box Testing Techniques: Equivalence Class Partitioning Method(5) • the same principle of dividing into equivalence classes can also be used for the output data • Identification of the individual test case is more expensive, because for every output-representative the corresponding input value combination causing this output must be determined • Regard also the output values for the equivalence classes with incorrect values

13. Black Box Testing Techniques: Equivalence Class Partitioning Method(6) Boundaries of the equivalence classes • Partitioning into equivalence classes and selecting the representative values should be carefully done • Probability of failure detection is highly dependent upon the quality of the partitioning, as well as which test cases are executed • The best values are those verifying the boundaries of the equivalence classes • Often misunderstanding or inaccuracies in the requirements at these spots • E.g., “… more than 32 credits…” might mean • The value 32 being inside equivalence class (EC : x >= 32) or • Outside equivalence class (EC : x > 32) • Additional test case with x=32 might detect a misunderstanding

14. Black Box Testing Techniques: Equivalence Class Partitioning Method(7) Equivalence class construction for integer values(1) • All possible classes are to be identified • Example : based on integer parameter extras of the function calculate_price() • the domain of inputs partitioned into equivalence classes Incorrect values: Numbers that are greater or Smaller than the range of the Applicable interval or every Nonnumeric value including Floating point numbers

15. Black Box Testing Techniques: Equivalence Class Partitioning Method(8) Equivalence class construction for integer values(2) • Further divide the equivalence classes with correct values because negative and positive values often must be treated differently • Zero is a further input value that often lead to failures • Refinement of Equivalence classes and representatives • Result: the equivalence class method, taking into account boundary values, generate the following test values: (“f”, MIN_INT, -123, -1, 0, 654 MAX_INT)

16. Black Box Testing Techniques: Equivalence Class Partitioning Method(9) Guidelines for Determining Equivalence class(1) • For the input as well as for the outputs, identify the restrictions and conditions in the specification • For every restriction or condition, partition into equivalence classes: • If a continuous numerical domain is specified, then create onevalid and two invalid ECs • E.g., “the item count can be from 1 to 999”, identify one vEC (1 < item < 999) and two iEC (item count < 1 and item count > 999) • If a number of values should be entered, then create one valid (with all possible correct values) and two invalid ECs (less and more than) • E.g., “one through six owners can be listed for the car”, identify one vEC and two iEC (no owner and more than six owners)

17. Black Box Testing Techniques: Equivalence Class Partitioning Method(10) Guidelines for Determining Equivalence class(2) • For every restriction or condition, partition into equivalence classes: • If a set of values is specified where each value may possibly be treated differently, then create one vEC for each value of the set (containing exactly this one value) and one additional iEC (containing all possible other values) • E.g., “type of vehicle must be BUS, TRUCK, TAXI, MOTORCYCLE”, identify one vEC from the set and one iEC (“BICYCLE”) • If there is a condition that must be fulfilled, then create one vEC and one iEC • E.g., “first character of the identifier must be a letter”, identify one vEC (it is a letter) and one iEC (it is not a letter) • If there is any doubt that the values of one EC are treated equally, then the EC should be further divided into subclasses

18. Black Box Testing Techniques: Equivalence Class Partitioning Method(11) Test Cases (1) • Combination of the representatives • Rules for test case determination • The representative values of all vECs should be combined to test cases, meaning that all possible combinations of vECs will be covered. Those combinations build “valid test case” or “positive test case” • Separate test for invalid value • The representative value of an iEC can only be combined with representatives of other vECs. Thus every iEC leads to an additional “invalid test case” or “negative test case”. • If a test case combines more than one incorrect value, defect masking may result • Thus only one of the possible exceptions is actually triggered and tested

19. Black Box Testing Techniques: Equivalence Class Partitioning Method(11) Test Cases (2) • Restriction of the number of test cases • Reducing the number of valid test cases as the result of multiplicative combination of representatives • Rules for test case restriction • Combine the test cases and sort them by frequency of occurrence. Prioritize the test cases in this order. That way only relevant test cases are tested • Test cases including boundary values or boundary value combinations are preferred • Combine every representative of one EC with every representative of other EC (dual combination) • Ensure as minimum criteria that every representative of an EC appears in at least one test case • Representatives of iECs should not be combined with representatives of other iECs

20. Black Box Testing Techniques: Equivalence Class Partitioning Method(11) Test Completion Criteria • Define as the percentage of executed ECs in comparison to the total number of specified EC EC-coverage = (number of tested EC / total number of EC) * 100% • Test coverage is a measurable criterion for finishing testing

21. Black Box Testing Techniques: Equivalence Class Partitioning Method(11) The value of ECP • Contribute to a complete test where specified conditions and restrictions are not overlooked • Minimize the generation of unnecessary test cases • EC can be determined by for input and output • Can also be prepared for internal values and states, time dependent values, and interface parameters • Can be used in system testing, integration testing, and component testing • Only single input or output conditions are considered, while possible dependencies or interactions between conditions are ignored • May not consider boundary problems

22. Black Box Testing Techniques: Boundary Value Analysis (BVA) (1) • Faults often appear at the boundaries of ECs or boundary conditions • This is because of misunderstanding or unclear definition • Boundary conditions are those situations directly on, above, and beneath the edges of input ECs and output ECs • BVA differs with ECP in two aspects: • Rather than selecting any element in an EC as representative, BVA requires the one or more elements be selected such that each edge of the EC is the subject of a test • Rather than just focusing attention on the input conditions, test cases are also derived by considering the output ECs

23. Black Box Testing Techniques: Boundary Value Analysis (BVA) (2) Guidelines for BVA • If an input condition specifies a range of values, write test cases for the ends of the range, and invalid-input test cases for situations just beyond the ends. • E.g., if the valid domain is [-1.0,…,+1.0], write the test cases for the situations -1.0, +1.0, -1.001, +1.001 • If an input condition specifies a number of values, write test cases for the minimum and maximum number of values and one beneath and beyond these values • E.g., if an input file can contain 1-255 records, write test cases for 0, 1, 255, 256

24. Black Box Testing Techniques: Boundary Value Analysis (BVA) (3) Guidelines for BVA • If an output domain serve as the basis, then the analysis can be done as follows: • The output of the test object is an integer value between 500 and 1000. Test output that should be achieved: 500, 1000, 499, 1001 • If the permitted number of output values is to be tested, proceed just as with the number of input values: if outputs of 1 to 4 data values are allowed, the test outputs to produce are: 1, 4 as well as 0 and 5 data values • If the input or output of a program is an ordered set (a sequential file, a linear list or a table), focus attention on the first and last elements of the set • If complex data structures are given as input or output, an empty list or zero matrixes can be considered a boundary value

25. Black Box Testing Techniques: Boundary Value Analysis (BVA) (4) Guidelines for BVA • For numeric calculations, values that are close together, as well as values that are far apart, should be taken into consideration as boundary values • For iECs, BVA is only useful when different exception handling for the test object is expected depending on an EC boundary • Extremely large data structures, lists, tables, etc. should be chosen. For example, those that exceed buffer, file, or data storage boundaries; to check behavior in extreme cases

26. Black Box Testing Techniques: Boundary Value Analysis (BVA) (5) Test Cases • The valid boundaries inside an EC may be combined as test cases • The invalid boundaries must be verified separately and cannot be combined with other invalid boundaries

27. Black Box Testing Techniques: Boundary Value Analysis (BVA) (6) Test Completion Criteria BV-Coverage = (number of tested BV/total number of BV) *100% The Value of the Technique • Should be done together with ECP • Require a lot of creativity to define test data at boundaries

28. Black Box Testing Techniques: State Transition Testing(1) • Base on history of execution or event or input, influences of the output and how the test object behaves • Test object starts from an initial state and then come into different states • Events trigger state transitions • Event is normally a function invocation • State transitions can involve actions • The test object can be a complex system with different system states, as well as a class in OOS with different states • Finite state machine, state diagrams, or state transition tables model this behavior

29. Black Box Testing Techniques: State Transition Testing(2) • Example: Stack Top Pop [height > 1] Push [height < max -1] Push* top initialize Push [height = max – 1] push empty filled full Pop [height = 1] pop delete

30. Black Box Testing Techniques: State Transition Testing(3) • A possible test case for accepting strings Precondition: stack is initialized; state is “empty” Input: push (“hello”) Expected result: stack contains “hello” Post-condition: state of the stack is “filled” • Further test cases for the stack example • Test case 1: initialize [empty], push [filled], push, push, push [full] • Test case 2: initialize [empty], push [filled], top, pop [empty], delete *note: not all states have been reached

31. Black Box Testing Techniques: State Transition Testing(4) Test criteria • Should execute all specified functions of a certain state at least once • The compliance between the specified and the actual behavior can thus be checked. • To identify test cases • The finite state machine is transformed into a transition tree which includes certain sequences of transitions

32. Black Box Testing Techniques: State Transition Testing(5) Building a transition tree • The initial or start state is the root of the tree • For every possible transition from the initial state to a following state in a transition diagram, the transition tree receives a branch from its root to a node, representing this next state • The process for step 2 is repeated for every leaf in the tree (every newly added node) until one of the following two end-conditions is fulfilled: • The corresponding state is already included in the tree on the way from the root to the node. This end-condition corresponds to one pass of a cycle in the transition diagram • The corresponding state is a final state.

33. Black Box Testing Techniques: State Transition Testing(6) Building a transition tree (example) initial initialize empty delete push deleted filled pop pop push push pop empty filled full filled filled top pop push* full full filled

34. Black Box Testing Techniques: State Transition Testing(7) • For the example: there are 8 different paths. Each path represents a test case • In addition, the reaction of the state machine for wrong usage must be checked • The functions are called in state in which they are not supposed to be called (test of robustness) • E.g., to delete a stack while in “full” state • So, extend the transition tree to include a branch for every function from every node

35. Black Box Testing Techniques: State Transition Testing(8) Building a transition tree (example) initial pop initialize failure empty delete push failure top deleted delete filled failure pop pop push push pop empty filled full filled filled delete top pop push* failure full full filled

36. Black Box Testing Techniques: State Transition Testing(9) • Also good for system testing • E.g., to test GUI by modeling it as a finite state machine • Screen and user controls as states • Input reactions as state transition • Appropriate test cases and test coverage can be identified from the model

37. Black Box Testing Techniques: State Transition Testing(10) Test Cases • The following information is necessary for a complete definition of a state-based test case • The initial state • The input • The expected behavior • The expected final state • Further, for each expected transition of the test case the following aspects must be defined • The state before the transition • The initiating event that triggers the transition • The expected reaction triggered by the transition • The net expected state

38. Black Box Testing Techniques: State Transition Testing(10) Test Completion Criteria • Every state has been reached at least once • Every transition has been executed at least once • Every transition violating the specification has been checked • Percentages can be defined for the above

39. Black Box Testing Techniques: State Transition Testing(10) The value of this technique • Apply where states are important and where the functionality is influenced by the state • State transition testing is important for object-oriented testing • Takes into account the special aspects of object orientation

40. Black Box Testing Techniques: Cause-effect Graphing Technique(1) • Previous techniques do not regard combinations of input circumstances • Cause-effect graphing is a technique that uses the dependencies for identification of the test cases • The logical relationships between causes and their effects in a component or a system are displayed in a cause-effect graph • Every cause is described as a condition that consists of input conditions (or combinations of those) can be TRUE or FALSE • The conditions are connected with logical operators (e.g, AND, OR and NOT)

41. Black Box Testing Techniques: Cause-effect Graphing Technique(2) Basic cause-effect graph symbols NOT identify a b a b If a is TRUE, b is FALSE else b is TRUE If a is TRUE, b is TRUE else b is FALSE a a AND OR   b d c If a and b are TRUE, c is TRUE else c is FALSE If a or b or c is TRUE d is TRUE else d is FALSE b c

42. Black Box Testing Techniques: Cause-effect Graphing Technique(3) - example • withdrawing money at an ATM. In order to get the money from the machine, the following conditions must be fulfilled: 1. The bankcard is valid 2. The PIN must be correctly entered 3. The maximum number of PIN input is three 4. There is money in the machine, and in account • The following actions/effects are possible at the machine: 10. Reject card 11. Ask for another PIN input 12. “eat” the card 13. Ask for alternate dollar amount 14. Pay the requested amount of money

43. Black Box Testing Techniques: Cause-effect Graphing Technique(4)- example 10 1 11  2  12 3  13 4  14

44. Black Box Testing Techniques: Cause-effect Graphing Technique(5)- example • The graph must be transformed to a decision table • A decision table has four quadrants • The upper left is list of the inputs (causes) • The upper right is combinations of inputs • The lower left consists of the effects • The lower right is the expected effects • The steps are: • Choose an effect • Looking in the graph, find combinations of causes that have the effect and combinations that do not have this effect • Add one column into the table for every one of these cause-combinations and the cause states of the remaining effects • Check if decision table entries occur several times and, if yes, delete them

45. Black Box Testing Techniques: Cause-effect Graphing Technique(6)- example

46. Black Box Testing Techniques: Cause-effect Graphing Technique(7) • Test cases • Every column is a test case • The table defines the logical test cases • Must feed with concrete data values • Annotate necessary preconditions and post-conditions • The value of the technique • May reveal combinations which are not included when using other test case design techniques • However, errors can result from optimization of the table • Graph and table may grow very quickly and lose readability when the number of conditions and dependent actions increases • Without support by tools, the technique is not easily applicable