Part III: Execution – Based Verification and Validation

1. Part III: Execution – Based Verification and Validation Katerina Goseva - Popstojanova Lane Department of Computer Science and Electrical Engineering West Virginia University, Morgantown, WV katerina@csee.wvu.edu www.csee.wvu.edu/~katerina

2. Introduction Definitions, objectives and limitations Testing principles Testing criteria Testing techniques Black box testing White box testing Fault based testing Fault injection Mutation testing Hierarchy for test adequacy criteria Outline

3. Testing levels Unit testing Integration testing Top-down Bottom-up Regression testing Validation testing Non-functional testing Configuration testing Recovery Testing Safety testing Security testing Stress testing Performance testing Outline

4. Black box testing • Also called behavioral or functional testing • The program test cases are based on the system specification • Test planning can begin early in the software process

5. Black box testing

6. Black box testing - Equivalence partitioning • Input data and output results often fall into different classes where all members of a class are related • Each of these classes is an equivalence partition where the program behaves in an equivalent way for each class member • An equivalence class represents a set of valid or invalid states for input conditions

7. Black box testing - Equivalence partitioning • Equivalence classes may be defined according to the following guidelines: • If an input condition specifies a range, one valid and two invalid equivalence classes are defined • If an input condition requires a specific value, one valid and two invalid equivalence classes are defined • If an input condition specifies a member of a set, one valid and one invalid equivalence class are defined • If an input condition is Boolean, one valid and one invalid class are defined

8. Black box testing - Equivalence partitioning • Example • If input is a 5-digit integer between 10,000 and 99,999 (input condition range) • equivalence partitions are • less than 10,000 (invalid) • between 10,000 and 99, 999 (valid) • more than 99,999 (invalid) • Choose test cases at the boundary of these sets 9,999; 10,000; 99,999; 100,000

9. Black box testing – Boundary value analysis • Boundary value analysis (BVA) requires selection of test cases that exercise bounding values • Most software engineers intuitively perform BVA to some degree • BVA is a technique that complements equivalence partitioning • Rather than focusing solely on input conditions, BVA derives test cases from the output domain as well

10. Black box testing – Random testing • Random testing is essentially a black box testing technique in which a program is tested by randomly selecting some subsets of all possible inputs • The distribution of the inputs may be arbitrary or may reflect the distribution of the inputs in the application environment (i.e., operational profile)

11. White box testing • Also called glass-box testingor structural testing • Derivation of test cases according to program structure • Control flow coverage • Data flow coverage • Higher coverage is better • White box testing is not an alternative to black box testing. Rather, it is a complementary approach

12. White box testing • The starting point for white box testing is a program flow graph • Nodes represent one or more statements • Arcs (also called edges or links) represent flow of control

13. Binary search (Java)

14. Binary search program flow graph

15. Control flow coverage • All-paths coverage – traverse all possible paths • Equivalent to exhaustively testing the program • In general, all-paths coverage is not possible • Loops result in an infinite number of possible paths • If there are no loops, but only branching instructions, the number of possible paths increases exponentially with the number of branching points • There may be paths that are never executed (quite likely, the program contains a fault in this case)

16. Control flow coverage • All-statements coverage – covers all possible statements • Rather weak criterion • It is relatively easy to achieve 100% statement coverage

17. Control flow coverage • All-branches coverage – all possible branches are chosen at least once • Nodes that contain a condition, such as the boolean expression in an if statement, can be a combination of elementary predicates connected by logical operators • i>0 V j>0 requires at least two tests to guarantee that both branches are taken. For example: i=1,j=1 and i=0,j=1. Other possible combinations of through values of the atomic predicates (i=1,j=0 and i=0,j=0) need not be considered to achieve branch coverage • Extended branch coverage – requires that all possible combinations of elementary predicates in conditions to be covered

18. Control flow coverage • All-independent paths coverage • Independent path is any path through the program which traverses at least one new edge in the flow graph (exercising one or more new conditions) • 1, 2, 3, 8, 9 • 1, 2, 3, 4, 6, 7, 2 • 1, 2, 3, 4, 5, 7, 2 • 1, 2, 3, 4, 6, 7, 2, 8, 9 • If all independent program paths are executed we can be sure that • Every statement has been executed at least once • Every branch has been executed for true and false conditions

19. Control flow coverage • The number of independent paths in a program can be discovered by computing the cyclomatic complexity Cyclomatic complexity = Number of edges - Number of nodes + 2 = 11 - 9 + 2 = 4 • For programs without goto statements cyclomatic complexity equals number of conditions in a program • Design test cases to execute each of these paths • A dynamic program analyser may be used to check that paths have been executed

20. Data flow coverage • Data flow testing selects test paths of a program according to the location of definitions and uses of variables in the program • A variable is defined in a certain statement if it is assigned a (new) value DEF(S) ={X | statement S contains a definition of X} • After that, a new value will be used in subsequent statements USE(S) ={X | statement S contains a use of X}

21. Data flow coverage • Variable definition in statement X is alive in statement Y if there exists a path from X to Y in which that variable does not get assigned a new value in some intermediate node; such a path is called definition-clear • Two typesof variable use: • P-uses – predicate uses (like those in the conditional part of an if-statement) • C-uses – all other uses (like uses in computations or I/O statements) • With data flow analysis too, we may define test adequacy criteria and use these criteria to guide testing

22. Data flow coverage • All-uses coverage – traverse a definition-clear path between each definition of a variable an each (P- or C-) use of that definition and each successor of that use • Each successor is included to force all branches following a P-use to be taken • All-DU-paths – requires that each definition-clear path is either cycle-free or a simple cycle • Slightly stronger criterion than all-uses coverage

23. Data flow coverage • All-def coverage – requires each definition to be used at least once • All-C-uses/Some-P-uses coverage – requires definition-clear paths from each definition to each computational use; if definition is used only in predicates, at least one definition-clear pat to a predicate use must be executed

24. Data flow coverage • All-P-uses/Some-C-uses coverage – requires definition-clear paths from each definition to each predicate use; if definition is used only in computations, at least one definition-clear path to a computational use must be executed • All-P-uses coverage – requires definition clear paths from each definition to each predicate use

25. Telcordia’s ATAC – white box testing tool http://xsuds.argreenhouse.com

26. ATAC • ATAC is a coverage analysis tool that aids in testing programs written in the C or C++ programming language • Using ATAC focuses on three main activities: • instrumenting the software to be tested • occurs at compile-time • large systems can be instrumented a-piece-at-a-time • executing software tests • determining how well the software has been tested • display uncovered source code • generate reports that reveal the current coverage measures for each criteria

27. Program instrumentation, test execution and coverage analysis

28. ATAC • ATAC highlights the covered and uncovered blocks in the source code and prioritizes them into an order in which you should try to cover them • Constructing the tests is the role of the tester – ATAC does not construct the tests or determine what inputs are needed to cover the uncovered code • ATAC, however, simplifies the tester's job by guiding him or her into creating a small set of high-efficiency, high-leverage test cases that yield high coverage quickly

29. ATAC – Display of uncovered source code

30. ATAC - Display of uncovered source code • Each color represents a certain weight • If, for example, a block has weight 30, it means any test case that causes that block to be exercised, or covered, is guaranteed to cover a minimum of 29 other blocks as well • White represents zero weight and red represents the highest weight among all blocks in the file • If a block is highlighted in white, it means that it has already been covered by a test case and covering it again will not add new coverage • If, on the other extreme, a block is highlighted in red, it means that it has not been covered by any test case so far and covering it first is the most efficient way to add new coverage to the program

31. ATAC - Display of uncovered source code

32. ATAC - Display of uncovered source code

33. ATAC • Coverage criteria • Block - a sequence of instructions that, except for the last instruction, is free of branches and function calls • a block may contain more than one statement if no branching occurs between statements • a statement may contain multiple blocks if branching occurs inside the statement • an expression may contain multiple blocks if branching is implied within the expression (e.g., conditional, logical-and, and logical-or expressions)

34. ATAC – Basic blocks Block 1 consists of a logical-expression embedded within a compound conditional-expression Block 2 consists of an entire conditional expression Block 3 consists of the entire body of an if-statement

35. ATAC – Report on current coverage

36. ATAC – Report on current coverage

37. ATAC • Coverage criteria • Decision (i.e., branch coverage) • C-uses • P-uses • All-uses (sum of P-uses and C-uses coverage measures)

38. ATAC - An example C-uses

39. ATAC - An example P-uses

40. ATAC • Other coverage criteria • function-entry - all functions are called at least once • function-return - all explicit and implicit returns or exits from a function are executed at least once • complete function- return coverage usually guarantees complete function-entry coverage, since, a function usually has at least one return or exit • function-call - each call to a function is covered at least once • Complete function-call coverage does not guarantee complete function-entry coverage since it is possible to have a function that does not contain any function calls

41. ATAC – Report on current coverage

42. Fault based testing • Aimed at finding a test set with a high ability to detect faults • Two techniques • Fault injection • Mutation testing

43. Fault injection • Non-software example • We want to estimate the number of pikes in Lake Soft • Catch a number of pikes, N, in Lake Soft • Mark them and throw them into Lake Soft • Catch a number of pikes, M, in Lake Soft • Suppose that M’ out of the M pikes are found to be marked, the total number of pikes originally present in Lake Soft is then estimated as (M-M’)N/M’

44. Fault injection • Fault injection • Artificially inject (seed) a number faults in program • When program is tested, we will discover both injected faults and new ones • Total number of faults is then estimated from the ration of those two numbers

45. Fault injection • Several assumptions underlie this method • Injected faults are representative for real faults • Both real and injected faults have the same distribution • The results can be trusted if we find many injected faults and relatively few others • The opposite in not true

46. Mutation testing • A large number of variants (mutants) of a program is generated • Each of these mutants slightly differs from the original version • Usually mutants are obtained by mechanically applying a set of simple transformations called mutation operations • Replace a constant by another constant • Replace a variable by another variable • Replace a constant by a variable • Replace an arithmetic operator by another arithmetic operator • Replace a logical operator by another logical operator • Insert a unary operator • Delete a statement

47. Mutation testing • Next, all these mutants are executed using a given test set • When a test produces a different result for one of the mutants, that mutant is said to be dead • Mutants that produce the same results for all the tests are said to be alive • Mutation adequacy score of a test set is given by D/M, where D is the number of dead mutants and M is the total number of mutants

48. Mutation testing • Suppose we have a program P with a component T • Two variants of mutation testing • strong mutation testing • Requires that tests produce different results for program P and a mutant P’ • weak mutation testing • Requires that component T and its mutant T’ produce different results; at the level of P this difference need not crop up

49. Mutation testing • Several assumptions underlie mutation testing • Incorrect versions of a program differ from a correct version by relatively minor faults • Tests that can reveal simple faults can also reveal complex faults

50. Comparison of test techniques • Most techniques are heuristic in nature and lack a sound theoretical basis • Manual test techniques rely heavily on the qualities of the participants in the test process • Relating different test adequacy criteria • Sound theoretical answer to the questions such as “Is All-uses adequacy criterion stronger or weaker that the All-branches adequacy criterion?”