What is testing?

1. What is testing? • Testing is not showing that there are no errors in the program. • Testing cannot show that the program performs its intended goal correctly. So, what is testing? Testing is the process of executing the program in order to find errors. A successful test is one that finds an error.

2. Some drawbacks of testing • There are never sufficiently many test cases. • Testing does not find all the errors. • Testing is hard and takes a lot of time. • Testing is still a largely informal task.

3. Black-Box (data-driven, input-output) testing The testing is not based on the structure of the program (which is unknown). In order to ensure correctness, every possible input needs to be tested - this is impossible! The goal: to maximize the number of errors found.

4. White-Box testing Is based on the internal structure of the program. There are several alternative criterions for checking “enough” paths in the program. Even checking all paths (highly impractical) does not guarantee finding all errors (e.g., missing paths!)

5. Some testing principles • A programmer should not test his/her own program. • One should test not only that the program does what it is supposed to do, but that it does not do what it is not supposed to. • The goal of testing is to find errors, not to show that the program is errorless. • No amount of testing can guarantee error-free program. • Parts of programs where a lot of errors have already been found are a good place to look for more errors. • The goal is not to humiliate the programmer!

6. Inspections and Walkthroughs • Manual testing methods. • Done by a team of people. • Performed at a meeting (brainstorming). • Takes 90-120 minutes. • Can find 30%-70% of errors.

7. Code Inspection • Team of 3-5 people. • One is the moderator. He distributes materials and records the errors. • The programmer explains the program line by line. • Questions are raised. • The program is analyzed w.r.t. a checklist of errors.

8. An example: Sieve of Erathosthenes 2 3 5 6 - is dropped because it is divisible by 2. 7 - is added to the next additional slot. 8 - dropped (divisible by 2). 9 - dropped (divisible by 3). ...

9. int P[101]; int V=1; int N=1; int I; main() { while (N<101) { for (I=1;I<=N;I++) if (P[I]=0) { P[I]=V++ ; N++ ; } else if (V%P[I]==0) I=N+1; else I++} } Is this program correct? The Sieve program

10. Data declaration All variables declared? Default values understood? Arrays and strings initialized? Variables with similar names? Correct initialization? Control flow Each loop terminates? DO/END statements match? Input/output OPEN statements correct? Format specification correct? End-of-file case handled? Checklist for inspections

11. Walkthrough • Team of 3-5 people. • Moderator, as before. • Secretary, records errors. • Tester, play the role of a computer on some test suits on paper and board.

12. Selection of test cases (for white-box testing) The main problem is to select a good coverage criterion. Some options are: • Cover all paths of the program. • Execute every statement at least once. • Each decision has a true or false value at least once. • Each condition is taking each truth value at least once. • Check all possible combinations of conditions in each decision.

13. Cover all the paths of the program Infeasible. Consider the flow diagram on the left. It corresponds to a loop. The loop body has 5 paths. If the loops executes 20 times there are 5^20 different paths! May also be unbounded!

14. How to cover the executions? IF (A>1)&(B=0) THEN X=X/A; END; IF (A=2)|(X>1) THEN X=X+1; END; • Choose values for A,B,X. • Value of X may change, depending on A,B. • What do we want to cover? Paths? Statements? Conditions?

15. By choosing A=2,B=0,X=3 each statement will be chosen. The case that the tests fail is not checked! IF (A>1)&(B=0) THEN X=X/A; END; IF (A=2)|(X>1) THEN X=X+1; END; Edge coverageExecute every statement at least once

16. Can be achieved using A=3,B=0,X=3 A=2,B=1,X=1 Problem: Does not test individual conditions. E.g., when X>1 is erroneous in second decision. IF (A>1)&(B=0) THEN X=X/A; END; IF (A=2)|(X>1) THEN X=X+1; END; Decision coverageEach decision has a true and false outcome at least once.

17. For example: A=1,B=0,X=3 A=2,B=1,X=0 lets each condition be true and false once. Problem:covers only the path where the first test fails and the second succeeds. IF (A>1)&(B=0) THEN X=X/A; END; IF (A=2)|(X>1) THEN X=X+1; END; Condition coverageEach condition has a trueand false value at least once.

18. A>1,B=0 A>1,B≠0 A<=1,B=0 A<=1,B≠0 A=2,X>1 A=2,X<=1 A≠2,X>1 A≠2,X<=1 IF (A>1)&(B=0) THEN X=X/A; END; IF (A=2)|(X>1) THEN X=X+1; END; Multiple Condition CoverageTest all combinations of all conditions in each test.

19. A=2,B=0,X=4 A=2,B=1,X=1 A=1,B=0,X=2 A=1,B=1,X=1 Note the X=4 in the first case: it is due to the fact that X changes before being used! IF (A>1)&(B=0) THEN X=X/A; END; IF (A=2)|(X>1) THEN X=X+1; END; A smaller number of cases:

20. How to find values for coverage? (A=2 | X>1) & ¬(A>1 & B=0) • Put true at end of path. • Propagate path backwards. • On assignment, relativize expression. • On “yes” edge of decision, add decision as conjunction. • On “no” edge, add negation of decision as conjunction. • May need to give more specific choice of covering condition, which ones are true or false. A>1 & B=0 yes no X=X/A A=2 | X>1 A=2 | X>1 yes true no X=X+1 true

21. How to find values for coverage? (A≠2 & X/A>1) & (A>1 & B=0) A>1 & B=0 yes A≠2 & X/A>1 Need to find a satisfying assignment: A=3, X=6, B=0 no X=X/A A≠2 & X>1 A=2 | X>1 yes true no X=X+1 true

22. Test case design for black box testing • Equivalence partition • Boundary value analysis • Cause-effect graphs

23. Test cases based on data-flow analysis • Partition the program into pieces of code with a single entry/exit point. • For each piece find which variables are set/used/tested. • Various covering criteria: • from each set to each use/test • From each set to some use/test.

24. Valid equivalence class Invalid equivalence class Specification condition Equivalence partition • Goals: • Find a small number of test cases. • Cover as much possibilities as you can. • Try to group together inputs for which the program would likely to behave the same.

25. Example: A legal variable • Begins with A-Z • Contains [A-Z0-9] • Has 1-6 characters. Valid equivalence class Specification condition Invalid equivalence class Starting char Starts A-Z Starts other 1 2 Chars [A-Z0-9] Has others 3 4 1-6 chars 0 chars, >6 chars Length 5 6 7

26. Equivalence partition (cont.) • Add a new test case until all valid equivalence classes have been covered. A test case can cover multiple such classes. • Add a new test case until all invalid equivalence class have been covered. Each test case can cover only one such class. Valid equivalence class Invalid equivalence class Specification condition

27. AB36P (1,3,5) 1XY12 (2) A17#%X (4) (6) VERYLONG (7) Valid equivalence class Specification condition Invalid equivalence class Starting char Starts A-Z Starts other 1 2 Chars [A-Z0-9] Has others 3 4 1-6 chars 0 chars, >6 chars Length 5 6 7

28. Boundary value analysis • In every element class, select values that are closed to the boundary. • If input is within range -1.0 -- +1.0, select values -1.001, -1.0, -0.999, 0.999, 1.0, 1.001. • If needs to read N data elements, check with N-1, N, N+1. Also, check with N=0.