When to Test Less

1. When to Test Less Authors: Tim Menzies and Bojan Cukic Presenter: Genet Balaker

2. Agenda • Introduction • The Theory of Testing • The Realty of Testing • The Average Shape of Software • Implementation • Conclusion

3. Introduction • Testing software in the large is different from testing it in the small • Small projects often lack funds to set up software test environments and have to depend on manual and simple automatic testing schemes.

4. Introduction (cont.) • a program's shape can help determine whether rapid testing with limited resources will be as effective as elaborate and expensive testing regimes. • for many small-scale projects, a small number of randomly selected tests will adequately probe the software

5. Theory of Testing • Black-box testing to detect faults in a software • How many random black-box tests do we need to find a fault? • A randomly chosen input has odds x that it stumble across some fault, • This input will miss that fault with odds of 1-x • If we conduct N random black-box tests, the odds of a failure not occurring is (1-x)N, so, the probability y of finding a fault in N random tests is y = 1- (1-x) N

6. Theory of Testing (cont.) • Black-box testing can be expensive. • Formal-methods testing • Benefit of formally checking a system is that such formal proofs can find faults more systematically than standard testing. • A single formal first-order query is equivalent to many black-box test inputs. • The cost of such rigorous formal analysis ca be impractically high

7. Theory of Testing (cont.) • We can only use formal methods to test small, critical portions of the system. • Even when applying formal verification, its recommended to use black-box testing. • Another approach to testing small-scale software is white-box testing. • Test inputs exercise K different partitions. • Each partition exercises one interesting feature of a program

8. Theory of Testing (cont.) • The odds of detecting a fault with white-box methods are nearly the same as black-box methods. • White-box method using k partitions is only k times better at finding errors than black-box method testing, but it slightly reduces the number of tests required • Building the partitions is time consuming.

9. The Realty of Testing • Given the mathematics of black-box testing, software-in-the-small project doesn’t crash more often. • Program shape = the shape of the pathways within the program • Numerous and tangled like spaghetti • Few and not complex • What should the shape of the software be to adequately test it using limited resources?

10. The realty of Testing (cont.) • Studies suggest that simple shapes are common • Du-pathways • A link from where a variable is defined to where it is used • As the du-pathways shrink, the number of inputs required to reach part of the program also shrinks • Control flow diagrams • They link program statements: while, repeat, and for statements • They grow as the program grow • A worst-case control-flow • Saturation effect • Consistent with programs containing either many portions with simple shapes or many portions that are so twisted in shape

11. Shape of a software • If it is possible to show a software has the right shape, it can be tested quickly. • What’s the right shape of a software? • What’s an average shape of a software?

12. Shape of a software (cont.) • Assumptions about the tested program’s structure to find out the shape • Programs are networks connecting system concepts • A fault explanation tree is a tree whose leaves are inputs and whose root is fault • Testing is a process of trying to generate a fault explanation tree from the program network • An explanation tree has edges and nodes • Testing is the process of extracting NAYO trees (no, and, yes and or)

13. Implementation • Based on the above assumptions, it’s possible to simulate the construction of an explanation tree across a program • It is possible to characterize the tree’s shape by the number of tests N required to reach it’s root to find a fault • Complex shaped trees require large N

14. Implementation (cont.) simple if 0 < N < 102 Shape = moderate if 102 <= N <104 hard 104 <= N <106 overly complex if N >= 106

15. Implementation Given in inputs to a NAYO network with V nodes, the odds of hitting a fault straight away from the inputs is Xo = in/v (A) The probability of reaching an and node with andp parents is the probability of reaching all its parents: Xand = xiandp (B) where xi is the probability we computed in the prior step of the simulation (and the base case of x i = 0 is computed from Equation A). The probability of reaching an or node with orp parents is the probability of not missing any of its parents; that is, Xor= 1 − (1 − x)orp (C)

16. Implementation (cont.) If the ratio of and nodes in a NAYO network is andf, then the ratio of or nodes in the same network is or f = 1 – andf. The odds of reaching some random node xj is the weighted sum of the probabilities of reaching and nodes or or nodes xj = andf * xand * orf * xor (D) The number of tests required to be 99% sure of finding a fault with probability xj: y = 0.99 = 1 − ((1 − xj)N) (E) Therefore: N = log(1- 0.99) / log(1-xj)

17. Implementation (cont.) Results from the Simulation Runs ____________________________________ Classification Threshold Percent Simple 0 < N < 102 36 Moderate 102 <=N < 104 19 Hard 104 <= N < 106 25 Overly complex N > =106 20

18. Implementation (cont.) • Over half of the time, 100 random tests will yield as much information as much more prolonged series of tests • 10,000 randomly selected tests will probe the program as much as very prolonged series of tests will.

19. conclusion • Software-in-the-small projects should routinely execute overnight test runs in which tens of thousands of test cases are generated at random • software-in-the-small projects, a cost-cutting heuristic is to avoid elaborate and expensive testing regimes.