Evolutionary Testing Metaheuristic search techniques applied to test problems

1. Evolutionary TestingMetaheuristic search techniques applied to test problems Stella Levin Advanced Software Tools Seminar Tel-Aviv University 11.2004

2. Contents • Introduction • Metaheuristic Search Techniques • White-Box Testing • Black-Box Testing • Object-Oriented Testing • Non-Functional Testing • Search Based Software Engineering

3. Introduction - Why testing? • “The biggest part of software cost is the cost of bugs: the cost of detecting them, the cost of correcting them, the cost of designing tests and the cost of running those tests” - Beizer • “In embedded systems errors could result in high risk,endanger human life, big cost” - Wegener

4. Successful EA applications • NASA evolvable antenna Equal to 12 years working of experienced designer There is no guarantee that a design would be as good [8]

5. 1. Metaheuristic Search Problem characteristics • Large solution space • No precise algorithm, no “best” solution • Classification of “better” solution “Due to non-linearity of software (if, loops…) test problems are converted to complex, discontinuous, non-linear search spaces”- Baresel

6. Transform a problem to optimization problem • Candidate solution representation – individual • Fitness function for individual • Movement from one individual to another

7. Hill Climbing - “local” search • Select a point in the search space • Investigate neighboring points • If there is a better neighbor solution (with fitness function), jump to it • Repeat steps 2-3 until current position has no better neighbors Require definition of neighboring points

8. Hill Climbing - Problem Is there a bigger hill?

9. Simulated Annealing • Analogy of the chemical process of cooling of a material in a heat bath • If F(X2)<F(X1) then move to neighbor X2 • Else move to X2 with probability P=e^(-ΔF/T) • Initially T is high; T decreases more like hill climbing Require definition of neighbor and cooling function

10. Simulated Annealing

11. Evolutionary Algorithms • Genetic algorithms Developed by J. Holland in the 70s • Evolution strategies Developed in Germany at about the same time • Analogy with Darwin’s evolution theory, survival of the fittest

13. Evolutionary Algorithms • Selection: roulette wheel with fitness • Crossover: 01101 01000 11000 11101 Cross at random point • Mutation: 11101 10101 Random bit change

14. Genetic Programming • Program is an individual • Crossover and mutation of program’s abstract syntax tree • Particular use – to find functions which describe data

15. Genetic Programming - Crossover

16. 2. White-Box Testing • Statement coverage • Branch coverage • Specific path(statement) selection

17. White-Box Testing • Input variables (x1, x2, … Xk) • Program Domain D1•D2•…Dk • Individuals decode input of the program • Goal: to find input data that satisfies coverage criteria (statement / branch / path)

18. Fitness Function = AL + D • AL: Approximation Level acc. McMinn [3] Critical branch: branch missing the Target AL =(Number of critical branches between Target and diverging point) - 1

19. Fitness Function = AL + D D: branch distance If (x==y) then … if x!=y then D=abs(x-y) else D=0 D normalize to [0,1] Goal: min fitness (min D) if (x<y) then … If x>=y then D=x-y else D=0 If (flag) then … If flag==false then D=K else D=0

20. Example: Triangle Classification Input: int a,b,c from [0,15] • Sort a,b,c that a<=b<=c • If a+b<=c then NOT A TRIANGLE • If a==b or b==c then EQUILATERAL • If a==b and b==c then ISOSCELES • Else REGULAR

21. Optimization problem • Program domain: I•I•I • Individual string • Goal: branch coverage • Sub-goals: NOT A TRIANGLE, EQUILATERAL, ISOSCELES, REGULAR

22. Simulation by hand Goal: EQUILATERAL Generation: <9,13,5> REGULAR <10,4,2> NOT A TRIANGLE <3,11,7> NOT A TRIANGLE 1010 01 00 0010 1010 0111 0111 0011 10 11 0111 0011 1000 0010 <10,7,7> EQUILATERAL

23. GA vs. Random Testing Schatz[11] Comp(x,y) = x nesting “if” complexity; y condition complexity

24. Real Example • Schatz[11] • Autopilot system • 2046 LOC • 75 conditions • Branch cov. • Performance GA vs. random

25. Real Example Performance of gradient descent algorithm

26. White Box Testing Summary • Variety of problem mapping and fitness functions • Flag and state problems

27. 3. Black Box Testing - Tracey [4] • Specification with pre/post-conditions • Search for test input data that satisfy pre-condition && ! post-condition • Good fitness for data that is near to satisfy

28. int wrap_counter(int n) {//pre (n>=0 && n<=10) if (n>=10) i=0; else i=n+1; return i; //post (n<10 -> i=n+1) //post (n=10 -> i=0)} Goal1: n>=0 && n<=10 && (n<10 && i!=n+1) Goal2: n>=0 && n<=10 && (n=10 && i!=0)

29. Example Insert Error: if (n>10) i=0 Goal2: n>=0 && n<=10 && (n=10 && i!=0) n=2 i=3 : 0+0+(8+K)+0=8+K n=7 i=8 : 0+0+(3+K)+0=3+K n=10 i=11 : 0+0+0+0=0 FOUND!!!

30. Application • Applied to safety-critical nuclear protection system • Use simulated annealing and GA for search • Use mutation testing to insert errors • About 2000 lines of executable code • 733 different disjunctive goals • 100% error detection • The code was simple

31. 3. Black-Box Testing Automated Parking System [10] • Individual: geometry data of parking space and vehicle – 6 parameters • Fitness: minimum distance to collision area • Results: 880 scenarios-25 incorrect

32. Parking System - Results

33. Parking System - Results

34. 4. Object-Oriented Testing Tonella [5]: Unit Testing of Classes • Create object of class under test • Put the object to proper state Repeat 1 and 2 for all required objects • Invoke method under test • Examine final state

35. Individual String and Fitness $a=A():$b=B():$a.m(int,$b)@3 That means A a = new A(); B b = new B(); a.m(3,b); Goal: branch coverage Fitness: proportion of exercised decision nodes that lead to the target

36. Crossover • Crossover at random point after constructor and before tested method • Repair the individual string $a=A():$b=B(int):$a.m(int,$b)@1,5 $a=A(int,int):$b=B():$b.g():$a.m(int,$b)@0,3,4 $a=A():$b=B(int):$b.g():$a.m(int,$b) @1,4 $a=A(int,int):$b=B():$a.m(int,$b)@0,3,5

37. Mutations • Mutation of input value • Constructor change • Insertion of method call • Removal of method call $a=A():$b=B(int):$a.m(int,$b)@1,5 $a=A():$b=B(int):$a.m(int,$b)@1,7 $a=A():$c=C():$b=B($c):$a.m(int,$b)@1,7 $a=A():$c=C():$b=B($c):$b.f():$a.m(int,$b)@1,7

38. Public methods Branch Coverage Test cases

39. 5. Non-Functional TestingExecution Time Testing • Real-time systems: WC/BC exe time • Fitness: execution time for specified input • Problem: no sufficient guidance for search • Wegener[6] on real systems: • GA is better than random and hand-made • Problem with low probability branches • No guarantee to find WC/BC execution time

40. 6. SBSESearch Based Software Engineering • Module Clustering using simulated annealing, genetic algorithms • Cost/Time Estimation using genetic programming • Re-engineering using Program Transformation • Ryan[7]: Automatic parallelization using genetic programming

41. References • Goldberg “Genetic Algorithms” • McMinn “SBS Test Data Generation: A Survey” • McMinn “Hybridizing ET with the Chaining Approach” • Tracey “A Search Based Automated Test-Data Generation Framework for Safety-Critical Systems” • Tonella “Evolutionary Testing of Classes” • Wegener,Pitschinetz,Sthamer “Automated testing of real-time tasks” • Ryan “Automatic re-engineering of software using genetic programming” • Nasa “Intelligence report” • “Reformulating Software Engineering as a Search Problem” Clarke,Jones…(11 authors) • Buehler,Wegener “Evolutionary Functional Testing of an Automated Parking System” • McGraw, G., Michael, C., Schatz, M. “Generating Software Test Data by Evolution”