Technology of Test Case Generation

1. Technology ofTest Case Generation Levi Lúcio University of Geneva Marko Samer Vienna University of Technology

2. Overview Introduction Model Checking Symbolic Execution Theorem Proving Conclusions H. S. Hong, L. Lúcio, M. Samer: Technology of Test Case Generation

3. Introduction • In this presentation three technologies for test case generation are introduced; • Symbolic Execution; • Theorem Proving; • Model Checking (presentation not available); • All the models used by these methods can be reduced to state machines; • Test case generation can be seen as state space exploration for traces obeying a criteria. H. S. Hong, L. Lúcio, M. Samer: Technology of Test Case Generation

4. Overview Introduction Model Checking Symbolic Execution Theorem Proving Conclusions H. S. Hong, L. Lúcio, M. Samer: Technology of Test Case Generation

5. Overview Introduction Model Checking Symbolic Execution Theorem Proving Conclusions H. S. Hong, L. Lúcio, M. Samer: Technology of Test Case Generation

6. Symbolic ExecutionTalk Overview • The technique • How symbolic execution works; • Issues related to the approach; • Symbolic execution of abstract models; • Test case generation • A classification of test case generation frameworks using symbolic execution; • Abstract model based test case generation; • Code based test case generation. H. S. Hong, L. Lúcio, M. Samer: Technology of Test Case Generation

7. Symbolic Execution • Technique inventedin the 1970s for verifying the consistency of code; • The main principle consists of executing the code with symbolic inputs… • … in which a symbolic input consists of replacing a « real input » (number, string, structure, object) with a symbol; • For example, if the program asks a number to the user, « a » would be passed to the program as a representation of all possible numbers. H. S. Hong, L. Lúcio, M. Samer: Technology of Test Case Generation

8. The technique • Allows exploring the possible control paths in an program. Calculates thepath conditionfor eachcontrol path, composed of symbolic equations. Int foo (int a, int b) { 1 a++; 2 if (a>b) 3 a=a-b; 4 else 5 a=b-a; 6 if (a<=-1) 7 a=-a; 8 return a } H. S. Hong, L. Lúcio, M. Samer: Technology of Test Case Generation

9. (1) a: a1+1, b:a2 PC: True (2 -true) (2 - false) a: a1+1, b:a2 PC: a1+1>a2 a: a1, b:a2 PC: a1+1<=a2 (3) … a: a1+1-a2, b:a2 PC: a1+1>a2 (6 - true) (6 - false) a: a1+1-a2, b:a2 PC: a1-a2>-1, a1-a2<=-2 a: a1+1-a2, b:a2 PC: a1-a2>-1, a1-a2>-2 (8) No solution for PC equations: Infeasible Control Path (search backtracks) a: a1+1-a2, b:a2 PC: a1-a2>-1, a1-a2>-2 The technique (cont) a: a1, b:a2 PC: True Int foo (int a, int b) { 1 a++; 2 if (a>b) 3 a=a-b; 4 else 5 a=b-a; 6 if (a<=-1) 7 a=-a; 8 return a } H. S. Hong, L. Lúcio, M. Samer: Technology of Test Case Generation

10. The technique (cont) • From the symbolic execution of foo, one can retrieve the following information: • Each path condition fully documents the decisions taken during the execution of the code. H. S. Hong, L. Lúcio, M. Samer: Technology of Test Case Generation

11. Test case generation • It is possible to generate test cases by finding values that satisfy the path condition equations: • Test case (a=1, b=3) would force the execution of foo to follow the control path 1,2,5,6,8; • Test cases are thus created as by-products of the activity of symbolic execution; • Symbolic execution uncovered that in the case of foo instruction 7 is never reached... H. S. Hong, L. Lúcio, M. Samer: Technology of Test Case Generation

12. Issues related to the approach • Dealing with loops • Loops generate infinite control paths (unless the number of iterations is known). Solution? • Prompt the user at each loop iteration; • Establish an upper bound for the number of iterations; • Try to automatically find a fixed point to the loop (costly). • Solving path condition equations • Necessary for: • understanding when a path is infeasible; • generating test cases. • Algorithms heavy on computing resources. H. S. Hong, L. Lúcio, M. Samer: Technology of Test Case Generation

13. Code Abstract Model (state machine) Symbolic Execution Test Cases Symbolic execution of abstract models • Up until now we have considered symbolic execution only as a “white box” verification technique… • However, nothing prevents from applying the same technique to an abstract model (state machine) • State space of a model can be symbolically searched for interesting control paths; • Symbolic execution reduces state space explosion by associating classes of inputs. H. S. Hong, L. Lúcio, M. Samer: Technology of Test Case Generation

14. Symbolic execution for test case generation • In the text several frameworks for test case generation from different models of the application are studied; • We classify the frameworks according to two axis: Synergies with other verification techniques Model Checking Theorem proving Abstract (B, AUTOFOCUS, CO-OPN), Code (Java) Application model H. S. Hong, L. Lúcio, M. Samer: Technology of Test Case Generation

15. Abstract Model (state machine) Logic Programming Model Test Cases Translation step Symbolic state space search Abstract model based test case generation • In the text three frameworks are described that have their starting point on different abstract models: • B (framework from Legeard, Peureux et al) • AUTOFOCUS (framework from Pretschner et al) • CO-OPN (framework from Buchs et al) • The models can be considered as state machines and the technique for test case generation is similar: H. S. Hong, L. Lúcio, M. Samer: Technology of Test Case Generation

16. Abstract model based test case generation (cont) • The step of translation into a logic programming language (theorem prover) is necessary to: • Animate the (static) abstract specification; • Search the state space; • Perform symbolic execution. • Two kinds of logic programming languages may be used: • Pure Prolog; • Constraint Logic Programming (CLP) Languages: enabled to deal with numeric constraints at the unification level. H. S. Hong, L. Lúcio, M. Samer: Technology of Test Case Generation

17. Code based test case generation • Why take a detour from model based test case generation (title of the section)? • In the text we present a framework from Khurshid et al that generates test cases from Java code. It uses: • A model checker (Java PathFinder) to overcome some difficulties in symbolic execution; • The symbolic execution technique to overcome some difficulties in model checking. • We think an insight on the framework is enriching for the discussion… H. S. Hong, L. Lúcio, M. Samer: Technology of Test Case Generation

18. Logic Formulas Model Checker State Space Formulas hold Formulas don’t hold Witness Trace Counter-example Trace Test generation from codeExample • Symbolic execution helps dealing with state space explosion by dealing with inputs as classes of values; • Model checking deals automatically with loops, recursion or method invocation… H. S. Hong, L. Lúcio, M. Samer: Technology of Test Case Generation

19. Java Code Instrumented Java code Test case traces Model checking Java Symbolic Execution libraries Test generation from codeExample (cont) • The code is instrumented to deal with symbolic values and accumulate path conditions (via Java libraries): • The algorithm that performs symbolic execution knows how to deal with complex dynamic structures (objects)! • A phase of test case instantiation is required… H. S. Hong, L. Lúcio, M. Samer: Technology of Test Case Generation

20. Overview Introduction Model Checking Symbolic Execution Theorem Proving Conclusions H. S. Hong, L. Lúcio, M. Samer: Technology of Test Case Generation

21. Overview Introduction Model Checking Symbolic Execution Theorem Proving Conclusions H. S. Hong, L. Lúcio, M. Samer: Technology of Test Case Generation