Automatic Unit Tests Generation Tools

1. Automatic Unit Tests Generation Tools SEPL Seminar Benny Pasternak December 2007

2. Agenda • Brief Introduction • Automatic Generation Tools • JCrasher • Symstra • GenUTest • Summary

3. Definition • A unit is the smallest testable part of an application • In the object oriented paradigm it is a class • A unit test consists of a fixed sequence of method invocations with fixed arguments • Unit test explores a particular aspect of the behavior of the Class Under Test (CUT)

4. Yet Another Calculator Example public class Calculator { public int add(int a, int b) { return a + b; } public int subtract(int a, int b) { return a - b; } public int multiply(int a, int b) { return a * b; } public int divide(int a, int b) { return a / b; } }

5. Small JUnit Test public class CalculatorTest { @Test public void add3and5Equals8() { Calculator calc = new Calculator(); int result = calc.add(4, 6); assertEquals(result, 10); } @Test(expected= java.lang.ArithmeticException.class) public void divideByZeroThrowsException() { Calculator calc = new Calculator(); int result = calc.divide(3,0); } @Test public void subtract5from6Equals2() { Calculator calc = new Calculator(); int result = calc.subtract(6, 5); assertEquals(result, 2); } }

6. Test run and feedback

7. Effective Unit Tests • Effective unit test should be: • Comprehensive – unit test should exercise all class methods and achieve high code coverage rate • Test in Isolation – CUT dependent objects (e.g., Logger, Serializer) should not be tested

8. StackInt push(int) int pop() int top() bool empty() reverse() Comprehensiveness Unit Test test2() test1() . . testn()

9. StackInt Unit Test push(int) int pop() int top() bool empty() reverse() test1() test2() . . testn() Isolation MockLogger Logger MockLinkedList LinkedList MockSerializer Serializer

10. Benefits • Enables the immediate detection of bugs introduced into a unit whenever code changes occur  Unit tests provide a safety net of regression tests and validation tests which encourage developers to refactor existing code. • Elimination of uncertainty in units  Simplifies integration • Provides a sort of living documentation of the system

11. However… • Writing effective unit tests is a hard and tedious task… • Lots of unit tests need to be written (unit tests code may be larger than project code) Provide tools to assist developers

12. Tool Classification • Frameworks – JUnit, NUnit • Mock Frameworks – EasyMock, jMock • Generation – Automatic generation of unit tests • Selection – Selecting a small set of unit tests from a large one • Prioritization – Deciding what the “best order” to run the tests

13. Unit Test Generation • Test Generation – generate a sequence of CUT method-calls • Test Assertion – Provide an assertion to determine if the test result is correct

14. Test Generation Techniques • Random Execution • random sequences of method calls with random values • Symbolic Execution • method sequences with symbolic arguments • builds constraints on arguments • produces actual values by solving constraints • Capture & Replay • capture real sequences seen in actual program runs or test runs

15. Test Assertion Generation Techniques • Manual (Design by Contract) • Uncaught Exceptions • Classifies a test as potentially faulty if it throws an unexpected exception • Operation Model • Infer an operational model from manual tests • Properties: objects invariants, method pre/post conditions • Properties violation  potentially faulty • Capture & Replay • Compare test results/state changes to the ones captured in actual program runs and classify deviations as possible errors.

16. Generation Tool Map Selection Generation Prioritization Operational Model Uncaught Exceptions JCrasher Jartege Random Execution Eclat JTest Rostra Symstra Symbolic Execution PathFinder Symclat Test Factoring Capture & Replay CR Tool SCRAPE GenuTest Substra

17. Tools Presented Today • JCrasher (Random & Uncaught Exceptions) • Symstra (Symbolic & User provided specifications) • GenUTest (Capture & Replay)

18. JCrasher – An Automatic Robustness Tester for Java (2003) Christoph Csallner Yannis Smaragdakis Available at http://www-tatic.cc.gatech.edu/grads/c/csallnch/jcrasher/

19. Goal • Robustness quality goal – “A class should not crash with an unexpected runtime exception, regardless of the parameters provided.” • Do not assume anything about domain • Robustness goal applies to all classes • Function to determine class under test robustness: expected exception type  pass unexpected exception type  fail

20. Parameter Space • Huge parameter space. • Example: m(int,int) has 2^64 param combinations • Covering all parameters combination is impossible • May not need all combinations to cover all control paths that throw an exception • Pick a random sample • Control flow analysis on byte code could derive parameter equivalence class

21. Architecture Overview

22. Type Inference Rules • Search class under test for inference rules • Transitively search referenced types • Inference Rules • Method Y.m(P1,P2,.., Pn) returns X: X  Y, P1, P2, … , Pn • Sub-type Y {extend | implements } X: X  Y • Constants: int  -1, 0 ,1 • Add each discovered inference rule to mapping: X  inference rules returning X

23. Generate Test Cases For a Method

24. Exception Filtering • JCrasher runtime catches all exceptions • Example generated test case: Public void test1() throws Throwable { try { /* test case */ } catch (Exception e) { dispatchException(e); // JCrasher runtime } • Uses heuristics to decide whether the exception is a • Bug of the class  pass exception on to JUnit • Expected exception  suppress exception

25. Exception Filter Heuristics

26. Symstra: Framework for Generating Unit Tests using Symbolic Execution (2005) Tao Xie Darko Wolfram David Marinov Schulte Notkin

27. Binary Search Tree Example public class BST implements Set { Node root; int size; static class Node { int value; Node left; Node right; } public void insert (int value) { … } public void remove (int value) { … } public bool contains (int value) { … } public int size () { … } }

28. Other Test Generation Approaches • Straight forward – generate all possible sequences of calls to methods under test • Cleary this approach generates too many and redundant sequences BST t1 = new Bst(); BST t2 = new Bst(); t1.size(); t2.size(); t2.size();

29. Other Test Generation Approaches • Concrete-state exploration approach • Assume a given set of method calls arguments • Explore new receiver-object states with method calls (BFS manner)

30. 1st Iteration remove(1) remove(2) remove(3) insert(3) insert(2) insert(1) 2 1 3 Exploring Concrete States • Method arguments: insert(1), insert(2), insert(3), remove(1), remove(2), remove(3) new BST()

31. Exploring Concrete States • Method arguments: insert(1), insert(2), insert(3), remove(1), remove(2), remove(3) new BST() 2nd Iteration remove(1) remove(1) remove(2) remove(3) insert(3) insert(2) insert(1) remove(2) 2 1 3 remove(3) insert(2) insert(3) 1 1 2 3

32. Generating Tests from Exploration • Collect method sequences along the shortest path new BST() 2nd Iteration remove(1) remove(1) remove(2) remove(3) insert(3) insert(2) insert(1) remove(2) 2 1 3 remove(3) insert(2) insert(3) BST t = new BST(); t.insert(1); t.insert(3); 1 1 2 3

33. Exploring Concrete States Issues • Not solved state explosion problem • Need at least N different insert arguments to reach a BST with size N • experiments shows memory runs out when N = 7 • Requires given set of relevant arguments • in our case insert(1), insert(2), remove(1), …

34. Concrete States  Symbolic States new BST() new BST() remove(1) remove(1) remove(2) remove(3) insert(x1) insert(3) insert(2) insert(1) remove(2) 2 1 3 x1 remove(3) insert(2) insert(3) insert(x2) X1<x2 1 1 x1 2 3 x2

35. Symbolic Execution • Execute a method on symbolic input values • Inputs: insert(SymbolicInt x) • Explore paths of the method • Build a path condition for each path • Conjunct conditionals or their negations • Produce symbolic states (<heap, path condition>) • For example X1<x2 x1 x2

36. Exploring Symbolic States public void insert(SymbolicInt x) { if (root == null) { root = new Node(x); } else { Node t = root; while (true) { if (t.value < x) { // explore rigtht subtree } else if (t.value > x) { // explore left subtree } else return; } } } size++; } new BST() S1 insert(x1) S2 x1 insert(x2) S4 S5 S3 X1= x2 X1<x2 X1>x2 x1 x1 x1 x2 x2

37. Generating Tests from Exploration • Collect method sequences along the shortest path • Generate concrete arguments by using a constraint solver new BST() S1 insert(x1) BST t = new BST(); t.insert(x1); t.insert(x2); S2 x1 insert(x2) S4 S3 X1<x2 X1>x2 x1 x1 X1>x2 x2 x2 BST t = new BST(); t.insert(-1000000); t.insert(-999999);

38. Results

39. Results

40. More Issues • Symstra uses specifications (pre, post, invariants) written in JML. These are transformed to run-time assertions • Limitations: • can not precisely handle array indexes • currently supports primitive arguments • can generate non-primitive arguments as sequence of method calls. These eventually boil down to methods with primitive arguments

41. GenUTest: A Unit Test and Mock Aspect Generation Tool Benny Pasternak Shmuel Tyszberowicz Amiram Yehudai

42. Motivation • Writing effective unit tests is a hard and tedious process • At maintenance phase, writing tests from scratch is not considered cost effective  • Corollary: Maintenance remains a difficult process • Goal: Automatically generate unit tests for projects in maintenance phase

43. Example • Developers are asked to create effective unit tests for an existing software project • StackInt is an implementation of integers’ stack, with the operations: • Push • Pop • Top • Empty • Reverse • Goal is to test StackInt effectively

44. new() stack :IntStack lst :LinkedList new() push(2) addFirst(2) push(3) addFirst(3) reverse() pop() removeFirst() 2 2 Obtaining Test Cases From Existing Tests • System/Module test that exercises IntStack as follows: • Test can be used to obtain test cases for unit tests …

45. GenUTest • Captures and records execution of IntStack during module/system tests in order to obtain test cases • Recorded events are used to generate unit tests for IntStack • Unit tests assist developers in the testing process

46. Example – Generated Unit Test new() Unit Test Code stack :IntStack lst :LinkedList new() 1 @Test public void testpop1() 2 { 3 // test execution statements 4 IntStack IntStack_2 = new IntStack(); // #1 5 IntStack_2.push(2); // #2 6 IntStack_2.push(3); // #3 7 IntStack_2.reverse(); // #4 8 int intRetVal6 = IntStack_2.pop(); // #5 9 10 // test assertion statements 11 assertEquals(intRetVal6,2); 11 } push(2) addFirst(2) push(3) addFirst(3) reverse() … pop() removeFirst() 2 2

47. StackInt Unit Test push(int) int pop() int top() bool empty() reverse() test1() test2() . . testn() Example Logger LinkedList Serializer

48. Around Advice print(“Before”); execute join point print(“After”); Pointcut 1 Pointcut 2 print(“Before”); print(“After”); Aspect Oriented Programming • Join points • object instantiation • method-calls • field setter/getter class StackInt { void reverse() { LinkedList newlst = new LinkedList(); int size = lst.size(); for (int i = 0; i < size; i++) { int elem = lst.get(i); newlst.addFirst(elem); } lst = newlst; } int pop() { int elem = lst.removeFirst(); return elem; } }

49. AOP – Quick Summary • Join points – well defined execution points in the control flow of the program (object instantiation, method-calls, field member access) • Pointcut – expression that specifies a set of join points • Advices – code specified to execute before, after, or aroundpointcuts • Aspects – The equivalent to class. Holds pointcut declarations and advices

50. Mock Advice mock object behavior code Pointcut 1 StackInt Unit Test push(int) int pop() int top() bool empty() reverse() test1() test2() . . testn() Pointcut 2 Mock Advice mock object behavior code Example Logger LinkedList Serializer