A Framework For Automated Parallel Random Unit Testing Of Sequential Programs

1. A Framework For Automated Parallel Random Unit Testing Of Sequential Programs by Martin Brown A Master’s Thesis Presented to the Faculty of the Computer and Information Sciences Department

2. 1. Introduction • Background A Framework for Automated Parallel Random Unit Testing of Sequential Programs

3. 1.1 Background • Definition of parallel computing • Definition of random testing • Definition of parallel software testing Introduction

4. 1.1.1 Parallel Computing • Parallel computing is the discipline of employing multiple processors running all at once to • solve the same problem in less time • to solve a larger problem in the same time Background

5. 1.1.2 Random Testing • Random testing is a black box testing technique that uses random number generators to produce random data to the inputs of a program Background

6. 1.1.3 Parallel Software Testing • Parallel software testing is the testing of sequential software on multiple processors Background

7. 2. Thesis Problem Statement • The power of parallel computing needs to be exploited in software testing. • Two proposed strategies • Data diversity • Technique diversity A Framework for Automated Parallel Random Unit Testing of Sequential Programs

8. 2.1 Data Diversity • Using a combination of data sources when generating random numbers Thesis Problem Statement

9. 2.2 Technique Diversity • Applying multiple testing techniques in parallel, e.g. boundary testing and random testing Thesis Problem Statement

10. 2.3 Thesis Problem Statement, cont’d • Can parallelism help to improve the quality of testing (test effectiveness)? • YES A Framework for Automated Parallel Random Unit Testing of Sequential Programs

11. 2.4 Scope of this Thesis • Develop a framework for automated parallel random unit testing of sequential programs • Implement data diversity through random testing • Evaluate the effectiveness of data diversity • Technique diversity – future work Thesis Problem Statement

12. 3. Research Contribution • Framework for parallel random unit testing • Demonstration of the effectiveness of parallel testing • Introduction of Parallel Java to the CIS Department A Framework for Automated Parallel Random Unit Testing of Sequential Programs

13. 3.1 Features of the Framework • An API for parallel programming in Java • A library for random number generation • A wizard that guides the tester in creating a test specification Research Contribution

14. 3.1 Features of the Framework, cont’d • The ability to save and load test specifications • The ability to store generated random test data. Research Contribution

15. 4. Related Work • Random unit testing using JML • Random testing of OO applications over time A Framework for Automated Parallel Random Unit Testing of Sequential Programs

16. 4.1 Random Unit Testing using JML • Leavens et al. • JML is a behavioral interface specification language for Java that uses pre- and post-conditions and invariants • JML is often used as a partial test oracle to random testing Related Work

17. 4.2 Random Testing of OO Applications • Ciupa, et al • Confirms the existence of data diversity in random testing • the seed used for an random number generator (RNG) can make an important difference in the number of bugs found over time • different seeds find different bugs Related Work

18. 5. Test Specification Wizard • guides the tester through the creation of a test specification A Framework for Automated Parallel Random Unit Testing of Sequential Programs

19. Test Specification System Under Test Test Specification Wizard

20. System Under Test Classes Under Test Class A Class B Class C Test Specification

21. Test Specification System Under Test Test Specification Wizard

22. Test Specification System Under Test Class Instance Variables Test Specification Wizard

23. Class Instance Variables • each selected constructor is associated with an instance variable • a, b, … z Test Specification

24. Test Specification System Under Test Class Instance Variables Test Sequences Test Specification Wizard

25. Test Sequence instantiation 1 or more method calls 0 or more assertion 1, at the end Test Specification

26. Test Specification System Under Test Class Instance Variables Test Sequences Random Variables Test Specification Wizard

27. Random Variables r1 type: int range: (0, 100) p1 type: int p2 type: int r2 type: double range: (0.0, 32.4) p3 type: double Test Specification

28. Test Specification System Under Test Class Instance Variables Test Sequences Random Variables Test Specification Wizard

29. Test Specification System Under Test Class Instance Variables Test Sequences Random Variables Assertions Test Specification Wizard

30. Assertions public class MyClass { public intmyField = 0; } MyClass m = newMyClass(); m.myField == 0 Test Specification

31. Assertions public class MyClass { public intmyField = 0; public intgetMyInt() { … } } MyClass m = newMyClass(); a.getMyInt() >= 0 Test Specification

32. Assertions public class MyClass { public intmyField = 0; public intgetMyInt() { … } } MyClass m = newMyClass(); a.myField == \old a.myField + 1 Test Specification

33. Test Specification System Under Test Class Instance Variables Test Sequences Random Variables Assertions Test Specification Wizard

34. Test Specification System Under Test Class Instance Variables Test Sequences Random Variables Assertions Test Execution Limit Test Specification Wizard

35. Test Execution Limit 5000 (5 seconds) Test Specification

36. Test Execution Limit 100 tests Test Specification

37. Test Specification System Under Test Class Instance Variables Test Sequences Random Variables Assertions Test Execution Limit Test Specification Wizard

38. Test Specification System Under Test Class Instance Variables Test Sequences Random Variables Assertions Test Execution Limit Random Number Generators Test Specification Wizard

39. Random Number Generators package umontreal.iro.lecuyer.rng RNG Pool 1 2 3 4 5 6 7 8 9 10 11 12 Test Specification

40. Random Number Generators package umontreal.iro.lecuyer.rng RNG Pool 1 2 3 4 2x 3x 5 6 7 8 9 10 11 12 Test Specification

41. Random Number Generators stream1 stream2 stream1 stream2 stream3 Thread-1 RNG collection stream1 stream2 stream1 stream2 stream3 Thread-2 RNG collection stream1 stream2 stream1 stream2 stream3 Thread-n RNG collection Test Specification

42. Random Number Generators Java.util.Random nextInt(collection_size – 1) stream1 stream2 stream1 stream2 stream3 Thread-n RNG collection

43. Random Number Generators Java.util.Random nextInt(collection_size – 1) stream1 stream2 stream1 stream2 stream3 Thread-n RNG collection nextInt(r_min, r_max) r1

44. Test Specification System Under Test Class Instance Variables Test Sequences Random Variables Assertions Test Execution Limit Random Number Generators Number of Threads Test Specification Wizard

45. Test Specification System Under Test Class Instance Variables Test Sequences Random Variables Assertions Test Execution Limit Random Number Generators Number of Threads What is done with each component? A Framework for Automated Parallel Random Unit Testing of Sequential Programs

46. 6. The Test Driver • Executes each test sequence sequentially • Each sequence is executed multiple times in parallel A Framework for Automated Parallel Random Unit Testing of Sequential Programs

47. Test Driver Parallel Java Library RNG Library A Framework for Automated Parallel Random Unit Testing of Sequential Programs

48. Test Specification System Under Test Class Instance Variables Test Sequences Random Variables Assertions Test Execution Limit Random Number Generators Number of Threads A Framework for Automated Parallel Random Unit Testing of Sequential Programs

49. Test Specification IN Test Driver OUT Test Results ?

50. Test Results pass / fail results Generated test data Total execution time Sequence execution times Thread execution times A Framework for Automated Parallel Random Unit Testing of Sequential Programs