Automated Testing of Software Components Based on Algebraic Specifications -- Method, Tool And Experiments

1. Algebriac Testing Automated Testing of Software Components Based on Algebraic Specifications-- Method, Tool And Experiments Hong Zhu Dept. of Computing and Electronics, Oxford Brookes University, Oxford, OX33 1HX, UK Email: hzhu@brookes.ac.uk

2. Outline • Background • Motivation • Related works • Overview of the approach • Specification language CASOCC • Testing tool CASCAT • Empirical evaluation • Conclusion and future work Algebriac Testing

3. Challenges in testing components • Components often have no user interface • Developers spend as much time in writing test harness • excessive overhead, inadequacy of testing, low effectiveness • Components are usually delivered as executable code • without the source code and design information • White-box testing, model-driven testing methods not applicable • contains no instrumentation • Internal behaviour observation and test adequacy measurement are virtually impossible • Existing approaches to the problems • Self-testing, e.g. (Beydeda, 2006): yet to be adopted by the industry • Specification-based testing: design-by-contract, FSM, etc. Algebriac Testing

4. Algebraic specification Spec NAT Sorts: nat; Operators: zero: -> nat; succ: nat -> nat; Axioms: zero  succ(x); succ(x) = succ(y) => x=y; End NAT • A signature: • a set of sorts and • a set of operators on the sorts • A set of axioms: • in the form of conditional equations Algebraic specification (AS) emerged in the 1970s. In the past three decades, it has developed into a mature formal method. Algebriac Testing

5. Basic idea of algebraic testing By substituting constants into variables, we can generate test cases S: stack, n: integer, S.push(n).height() = S.height()+1 Checking the equivalence between the values of the left and right hand sides is to check the correctness of test results A ground term corresponds to a sequence of procedure/ method/ operation calls Algebriac Testing

6. Related works Algebriac Testing

7. Overview of the proposed approach • Sorts to represent all types of software entities: • ADT, Class, Component • Test case generation: • Composition of observation contexts and axioms with ground normal forms substituted into non-primitive variables and random values for primitive variables • Test oracle: • Direct checking since test cases are checkable Algebriac Testing

8. Spec Stack observable F; import Int, String; operations creator create: String->Stack; constructor push: Stack,Int ->Stack; transformer pop: Stack->Stack; observer getId: Stack->String; top: Stack->Int; height: Stack->Int; vars S: Stack; n: Int; x: String; axioms 1: create(x).getId() = x; 2: findByPrimaryKey(x).getId() = x; 3: create(x).height() = 0; 4: S.push(n) = S; if S.height() = 10; 5: S.pop() = S; if S.height() = 0; 6: S.push(n).pop() = S; if S.height() < 10; 7: S.push(n).top() = n; if S.height() < 10; 8: S.push(n).height() =S.height()+1; if S.height() < 10; 9: S.pop().height() = S.height()-1; if S.height() >0; end CASOCC specification language Algebriac Testing

9. Behavioural semantics and observable sorts Definition 1. (Observable sort) In an AS <S, E>, a sort s is an observable sort implies that there is an operation _ == _ : ssBool such that for all ground terms t and t’ of sort s, E|-( (t == t’) = true) E |- ( t=t’ ). An algebra A (i.e. a software entity) is a correct implementation of an observable sort s if for all ground terms t and t’ of sort s, A |= (t=t’) A|= ( (t == t’) = true)  Pre-defined sorts of Java primitive classes and data types are observable. Algebriac Testing

10. Well founded formal specifications Let U be a set of specification units in CASOCC and S be a set of sorts. For each sort sS, there is a unit UsU that specifies the software entity corresponding to sort s. Let  be the importation relation on S. Definition 2. (Well founded specifications) A sort sS is well founded if s is observable, or for all s’ in the import list of Us, s’ is an observable sort, or s’ is well founded. A specification U is well founded if and only if the importation relation is a pre-order on the set S of sorts, and all sorts sS are well founded.  Algebriac Testing

11. Well-structured formal specifications Definition 3. (Well structured specifications) A specification U in CASOCC is well structured if it satisfies the following conditions. (1) It is well founded; (2) For every user defined unit Us U, (a) there is at least one observer in Us; (b) for every axiom E in Us, if the condition contains an equation t=t’, we must have, where s’ is the sort of terms t and t’.  A practice implication: for all sorts there are finite lengths of observable contexts. Algebriac Testing

12. Observation context Definition 1. (Observation context) A context of a sort c is a term C with one occurrence of a special variable of sort c. The value of a term t of sort c in the context of C, written as C[t], is the term obtained by substituting t into the special variable . An observation contextoc of sort c is a context of sort c and the sort of the term oc is. To be consistent on notations, we write _.oc: cs to denote an observation context oc. An observation context is primitive if s is an observable sort. In such cases, we also say that the observation context is observable and call the context observable context for short.  Algebriac Testing

13. Form of observation context • The general form of an observation context oc: _.f1(...).f2(...).....fk(...).obs(...) where • f1, ..., fk are transformers of sort sc, • obs is an observer of sort c, • f1(...), ..., fk (...) are ground terms. • A sequence of observation contexts oc1, oc2, …, ocn , where _.oc1: cs1, _.oci: si-1si,i =2,…,n, can be composed into an observation context _.oc1.oc2. ….ocn. • Example: • _.pop().pop().height() Algebriac Testing

14. Checkable test cases written T1=T2, [if C] • Test cases: • a test case is a triple <T1, T2, C>, where • T1 and T2 are ground terms • C (optional) is a ground term of Boolean sort. • It means that values of T1 and T2 should be equivalent if C evaluates to True. Definition 2. (Checkable test cases) A test case T1=T2, [if C]is directly checkable (or simply checkable), if and only if (a) the sort of terms T1 and T2 is observable, and (b) the sort of equations in C is observable, if any.  Algebriac Testing

15. Test case generation algorithm (Skeleton) Input: Spec s: CASOCC specification unit of the main sort; Sigs s1, s2, …, sk: The signature of imported sorts; TC: A subset of axioms in s (* the axioms to be tested *); vc: Integer (*complexity upper bound of variables*); oc: Integer (*complexity upper bound of observation contexts*) ; rc: Integer (* the number of random values to be assigned to variables of primitive sorts*) Begin Step 1: Initialisation Step 2: Generate normal form terms for non-primitive variables Step 3: Generate random values for primitive variables Step 4: Substitute normal forms into axioms Step 5: Substitute random values into test cases Step 6: Compose test case with observation context Step 7: Output test set End Algebriac Testing

16. Properties of the test case generation algorithm Theorem 1. The test case generation algorithm will always terminate if the specification is well founded.  Theorem 2. The test cases generated are checkable, i.e. for all test cases <t1=t2; if c> generated by the algorithm, t1, t2 and c are of primitive or observable sorts.  Theorem 3. The test cases are valid. That is, if the specification is well-structured and the observable sorts satisfy the constraints in Definition 1, we have the following properties. (a) The program correctly implements the specification with respect to the behavioural semantics of algebraic specifications implies that the evaluation of t1 and t2 using the program give equivalent results provided that c is evaluated to be true. (b) If the evaluation of t1 and t2 gives non-equivalent values in an implementation when c is evaluated to true, then there are faults in the program.  Algebriac Testing

17. Component Spec in CASOCC J2EE Component Deployed on JBoss Platform Test Report CASOCC Spec Parser Test Driver Test Result Evaluator Test Case Generator Test Cases CASCAT Tool Testing tool CASCAT Algebriac Testing

18. Algebriac Testing

19. Experiment 1: Evaluation of effectiveness • The experiment process • Selection of subject components: • from well established public sources. • Development of formal specification: • based on the document and source code. • Testcase generation: • automatically by the CASCAT tool from the specification. • Validation of formal specification. • The subject component is checked against its formal specification by executing the components on the test cases using the CASCAT tool. • Fault injection: used MuJava. • Eliminate equivalent mutants: manually examined • Test execution. • A mutant is classified as fault detected if at least on of the axioms of the component is violated or the execution is terminated abnormally. Algebriac Testing

20. The subjects: 1) Single Component Subjects Algebriac Testing

21. The subjects: 1)Multiple Components Subjects Algebriac Testing

22. Results of the experiment Algebriac Testing

23. Main findings (1) • The fault detecting ability is not sensitive to the scale of the subject under test. (Correlation coefficient =0.20) Algebriac Testing

24. Main findings (2) • The fault detecting ability decreases only slightly when testing multiple component subjects. Algebriac Testing

25. Main findings (3) • The method consistently detects significantly more faults in session beans than in BMP entities beans despite that entity beans are usually much less complex than session beans. This statement is supported by T-Test. Algebriac Testing

26. Main findings (4) • The development of axioms was less difficult than we expected. • There is a simple pattern of axioms for entity beans despite their differences in semantics. Algebriac Testing

27. Is algebraic testing practical? • Cost of algebraic testing: • Writing algebraic specification • Deploy the component to component platform, such as JBoss • Generation of test cases (automated by tool) • Review of test report (checking correctness is done automatically) • How expensive is writing algebraic specifications? • Is writing algebraic specification learnable? • What skill and knowledge are required to write algebraic specifications? Algebriac Testing

28. How to write algebraic specifications (1) • the description of the signature • the identification of the operations, e.g. • The signature of the operations can be derived from the type definitions of the methods given in the source code. • the classification of operations • Creators: create instances of the software entity and/or initialise the entity. They must have no parameters of the main sort, but result in the main sort. • Constructors: construct the data structure by adding more elements to the data. A constructor must have a parameter of the main sort and results in the main sort. It may occur in the normal forms if the axioms are used as term rewriting rules. • Transformers: manipulate the data structure without adding more data. Similar to constructors, a transformer must have the main sort as its parameter and results in the main sort. However, it cannot occur in any normal forms. • Observers: enable the internal states or data in the software entity to be observed from the outside. Observers must have a parameter of the main sort but result in an imported sort. Algebriac Testing

29. How to write algebraic specifications (2) • the determination of the axioms • For each setter setX(v) (set the value of attribute X to v), s,v. (s.setX(v).getX = v), Ifpre-condsetX(v) s,v. (s.setX(v).getY = s.getY), where X Y. • For each getter getX (get the value of attribute X) s,v. (s.[getX] = s), • For each creator C(x1,x2,…, xn) • x1,…, xn. . C(x1,…, xn).getXi = xi, If pre-cond C(x1,…, xn) • For each constructor and transformer F(x), s,x. (s.F(x).getX) = f(x, s.getX), if pre-condF(x), • For each operation P(x) that involves more than one parts A and B: s,x,y. (s. [A.P (x)].B = s.B.Q(y)), if pre-condA.P(x) Algebriac Testing

30. Experiment 2: Cost of writing algebraic spec • Subjects: • Students of computer science (35, year 3) • Mathematics course: Advanced University Mathematics Part A and Part B and Discrete Mathematics. • Programming courses: C++ Programming, Java Programming and Data Structure • No exposure to formal methods Table 1. Statistic Data of Student Capability Algebriac Testing

31. Distribution of capabilities Algebriac Testing

32. Algebriac Testing

33. Process of the experiment Lesson 0: • Introduction to formal methods. Lesson 1: • Introduced to algebraic specification and the CASOCC specification language. • An example formal specification of stacks • A brief introduction to software component for the first class test • The first class test (individual and independent) Lesson 2: • Sample answer to the class test question 1 • A brief introduction to software component for the second class test • The second class test (also individual and independent) Lesson 3: • Sample answer to the second class test question • A brief introduction to software component for the third class test • The third class test (also individual and independent) Lesson 4: • Sample answer to the previous class test • A brief introduction to software component for the final class test • The final class test Algebriac Testing

34. Marking scheme of class tests • Correctness of the answer: 50%. • It is assess according to the correctness of the signature and axioms in the student’s work. • Minor syntax errors that can be detected by CASCAD tool is deduced by 20% • Incorrect axioms were given no marks. • Completeness of the axiom system: 50%. • It is assessed according to the coverage of the operations by the axioms. • The coverage of each operation was given the equal number of marks. Algebriac Testing

35. Recording times • The time that each student took to complete the class test was recorded in the experiment. • The students were given no limit on the time to complete the class tests. • The students were asked to hand in their work as soon as possible. • The students were briefed about the function and the interface of the component before started to work on the class test question. The time taken to write the algebraic specification excludes the time to understand the components. Algebriac Testing

36. Components Used in Class Tests [1] 周长发, Java数值计算算法编程，电子工业出版社，2007. [2] Bodoff, S. et al. 2004. The J2EE Tutorial, 2nd Edt., Pearson 2004. Algebriac Testing

37. Main Findings 1 Is writing algebraic specifications learnable? Algebriac Testing

38. Changes in the distributions of scores Algebriac Testing

39. Distributions of Grades in Class Tests The students’ learning experience is not hard. They attained the knowledge and skill of algebraic specification in just a few lessons. Algebriac Testing

40. Main Findings 2 How expensive to write an algebraic specification for a software component? Algebriac Testing

41. Changes in the distributions of times On average a student took about half hour to complete the writing of an algebraic specification for a typical software component. Algebriac Testing

42. Main Findings 3 Does writing algebraic specification need good mathematical skills? Correlation Coefficients Algebriac Testing

43. Cluster analysis We divide the students into the following four groups and calculated their scores in class tests. • P>M: More capable of programming than mathematics. • P<M: More capable in mathematics than programming • P~M High: Equally capable of programming and mathematics • P~M Lower: Equally incapable of programming and mathematics Algebriac Testing

44. The students’ performances in class tests are more closely related to their programming capability than to mathematics knowledge and skills. • Notes: • the link between students’ performance in class tests and programming capability should be interpreted as their capability of learning algebraic specification rather than their final attainment. • the link is not strong since the absolute values of the correlation coefficients are in the range from 0.41 to 0.52. Algebriac Testing

45. Main Findings 4 Is writing algebraic formal specifications a job only for the most capable? Algebriac Testing

46. Writing algebraic specification must be the job of the most capable programmers. However, there is a potential bias. The average scores and average times contain the results of the first and second class tests. Thus, they do not reflect the situation after the students completed their training. Algebriac Testing

47. Algebriac Testing

48. (a) Relationship between final class test score and programming capability y = -0.0002x2 - 0.0148x + 98.405 (3) (b) Relationship between final class test times and programming capability y = 0.0017x2 - 0.4558x + 56.043 (4) After taking three lessons and class tests, the students are capable of writing algebraic specifications of almost equal quality, but the most capable ones took slightly less time. Writing algebraic specifications can be a job for any well trained software developer rather than just for the few most capable ones. Algebriac Testing

49. General conclusions of experiment 2 • Conclusion 1 (Learnability): Writing algebraic specification is learnable for ordinary software developers. • Conclusion 2 (Independence of mathematical skills) The knowledge and skill of programming is more important than mathematics to writing algebraic specifications of software components. • Conclusion 3 (Cost efficiency): Writing algebraic specification can be as cost efficient as programming in high level programming languages. • Conclusion 4 (Equality in performance): Writing algebraic specification can be a skill of every well trained software developer. Although their efficiency in writing algebraic specifications depends on their capabilities, there should be no significantly different from each other on the quality. Algebriac Testing

50. Limitations of the conclusions 1 The conclusions are only applicable to writing algebraic formal specifications. • They do not necessarily imply that writing formal specifications in other formalisms has the same properties. • Further research: to investigate whether the same claim can be made to other formalisms such as Z, Petri-nets, process algebras like CSP, CCS and -calculus, and labelled transition system in general. A notable advantage of algebraic specification is that the syntax and semantics of axioms are simple and easy to understand. They use little mathematics notations. Algebriac Testing