(OO) Regression Testing

1. (OO) Regression Testing Informally - Regression testing is the execution of a set of test cases on a program in order to ensure that its revision does not produce unintended faults, does not "regress" - that is, become less effective than it has been in the past INT-Evry (Masters IT– Soft Eng)

2. (OO) Regression Testing More Formally Given program P, its modified version P’, and a test set T that was used to previously test P, find a way to utilize T to gain sufficient confidence in the correctness of P’ INT-Evry (Masters IT– Soft Eng)

3. (OO) Regression Testing What About Changing Requirements? Test case revalidation problem Identify and modify/remove the obsolete test cases from T if specifications have changed. Question: What about adding new test cases ? INT-Evry (Masters IT– Soft Eng)

4. (OO) Regression Testing Test Suite Maintenance – What Order? • Select T’ ⊆ T, a set of test cases to execute on P’ • Regression test selection problem • Test P’ with T’, establishing correctness of P’ w.r.t. T’ • Test suite execution problem • If necessary, create T’’, a set of new functional or structural test cases for P’ • Coverage identification problem • Test P’ with T’’, establishing correctness of P’ w.r.t. T’’ • Test suite execution problem • Create T’’’, a new test suite and test execution profile for P’, from T, T’, and T’’. • Test suite maintenance problem INT-Evry (Masters IT– Soft Eng)

5. (OO) Regression Testing Test Suite Maintenance – Discarding baseline test cases • Broken test cases: fails to run • Obselete test cases: no longer match requirements • Uncontrollable test cases: may not be repeatable (see later) • Redundant test cases: where 2 or more test cases check the same thing (this is quite a complex concept) INT-Evry (Masters IT– Soft Eng)

6. (OO) Regression Testing Test Suite Maintenance – Reduction Inclusiveness – percentage of baseline tests that may show regression faults (safe => 100% inclusiveness). This requires formal dependency analysis and is a whitebox technique sometimes called retest within firewall. Precision – percentage of baseline tests in a reduced test suite that cannot reveal regression faults and that are not selected for the reduced test suite. Note: one can argue that no test suite can be both safe and 100 precise as there is no way to decide exactly which tests will pass or fail for the changed system Note: passing a “safe” regression suite does not guarantee absence of regression bugs – it only guarantees that all baseline test cases that could reveal a regression bug have been exercised INT-Evry (Masters IT– Soft Eng)

7. (OO) Regression Testing Test Suite Maintenance – Unsafe Reduction None of the following are safe because they are not based on dependency analysis: Systematic Sampling Random Sampling Coverage-based filtering By profiling (use) By Intuition INT-Evry (Masters IT– Soft Eng)

8. (OO) Regression Testing Can We Automate The Process? Resulti = Resulti’, forall i INT-Evry (Masters IT– Soft Eng)

9. (OO) Regression Testing Partial regression testing? If the time (and money …) interval allocated to regression testing is limited, only a fraction of the test suite can be executed, with the risk of missing test cases that are able to reveal defect not yet discovered INT-Evry (Masters IT– Soft Eng)

10. (OO) Regression Testing Partial regression testing: techniques [Regression test selection:] the cost of regression testing is reduced by selecting a subset of the existing test suite based on information about the program, modified version and test suite. [Test suite minimization:] the test suite is reduced to a minimal subset that maintains the same coverage as the original test suite with respect to a given coverage criterion. [Test case prioritization:] test cases are ordered so that those with the highest priority are executed earlier, for example with the objective of achieving code coverage at the fastest possible rate, or of exercising the modules according to their propensity to fail. INT-Evry (Masters IT– Soft Eng)

11. (OO) Regression Testing RT and Software Maintenance Corrective Maintenance – changes made to debug a system after a failure is observed Adaptive Maintenance – changes made to achieve continuing compatibility with the target environment Perfective Maintenance – changes designed/made to improve or add capabilities Preventive Maintenance – changes made to improve quality of software RT is required for each of these: to reveal side effects and bad fixes INT-Evry (Masters IT– Soft Eng)

12. (OO) Regression Testing Bad Fixes Bad fix injection rates are normally between 2 and 20% Iterative approach (with OO) demands much more care with bad fixes – as it focuses on changing code RT must be done in parallel with new code development – otherwise regression faults will go unnoticed INT-Evry (Masters IT– Soft Eng)

13. (OO) Regression Testing OO testing: re-use techniques – inheritance problem Getting a new subclass working without exercising the superclass is usually easy It is tempting to assume that inherited features just work INT-Evry (Masters IT– Soft Eng)

14. (OO) Regression Testing OO testing: component based development An adequate regression test suite that can be rerun by a consumer provides a convincing demonstration that a component works out of the box. RT does not reduce the need to develop and execute tests of new and changed capabilities OO technologies offer strong support for reuse but they cannot prevent regression bugs. INT-Evry (Masters IT– Soft Eng)

15. (OO) Regression Testing Regression Faults When a stable baseline system B and a delta component D pass individually adequate tests, but fail when used together D can cause some component in B to fail only if there is some dependency between them. Dependencies occur for many reasons and so regression faults can occur in many ways. Question: how many can you think of? There is no practical means to develop a test suite guaranteed to reveal all regression faults INT-Evry (Masters IT– Soft Eng)

16. (OO) Regression Testing Regression Faults: typical cases D has a side effect on B: B fails because a new action of D is inconsistent with B’s requirements, assumptions, or contract with respect to D INT-Evry (Masters IT– Soft Eng)

17. (OO) Regression Testing Regression Faults: typical cases D is a client of B: D sends a message that violates B’s invariant/precondition. B is not defensive and accepts the incoming message, leading to failure INT-Evry (Masters IT– Soft Eng)

18. (OO) Regression Testing Regression Faults: typical cases D is a server of B: B sends a message to D. D’s postconsitions have changed (or contain bugs). D returns a value that causes a violation of B’s invariant . B fails or returns an invalid value to another baseline component C. C fails INT-Evry (Masters IT– Soft Eng)

19. (OO) Regression Testing Regression Faults: typical cases D is a revised superclass: B is a client of E, a subclass of D. The interface of D has not changed but its behaviour has: and so the effective behaviour of E is different. This behaviour may introduce timing or nondeterministic problems which can lead to failure of B. INT-Evry (Masters IT– Soft Eng)

20. (OO) Regression Testing Regression Faults: typical cases D is a new subclass of a polymorphic class hierarchy that is not LSP (Liskov substitution principle ) compliant: B is a client of D and expects the new subtype to be compatible. D does not comply leading to an inconsistent state and so when B attempts to use this object it fails. INT-Evry (Masters IT– Soft Eng)

21. (OO) Regression Testing Regression Faults: typical cases • An undesirable feature interaction occurs: • B’s functionality extended by change to/addition of D1 • B’s functionality extended by change to/addition of D2 • B+D1 passes regression tests • B+D2 passes regression tests • But B+D1+D2 fails regression tests Note: this is a very abstract view of feature composition (+) INT-Evry (Masters IT– Soft Eng)

22. (OO) Regression Testing Regression and Unit Testing: test frameworks • A test framework provides reusable test functionality which: • Is easier to use (e.g. don’t have to write the same code for each class) • Is standardized and reusable • Provides a base for regression tests A unit testing framework is unit & regression testing on steroids INT-Evry (Masters IT– Soft Eng)

23. (OO) Regression Testing Regression and Unit Testing: unit test frameworks Each class must be tested when it is developed Each class needs a regression test Regression tests need to have standard interfaces Thus, we can build the regression test when building the class and have a better, more stable product for less work INT-Evry (Masters IT– Soft Eng)

24. (OO) Regression Testing Automating with unit test frameworks • The real power of regression tests happens when they are automated: this requires they report pass/fail results in a standardized way • Can set up jobs to: • Clean & check out latest build tree • Run tests • Put results on a web page & send mail (if tests fail) • JUnit & ant have code to do all of this (You have already seen these?) INT-Evry (Masters IT– Soft Eng)

25. (OO) Regression Testing Automating with unit test frameworks • Effective automated RG requires: • Version control • Modular structure • Smart Means of comparing baseline and delta results • Shuffling • Assertions for built-in test INT-Evry (Masters IT– Soft Eng)

26. (OO) Regression Testing Automating with unit test frameworks • Environmental factors can lead to less-than-identical test configuration: • Different OSs • Different content in persistant storage • SUT uses nondeterministic objects • SUT uses non-repeatable pseudorandom generation • SUT is sensitive to timing differences • SUT is sensitive to non-controllable system input • SUT uses time sensitive code INT-Evry (Masters IT– Soft Eng)

27. (OO) Regression Testing Lets learn about regression testing by looking at a practical example: • A simple array of integer pairs is required that can be initialised (with a specified size) and can be randomized with random values within a certain range. It must also provide a method to report if the values are strictly increasing in both dimensions. • Check out the Source Code at the web site • IntPair.java, • Compare.java, • IntPairArray.java, • TestIntPairArray.java • Is the following test suitable for regression testing? How can we improve things? INT-Evry (Masters IT– Soft Eng)

28. (OO) Regression Testing Lets learn about regression testing by looking at a practical example: Now, experiment by extending the system and deliberately inserting typical regression faults (see earlier) How well does the regression test suite do in finding faults? How should we maintain the test suite? INT-Evry (Masters IT– Soft Eng)