Software Testing Using Model Program

1. Software Testing Using Model Program DESIGN BY HONG NGUYEN & SHAH RAZA Dec 05, 2005

2. OverviewSoftware Testing Using Model Program • Briefly introduction of M-mp testing • What is M-mp testing? • How it work? • Why we need program base testing • The problem of software testing • Oracle test • N-Version programming • What is N-Version Programming? • Disadvantages • Advantages

3. Overview (continues) • M-mp testing • Framework for M-mp testing • Asymptotic behavior of M-mp testing • Input domain partitioning • An M-mp Experiment • Background and motivation • Selecting the application • Phases of the experiment • Conclusion

4. Introduction • What is M-mp testing? • Testing using M Model program • Abbreviated to ‘M-mp testing’ • It is alternative to software testing base on manual outcome prediction • M-mp testing is base on N-Version programming technique • Which is apply specially in the well-known software fault tolerant technique.

5. Introduction (Continues) • How it work • Model program implements selected part of functional specification of the software to be tested. • The M-mp testing strategy requires that M (M>= 1) • Model programs as well as the program under test, should be independently developed • P and M model program are then subjected to the same test data. • Difference analysis is conducted on the outputs and appropriate corrective action taken • P and the M model program jointly constituted an approximate test oracle (out come correct or not)

6. Introduction (Continues) • Why we need program base testing • For small number of test data, it may be feasible to work out the expected output manually • For thorough testing which required large test datasets manual testing could be time consuming and error prone • That is why M-mp base testing may be more feasible for those kinds of data set

7. The problem of software testing • The difficult task in software testing is to assess the correctness to the outcome of a program that is subjected to particular test input • In testing theory, the mechanism to adjudicate on whether or not an output is correct, somewhat neglected this problem is referred to as “the oracle problem. • Testing theory concern with the choice of test and testing methods, usually ignore the oracle problems.

8. The problem of software testing (continues) • Oracle Test • It means of determining whether a program passed or failed a test.

9. The problem of software testing (continues) • Oracle test (continues) • P is the program to be tested • Test strategy determines the set of test data to be used • To get the test result we compare P’s output with the oracle ‘s output • For small out put oracle manually base oracle technique is feasible • But for large industry base application manually base oracle testing is not feasible • One of alternative of manually base oracle is N- Version programming

10. N-Version Programming • N-Version is comprises N independently written version of the software • All has the same functional specification • At runtime voting base on majority agreement is used to decided the probable output • Disadvantage: • Concern in N-Version systems is the fact that correlated failures may limit the practical gain in reliability • N-Version systems should not be casually assumed to be completed reliable.

11. N-Version (continues) • Disadvantage (continues) • Is the increased development cost because at least three version being required for a voting system to work • Advantage • N-Version design appear to offer more reliability that we can gain from any other way • Even though the cost of failure is high: • It would be more cost effective to build N-Version systems rather than focusing on building ‘one good version’

12. M-mp Testing • Framework for M-mp testing

13. M-mp Testing (continues) • Framework (continues) • In M-mp testing P is primary program (the program under test) • mp1 –mpM are M so called ‘model program of P • M-mp testing treat P and mp1-mpM to be same test input • Any disagreement on the output indicates the presence of specification defects Or software faults in at least one of program • Once defects are detected and removed. The program are re-run

14. M-mp Testing (continues) • Framework (continues) • The cycle is repeat until all disagreements for a particular dataset are resolved. • Asymptotic behavior of M-mp testing • Difference between the M-mp and manual approaches to testing is the cost of outcome verification

15. M-mp Testing (continues) • Asymptotic behavior of M-mp testing (continues) • On the graph: the cost of M-mp outcome verification is high initially but it increase only slightly. • The cost growth is modeled as linear • In manual approach, the growth of the output verification cost may be expected to be much steeper. • Input domain partitioning • Divided into a standard domain and an exception domain • Exception domain may be split into an incomplete data domain and an invalid data domain. • The standard domain consists of inputs that a program can process ‘as is’

16. M-mp Testing (continues) • Input domain partitioning (continues) • The invalid domain contains the input that a program can not process it should be rejected with appropriate error message • The incomplete data domain is ‘between’ the standard and invalid data domains • It is made up of inputs that the program has to completed with default values before processing them • An M-mp Experiment • Background and motivation

17. M-mp Testing (continues) • Background and motivation (continues) • The primary program is a scheduling application, a non-interactive data processing subsystem of a large logistics management system • The program can be characterized as algorithmically complex • In practice, the testing process in the company was well defined and supported by standards and procedures but had several drawback • Designing test cases based on manual outcome prediction was not only difficult, error prone and time consuming • But consequently also unattractive and demoralizing

18. M-mp Testing (continues) • Selecting the application • The scheduling application was chosen as the primary program for following reasons • Developer was not familiar with the scheduling domain and application • The cost of developing from scratch could be assesses more authentically • The risk of correlated failures caused by common design error was reduces • scheduling application is representative of the family of operation management program • The time the experiment started, the scheduling subsystem had been in use for more than year • This seem a good opportunity to test the fault detection ability of M-mp testing, as the likelihood of finding defect was expected to be low

19. M-mp Testing (continues) • Phase of the experiment

20. M-mp Testing (continues) • Phase of the experiment • The experiment entailed a test design, a test execution and test evaluation phase as depicted • Test design phase comprised test data selection • Which includes activities related to test data generation, Model program design and model program implantation • The test execution phase includes activities relating to setting up the test environment and executing the program • The test evaluation phase includes the disagreement analysis procedures whereby disagreements between the program need to be arbitrated.

21. Conclusion • M-mp approach could test a scheduling program more adequately than manually designed test and at lower cost • M-mp testing should be judged on a case by case basic • M-mp approach will tend to be better option than the manual one when the number of tests required to achieve a particular adequacy level become large

22. Question • Finished • Please be easy on us.Thank you