Software Fault Interactions and Implications for Software Testing

1. Software Fault Interactions and Implications for Software Testing Presented By: Sai Praneeth Purighella.

2. Introduction • For example, 20 inputs, each having 10 possible values, yield a total of 1020 combinations of settings. • Only a few hundred test cases can be built and executed under most budgets • Empirical research into quality and reliability suggests that software failures in a variety of domains were caused by combinations of relatively few parameters. • If ‘n’ be the number of parameters involved, then testing all n-tuples of parameters is effectively equivalent to exhaustive testing (pseudo exhaustive). • The number of conditions required to trigger a failure is referred as the failure-triggering fault interaction (FTFI) number.

3. Related Work • Nair et al.’s case study of combinatorial testing for a small subsystem of a screen-based administrative database. • Wallace and Kuhn’s review of medical device recall data gathered by FDA. • Smith et al.’s investigation on the pairwise testing of RAX software on NASA’s Deep Space 1 mission.

4. Empirical Analysis • Analysis of 329 error reports from development and integration of a large distributed system at NASA Goddard Space Flight Center. • Results of analysis are shown in last column of Table 1 and in Table 2. • Distribution of failure-triggering conditions appears to follow a power law. (See figure 1, last four columns of Table 1).

7. Figure 1

8. Implications of Testing • Consider the example with 20 inputs, each with 10 possible values. • Exhaustive testing requires 1020 test cases. • But empirical studies show that most failures are triggered by a single erroneous parameter. However, all could be triggered by fewer than 6 parameters.

9. Background Calculations • Consider the problem of n-tuples of k-parameters, each of which has ‘v’ possible values. • The number of n-tuples drawn from k- parameters is

10. Implications (Continued) • Since each parameter has ‘v’ values, total number of test cases would be C(k,n).vn. • Let us see at the example previously considered. • For a given number N, of test cases, and a specifies level n-tuple, how many values can or must be covered can becalculated using the expression: vn<=N, so n log v <= log N.

11. Conclusions • If all errors in a particular class of software are triggered by finite combinations of ‘n’ values or less, then testing all combinations of n or fewer values would provide a form of “pseudoexhaustive testing”. • Appropriate levels of ‘n’ appear to be 4<=n<=6 when considering pseudoexhaustive testing, according to dependability requirements. • This approach might not be good effective real-time or other software that depends on testing event sequences, but may be applicable to subsystems within real-time software. • Many more empirical studies of other classes of software are needed to evaluate the applicability of combinatorial testing for other classes of systems.

12. Thank You One and All!