Exploiting Weights of Test Cases to Enhance Fault Localization

1. Exploiting Weights of Test Cases to Enhance Fault Localization YihanLi,Chao Liu, Zi Yuan Beihang University, Beijing,China

2. Background • Developers more or less introduce bugs during software development • Software debugging is employed to remove bugs so as to improve quality of software • Inference on fault location • Fault repair • Verify repair • Fault localization is one of the most expensive tasks in debugging.

3. Background • Spectrum based fault localization(SFL)technique are proposed • The number of failing tests • The number of passing tests • statistical formula • Exploit how program entities are correlated with program failure

4. Background • Four coverage statistics are collected for each program entity (statement,branch,function,etc) • aef(s): the number of failed test cases that covered statement s • anf(s):the number of failed test cases that do not coverstatement s • anp(s): the number of successful test cases that cover statement s • aep(s): the number of successful test cases that do not cover statement s

5. Statistical Formula

6. Motivation • Two statements that covered by the same number of failed and passed tests can’t be distinguished. • The number of failed tests is often relatively smaller than that passed tests Only use the number of passing and failing tests

7. Our work • Propose an approach to quantify weights for different tests and incorporated into existing techniques • Compare the effectiveness of the refinement techniques with original ones

8. Approach Four weights with respect to each statement are defined • Nef(s): the total sum of weights of failed test cases that cover statement s • Nnf(s): the total sum of weights of failed test cases that do not cover statement s • Nep(s): the total sum of weights of successful test cases that cover statement s • Nnp(s): the total sum of weights of successful test cases that do not cover statement s

9. Weights of Failed Tests ForNef(s) and Nnf(s), two factors are considered: • Weight of coverage of failed test with respect to statement • Weight of status of tests with respect to statement

10. Weight of Coverage of test • For a failed test ti that cover statement s, the following weight is defined about coverage: where m is the total number of basic block in program and bri is value in basic block vector<b1i,b2i,b3i,…,bmi>. The vector record the block information for test ti, where block bj is 1 if it is covered by tiotherwise 0.

11. Weight of Coverage of test • Similar, for a failed test ti that dose not cover statement s, the following weight is defined about coverage:

12. Weight of status of tests • In extensive testing, if a few tests failed, these failed tests provide more information for error diagnosis. • On the other hand, if a few tests passed, these passed tests provide more information for error diagnosis. • In information theory, information quantity is used to measure how much information an event contains. • An even that occurs with a small probability has large information quantity.

13. Weight of status of tests • For a failed testti that cover statement s and statement s is executed by h passed tests and k failed tests, the following weight of status of test is computed: Rci(s) = -log(k/(h+ak))= log((h+ak)/k) • Similarly, if a statement is not covered by a failed test ti, thefollowing weight of status of test is computed: Rui(s) = -log((|Tf|-k)/(| Tp|-h+a*(|Tf|-k)) = log((|Tp|-h+a*(|Tf|-k))/(|Tf|-k)) Where Tp is a set of all passing tests in program and Tf is a set of all failing tests in program

14. Compution of Nef and Nnf For each failing test pi and a statement s • If statement s is covered by test pi, Wpi(s) = Eci(s) * Rci(s) • If statement s is not covered by test pi Wpi( s ) = Eui( s ) * Rui ( s ) • Nef(s) = Σ Wpi(s) = ΣEci(s) * Rci(s) • Nnf(s) = Σ Wpi( s ) = ΣEui(s) * Rui(s)

15. Weight of Passed tests • Since no definiteconclusion can be made about activation of fault, weight of every passed testis 1. • Nep(s) = |{ j | test tj is a passing test and cover statement s}| • Nnp(s) = |{j | test tj is a passing test and do not cover statement s}|

16. Revised Formula

17. Experiments Setup

18. Evaluation Metric

19. Comparison • Expense of original techniques • Expense of techniques proposed by Lee Naish • Weight of failed test is inversely proportional to the number of suspect statements executed in the test minus one • Expense of revised techniques

20. Experiments for Tarantula

21. Experiments for Jaccard

22. Experiments for Ochiai

23. Experiments for SBI

24. Mean Expense on each program

25. Conclusion • The revised techniques consistently outperform the original techniques • On average, the revised techniques can locate 20.5% more faults than original ones by examined no more than 5% of all the code.

26. Related Works • Wong et. al proposed some heuristic strategy to assign different weights for passed and failed tests respectively. • Lee Naish et. al also proposed a weighting strategy for failed tests. The rationale behind the idea is that failed tests that cover few statements provide more information than other failed tests. • Bandyopadhyay et.al proposed a proximity based weighting method for passing test cases. The weights of passing test cases are assigned based on the measure of proximity.

27. Future work • Conduct more empirical studies by using larger scale programs and multiple-faults versions • Explore other factors that may impact on weights of test cases • Apply weighting techniques to other fault localization techniques • Compare with other fault localization techniques

28. Thanks