ECE 453 – CS 447 – SE 465 Software Testing & Quality Assurance Instructor Kostas Kontogiannis

1. ECE 453 – CS 447 – SE 465 Software Testing & Quality AssuranceInstructorKostas Kontogiannis

2. Overview • Structural Testing • Introduction – General Concepts • Flow Graph Testing • Data Flow Testing • Definitions • Some Basic Data Flow Analysis Algorithms • Define/use Testing • Slice Based Testing • Guidelines and Observations • Hybrid Methods • Retrospective on Structural Testing

3. Issues on Structural Testing • An important question relates to the point where we stop to test and release the system for beta testing. Some possible answers could be: • When all faults have been removed ! • When we run out of time • When continued testing produces no new failures • When continued testing reveals no new faults • When we have exhausted the testing criteria and the test cases we have designed • When we reach a point of diminishing returns • When the mandated coverage has been achieved • When the failure intensity (failures / CPU hour) level drops below a certain point

4. Metrics for Testing Method Comparison (1) • Functional testing methods may (eventually) provide a high level of path coverage, but at the cost of high redundancy • The question is to define metrics that relate the effectiveness of a functional technique with that of a structural technique (path coverage) • To start addressing this question we first observe that functional testing methods are measured according to the number of test cases they produce, while structural testing methods are measured according to the level of the corresponding path coverage they achieve.

5. Metrics for Testing Method Comparison (2) • Let’s assume a functional testing method M, and a structural metric S that identifies s coverage elements in the unit under test • Furthermore, let’s assume that the functional testing method M produces m test cases, and these m test cases traverse n of the s structural coverage elements. We then define the following: • Coverage of methodology M with respect to metric S is defined as C(M,S) = n/s • Redundancy of methodology M with respect to metric S is defined as R(M,S) = m/s • Net redundancy of methodology M with respect to metric S is defined as NR(M,S) = m/n

6. Sample Comparison of Functional Testing Methods with respect to a Structural Metric S: Program Paths s: total number of paths n: number of paths traversed using method M m: number of test cases generated by method M M: Functional method Ref: “Software testing: A Craftsman’s Approach”, P. Jorgensen

7. Sample Comparison of a Functional Testing Method (Output BVA) with respect to Structural Metrics (DD-Path, DU-Path, Slices) S: Structural Metric s: total number of paths in structural metric S n: number of paths traversed using method M m: number of test cases generated by method M M: Functional method Ref: “Software testing: A Craftsman’s Approach”, P. Jorgensen

8. Testing Sophistication vs. Number of Test Coverage Items Number of Test Coverage Items high low Sophistication DD-Path Basis Path DU-Path Slice Ref: “Software testing: A Craftsman’s Approach”, P. Jorgensen

9. Test Coverage Items Number of Test Coverage Items 40 33 11 5 DD-Path Basis Path DU-Path Slice Ref: “Software testing: A Craftsman’s Approach”, P. Jorgensen

10. Test Case Sophistication vs. Test Case Identification Effort Effort to Identify Test Coverage Items high low Sophistication DD-Path Basis Path DU-Path Slice Ref: “Software testing: A Craftsman’s Approach”, P. Jorgensen

11. Closing Remarks • Functional testing is prone to gaps and redundancies • Structural path-coverage testing considers infeasible paths • Data flow testing considers program semantics on CFG paths • Hybrid approaches that combine functional-structural methods are useful • Structural testing techniques are best suited for unit testing