A General Discussion on Functional (Black-box) Testing

1. A General Discussion on Functional (Black-box) Testing • What are some of the concerns of testers ? • Have we got enough time to test (effort & schedule)? • Have we tested all the areas (coverage & efficiency)? • Are we discovering the defects (effectiveness)?

2. Effort & (Schedule) • Effort≡amount of timerequired to i) develop, ii) execute, and iii) analyze the resultof each test case: • Decision table (highest) • May be less number of test cases • But requires deeper analysis of conditions (which come from looking at the various combinations and dependencies) • Completeness • Avoiding redundancy • Partitioned Equivalence class (middle) • Picking a representative from each of the partitioned class • Avoiding redundancy • Difficulty is in coming up with theequivalence relation • Complete and disjoint • Boundary value (lowest) • Easy to identify the boundaries • Generating test cases are easier • May expand into many test cases (especially for robust worst-case)

3. Testing Efficiency • Efficiency≡ amount of “coverage”provided by each test case. • Coverage of what? • Possible combinations of inputs and expected outputs • Possible logical conditions • What about redundancy? • With efficiency, we are striving to i) maximize coverage (completeness) and ii) reduce redundancy

4. Efficiency • Decision table (high) • Start with listing all n conditions and 2n combinations • Completeness • Analyze the irrelevant and reduce the redundant ones • Avoiding redundancy • Partitioned Equivalence class (high to medium) • Define an equivalence relationship • Completeness • Pick a representative test case from each disjoint class • Minimizing redundancy • “Strong Robust” Equivalence (e.g.illegal cases) can generate lots of test cases • Boundary value (medium to low) • Define the boundaries of valid and invalid inputs (a specialcase of equivalence relation?) • Completeness • Pick data from within , on , and outside of the boundaries • Reducing redundancy as much as possible

5. Effectiveness • Effectiveness≡number of defects discovered per test case executed. • It is difficult to gauge this except from past experience: • People tend to make mistakes with boundaries • Boundary value testing • Equivalence class testing • People tend to make mistakes with combinations of inputs (logic) where there are dependencies • Decision table testing

6. Testing (Effort) = Time to develop each test case = # of Test cases sophistication Boundary Value Equivalence Class Decision Table

7. Need for Metrics and History • In order for us to better understand and manage testing, we need to better define the attributes: • Effort • Efficiency • Effectiveness • We need to keep records (time; people-time; defects discovery - type & rate) of test cases: • Development of Test Cases • Execution of Test Cases • Test Result Analysis

8. Insurance Premium Problem(~ page 123 in your text – 3rd edition) • Premium = Base Rate * age multiplier – SafeDriver reduction • where Base Rate = $500 and premium is computed according to table below, assuming the driver has less than 12 points. Inputs to this system : 1) age and 2) points

9. Let’s consider 3 types of Functional Testing (Black Box) Approaches – varies a bit from p 123 of text • A easy way is to start with boundary value, but dependent: • age : 16 – 99 and points 0 – 12. • {16, 17, 45, 98, 99} • (0, 1, 5, 11, 12} • This will create 5 x 5 = 25 test cases ---- but, perhaps, not very satisfactory! • Continuing with boundary value, but looking at each range of ages and each of the potential points. • There are 5 age ranges and 5 boundary test cases for each range ->25 ages • There are 0 – 12 or 13 possible penalty points (if we use all 13 points). • Considering these are dependent or “worst case” gives us 25 x 13 = 325 cases • If we partition by age, then there are 5 partitions given, and partition the points by {0,1}; {2,3}; {4,5}; {6,7}; (8,9,10,11,12}. • We have 5 x 5 = 25 equivalent partitions --- potentially 25 test cases • We can also look to decision table approach: • Consider criterion: {16-24}; {25-34}; {35-44}; {45-59}; {60-99} • For each of the above criterion consider pts criterion (0-1); (2-3);(4-5);(6-7); (8-12) • Consider the action of rules of multiplying by age multiplier and the redcution for each of the possible criterion. ------ the possible criteria values are 210 ---- huge! -------- but many are “impossible” cases ---- so we can reduce it down.

10. Iterate through the Test Techniques • Page 123 of text on insurance premium • Start with a very simple boundary value testing; • End points of ages [16, 100) • End points of violation points [0, 12) • For more completeness of coverage, consider the intervals of • Age • Points • Think of intervals as equivalence classes and reduce test cases • Because there are conditions that involve dependencies of inputs, use decision tables and further reduce excessive test cases. • (e.g. combined conditions of penalty points beyond 1 and age interval of 16 to 25 do not need to be included since that combination is not allowed - - - - a saving of four “irrelevant” test cases – see page 130 of text.) Note the boundaries 1. Iterate from the simplest to the more difficult test case development method. 2. Review all the developed test cases and choose the ones you want to keep