Software Testing Techniques

1. Software Testing Techniques

2. Introduction • Many aspects to achieving software quality • Formal reviews (of both the software process and the various stages of development), audits, documentation, etc. • Unit testing • Integration testing • Verification • Does the module meet the specifications • Validation • Does the product meet the requirements

3. Introduction • A Critical element of the Software Quality Assurance • Represents a critical review of Specifications, Design and Coding • Destructive rather than Constructive (try to break the system) • Major objective is to find errors not to show the absence of errors (as distinct from Verification and Validation)

4. Objectives • Testing is a process of executing a program with the intenet of finding an error. • A good test case is one that has a high probability of finding an as-yet undiscovered error • A Successful test is one that uncovers an as-yet undiscovered error

5. Principles • All tests should be traceable to customer requirements • Tests should be planned long before testing begins • The Pareto principle applies to Testing • Typically, 80% of the errors come from 20% of the modules • Testing should begin ‘‘in the small’’ and progress towards “in the large” • Exhaustive Testing is not possible, but, • if time permits, conduct multiple failure mode testing • Test plans must receive independent review

6. Testability • The ease with which a computer program can be tested .

7. Characteristics for Testability • Operability • the better it works , the more efficiently it can be tested • The system has few bugs • No bugs block the execution of tests • The product evolves in functional stages

8. Characteristics for Testability • Observability • what you see is what you test • Distinct output for each input • System states and variables visible during execution • Past system states and variables are visible • All factors affecting the output are visible • Incorrect output is easily identified • Internal errors are automatically detected and reported

9. Characteristics for Testability • Controllability • the better we can control the software , the more testing can be automated • All possible outputs can be generated through some combination of input • All code is executable through some combination of input • Input and Output formats are consistent and structured • All sequences of task interaction can be generated • Tests can be conveniently specified and reproduced

10. Characteristics for Testability • Decomposability • By controlling the scope of testing , isolate problems and perform smarter retesting • The Software system is built from independent modules • Software modules can be tested independently • While this is very important, it does not obviate the need for integration testing

11. Characteristics for Testability • Simplicity • the less there is to test , the more quickly we can test it • Functional simplicity • Structural simplicity • Code simplicity

12. Characteristics for Testability • Stability • the fewer the changes , the fewer disruptions to testing • Changes are infrequent • Changes are controlled • Changes do not invalidate existing tests • The software recovers well from failures

13. Characteristics for Testability • Understandability • the more information we have , the smarter we will test • The design is well understood • Dependencies between internal, external and shared components well understood • Changes to design are well communicated • Technical documentation is instantly accessible, well-organized, specific and accurate

14. Types of Testing • White-Box Testing • Knowing the internal workings of a product, tests are conducted to ensure that “all gears mesh” i.e. internal operations are performed according to specifications and all internal components have been adequately exercised. • Black-Box Testing • Knowing the specified function that a product has been designed to perform , tests are conducted to demonstrate that each function is fully operational (note: this is still different from validation)

15. White Box Testing • Uses the control structure of the procedural design to derive test cases • Guarantees that all independent paths within a module have been exercised at least once • Exercises all loops at their boundaries and within their operational bounds • Exercises internal data structures to assure their validity - again, at their boundaries and with their operational bounds

16. Control Structure Testing • Attacks the control flow of the program • Provides us with a logical complexity measure of a procedural design • Use this measure as a guide for defining a Basis set of execution paths • Test cases derived to exercise the Basis set are guaranteed to execute every statement in the program at least once

17. Basis Path Testing • A Flow Graph created • represents the control flow of the program • each node in the graph represents one or more procedural statements • Any procedural design representation can be translated into a flow graph

18. Flow Graph Notation

19. Basis Path Testing ( contd.) • Example PDL procedure sort 1 : do while records remain read record ; 2 : if record field1 = 0 3 : then process record ; store in buffer ; increment counter ; 4 : elseif record field2 = 0 5 : then reset counter ; 6 : else process record ; store in file ; 7a: endif endif 7b: enddo 8 : end

20. Basis Path Testing ( contd.) 1 • Flow Graph 2 4 3 6 5 7a 7b 8

21. Basis Path Testing ( contd.) • Cyclomatic Complexity • Quantitative measure of the complexity of a program • is the number of independent paths in the basis set of a program • Upper bound for the number of tests that must be conducted to ensure that all statements have been executed at least once

22. Basis Path Testing ( contd.) • Cyclomatic Complexity calculation V (G) = E -N+ 2 = P + 1 = No. of regions in the graph where E = no. of edges, N = no. of nodes, and P = no. of predicate nodes • For the previous example • Independent pathspath 1 : 1 - 8 path 2 : 1 - 2 - 3 - 7b - 1 - 8 path 3 : 1 - 2 - 4 - 6 - 7a - 7b - 1 - 8 path 4 : 1- 2 - 4 - 5 - 7a - 7b - 1 - 8 • Cyclomatic complexity= 11 - 9 + 2 = 3 + 1 = 4

23. Basis Path Testing ( contd.) • Prepare test cases that will force execution of each independent path in the Basis Path set • Each test case is executed and compared to expected results

24. Example

25. Example

26. Condition Testing • Exercises all the logical conditions in a module • Types of possible errors • Boolean variable error • Boolean Parenthesis error • Boolean Operator error • Arithmetic expression error

27. Types of Condition Testing • Branch Testing • the TRUE and FALSE branches of the condition and every simple condition in it are tested • Domain Testing • for every Boolean expression of n variables , all of 2npossible tests are required

28. Data Flow Testing • Assume functions do not modify their arguments or global variables. Then define • DEF ( S ) = { X | Statement S contains a definition of X } • USE ( S ) = { X | Statement S contains a use of X } • Definition - Use chain ( DU chain ) • [ X , S , S ‘ ] , where X DEF ( S ) and X USE ( S ‘ ) and the definition of X in S is live at S ’ • Every DU chain to be covered at least once

29. Kinds of Loops

30. Loop Testing • Focus is on the validity of loop constructs • Simple loop ( n is the max. no. of allowable passes ) • Skip the loop entirely • Only one pass through the loop • Two passes • m passes , where m < n • n-1 , n , n+1 passes • Nested loop • Start at innermost loop • Conduct simple loop test for this loop • Move outwards one loop at a time

31. Loop Testing ( contd.) • Concatenated loops • Multiple simple loop tests if independent • Nested loop approach if dependent • Unstructured loops • Should be restructured into a combination of simple and nested loops

32. Black Box Testing • Focus is on the functional requirements of the software • Uncovers errors such as • Incorrect or missing functions • Interface errors • Errors in data structures • Performance errors • Initialization and Termination errors • Unlike White Box Testing , this is performed at later stages of testing

33. Graph Based Testing • Identify all objects modeled by the software • Identify the relationships that connect these objects • Create an Object-Relationship graph • node • node weights • links • link weights

34. Graph Testing ( contd.) • Example graph menu select generates Document window new file generation < 1 sec allows editing of Attributes : start dimension Background color text color is represented as contains Document text

35. Graph Test Generation • Add entry and exit nodes • For an object A, values for all objects in the transitive closure of Z must be tested for their impact on Z • Test the symmetry of all bi-directional links, e.g., “undo” • Be sure all nodes have a reflexive link. Test it for each node. • Test each relationship (the links).

36. Equivalence Partitioning • Input domain divided into classes of data from which test cases are derived • Goal is to design a single test case that uncovers classes of errors , thereby reducing the total number of test cases to be developed • Each class represents a set of valid or invalid states for input conditions

37. Equivalence Partitioning ( contd.) • Test case design is based on an evaluation of equivalence classes for an input condition • range specified , one valid and two invalid equivalence classes • requires a specific value , one valid and two invalid equivalence classes • specifies a member of a set , one valid and one invalid equivalence classes • is boolean , one valid & one invalid equivalence class

38. Equivalence Partitioning ( cont. ) • Example Automatic Banking • area code: input condition , boolean input condition , range [200,999] • prefix : input condition , range >200, no 0’s, < 1000 • suffix : input condition , value -- 4 digits • password : input condition , boolean input condition , value -- 6 char str • command : input condition , set

39. Boundary Value Analysis • Greater number of errors tend to occur at the boundaries of the input domain • Select test cases that exercise bounding values • Input condition • range , test cases are just below min and just above max • set , test cases are minimum and maximum values, if ordered • The above guidelines are also applied to output conditions • example • outputs that produce minimum and maximum values in the output range

40. Comparison Testing • Multiple copies of the software are constructed in case of critical applications • Example: Shuttle Flight Control Software • Each version is independently built from the specs by different teams • Test cases derived from other BB Testing techniques are provided as input to each version • Outputs are compared and versions validated • Not fool proof

41. Other Cases • GUI testing • See text for partial list of things to test • Client Server • Often distributed -- complicates testing • Additional emphasis on non-interference among clients • Documentation • Review & inspection (for editorial clarity) • Live test (uses the documentation in conjuction with the use of actual program) • Real-time • Beyond the scope of this course

42. Summary • Destructive rather than constructive • Main goal is to find errors not to prove the absence of errors • White Box Testing • control structure testing • Condition testing • Data flow testing • Loop testing • Black Box Testing - Functional requirements • Graph based testing • Equivalence partitioning • Boundary Value testing • Comparison testing