1 / 97

Model based Software Testing Test Assessment and Enhancement

Model based Software Testing Test Assessment and Enhancement. Aditya P. Mathur Purdue University Fall 2005. Last update: August 18, 2005. Learning Objectives. To understand the relevance and importance of test assessment. To learn the fundamental principle underlying test assessment.

jana
Télécharger la présentation

Model based Software Testing Test Assessment and Enhancement

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Model based Software TestingTest Assessment and Enhancement Aditya P. Mathur Purdue University Fall 2005 Last update: August 18, 2005

  2. Learning Objectives • To understand the relevance and importance of test assessment. • To learn the fundamental principle underlying test assessment. • To learn various methods and tools for test assessment. • To understand the relative strengths/weaknesses of test assessment methods. • To learn how to improve tests based on a test assessment procedure. Software Testing and Reliability Aditya P. Mathur 2002

  3. What is Test Assessment? • Once a test set T, a collection of test inputs, has been developed, we ask: How good is T? • It is the measurement of the goodness of T which is known as test assessment. • Test assessment is carried out based on one or more criteria. Software Testing and Reliability Aditya P. Mathur 2002

  4. Test Assessment (contd.) • These criteria are known as test adequacy criteria. • Test assessment is also known as test adequacy assessment. Software Testing and Reliability Aditya P. Mathur 2002

  5. Test assessment (contd.) • Test assessment provides the following information: • A metric, also known as the adequacy score or coverage, usually between 0 and 1. • A list of all the weaknesses found in T, which when removed, will raise the score to 1. • The weaknesses depend on the criteria used for assessment. Software Testing and Reliability Aditya P. Mathur 2002

  6. Test assessment (contd.) • Once the coverage has been computed, and the weaknesses identified, one can improve T. • Improvement of T is done by examining one or more weaknesses and constructing new test requirements designed to overcome the weakness(es). • The new test requirements lead to new test specifications and to further testing of the program. Software Testing and Reliability Aditya P. Mathur 2002

  7. Test Assessment (contd.) • This is continued until all weaknesses are overcome, i.e. the adequacy criterion is satisfied (coverage=1). • In some instances it may not be possible to satisfy the adequacy criteria for one or more of the following reasons: • Lack of sufficient manpower • Weaknesses that cannot be removed because they are infeasible. Software Testing and Reliability Aditya P. Mathur 2002

  8. Test Assessment (contd.) • The cost of removing the weaknesses is not justified. • While improving T by removing its weaknesses, one usually tests the program more thoroughly than it has been tested so far. • This additional testing is likely to result in the discovery of remaining errors. Software Testing and Reliability Aditya P. Mathur 2002

  9. Test Assessment (contd.) • Test assessment and improvement is applicable throughout the testing process and during all stages of software development. • Hence we say that test assessment and improvement helps in the improvement of software reliability. Software Testing and Reliability Aditya P. Mathur 2002

  10. 0 Develop T Select an adequacy criterion C. 1 2 Measure adequacy of T w.r.t. C. Yes 3 Is T adequate? Yes No 4 Improve T More testing is warranted ? 5 No 6 Done Test Assessment Procedure Software Testing and Reliability Aditya P. Mathur 2002

  11. Principle Underlying Test Assessment • There is a uniform principle that underlies test assessment throughout the testing process. • This principle is referred to as the coverage principle. • It has come about as a result of intensive research at Purdue and other research groups in software testing. Software Testing and Reliability Aditya P. Mathur 2002

  12. The Coverage Principle • To formulate and understand the coverage principle, we need to understand: • coverage domains • coverage elements • A coverage domain is a finite domain, related to the program under test, that we want to cover. Coverage elements are the individual elements of this domain Software Testing and Reliability Aditya P. Mathur 2002

  13. Requirements Classes Functions Interface mutations Exceptions The Coverage Principle (contd.) Coverage Domains Coverage Elements Software Testing and Reliability Aditya P. Mathur 2002

  14. The Coverage Principle (contd.) • Measuring test adequacy and improving a test set against a sequence of well defined, increasingly strong, coverage domains leads to improved confidence in the reliability of the system under test. Software Testing and Reliability Aditya P. Mathur 2002

  15. The Coverage Principle (contd.) • Note the following properties of a coverage domain: • It is related to the program under test. • It is finite. • It may come from program requirements, related to the inputs and outputs. Software Testing and Reliability Aditya P. Mathur 2002

  16. The Coverage Principle (contd.) • It may come from program code. Can you think of a coverage domain that comes from the program code? • It aids in measuring test adequacy as well as the progress made in testing. How? Software Testing and Reliability Aditya P. Mathur 2002

  17. The Coverage Principle (contd.) • Example: • It is required to write a program that takes in the name of a person as a string and searches for the name in a file of names. The program must output the record ID which matches the given name. In case of no match a -1 is returned. What coverage domains can be identified from this requirement? Software Testing and Reliability Aditya P. Mathur 2002

  18. The Coverage Principle (contd.) • As we learned earlier, improving coverage improves our confidence in the correct functioning of the program under test. • Given a program P and a test T suppose that T is adequate w.r.t. a coverage criterion C. • Does this mean that P is error free? Obviously……??? Software Testing and Reliability Aditya P. Mathur 2002

  19. Test Effort • There are several measures of test effort. • One measure is the size of T. By this measure a test set with a larger number of test cases corresponds to higher effort than one with a lesser number of test cases. Software Testing and Reliability Aditya P. Mathur 2002

  20. Error Detection Effectiveness • Each coverage criterion has its error detection ability. This is also known as the error detection effectiveness or simply effectiveness of the criterion. • One measure of the effectiveness of criterion C is the fraction of faults guaranteed to be revealed by a test T that satisfies C. Software Testing and Reliability Aditya P. Mathur 2002

  21. Effectiveness (contd.) • Another measure is the probability that at least fraction f of the faults in P will be revealed by test T that satisfies C. • Unfortunately there is no absolute measure of the effectiveness of any given coverage criterion for a general class of programs and for arbitrary test sets. Software Testing and Reliability Aditya P. Mathur 2002

  22. Effectiveness (contd.) • One coverage criterion results in an exception to this rule: What is it? • Empirical studies conducted by researchers give us an idea of the relative goodness of various coverage criteria. • Thus, for a variety of criteria we can make a statement like: Criterion C1 is definitely better than criterion C2. Software Testing and Reliability Aditya P. Mathur 2002

  23. Effectiveness-continued • In some cases we may be able to say: Criterion C1 is probably better than criterion C2. • Such information allows us to construct a hierarchy of coverage criteria. • This hierarchy is helpful in organizing and managing testing. How? Software Testing and Reliability Aditya P. Mathur 2002

  24. Strength of a coverage criterion • The effectiveness of a coverage criterion is also referred to as its strength. • Strength is a measure of the criterion’s ability to reveal faults in a program. • Criterion C1 is considered stronger than criterion C2 if C1 is is capable of revealing more faults than C2. Software Testing and Reliability Aditya P. Mathur 2002

  25. 0 1 coverage The Saturation Effect • The rate at which new faults are discovered reduces as test adequacy with respect to a finite coverage domain increases; it reduces to zero when the coverage domain has been exhausted. Software Testing and Reliability Aditya P. Mathur 2002

  26. N Remaining Faults M 0 tfs tfe tds tdfe tme Functional Testing Effort Saturation Effect: Fault View Software Testing and Reliability Aditya P. Mathur 2002

  27. R’m R’d R’df R’f Reliability Rm Rdf Mutation Rd Dataflow Decision Rf Functional True reliability (R) Estimated reliability (R’) Saturation region tfs tfe tds tde tdfs tdfe tms tfe Testing Effort Saturation Effect: Reliability View Functional, Decision, Dataflow, and Mutation tsting provide various test assessment criteria. Software Testing and Reliability Aditya P. Mathur 2002

  28. Coverage principle-discussion • Discuss: How will you use the knowledge of coverage principle and the saturation effect in organizing and managing testing? Can you think of any other uses of the coverage principle and the saturation effect? Software Testing and Reliability Aditya P. Mathur 2002

  29. Control flow graph • Control flow graph (CFG) of a program is a representation of the flow of execution within the program. • More formally, a CFG G is: • G=(N,A) where N: set of nodes and A: set of arcs • There is a unique entry node en in N. • There is a unique exit node ex in N. A node represents a single statement or a block. • A block is a single-entry-single-exit sequence of instructions that are always executed in a sequence without any diversion of path except at the end of the block. Software Testing and Reliability Aditya P. Mathur 2002

  30. Control flow graph (contd.) • Every statement in a block, except possibly the first one, has exactly one predecessor. • Similarly, every statement in the block, except possibly the last one, has exactly one successor. • An arc a in A is a pair (n,m) of nodes from N which represent transfer of control from node n to node m. • A pathof length k in G is an ordered sequence of arcs, from A such that: Software Testing and Reliability Aditya P. Mathur 2002

  31. Control flow graph (contd.) • The first node a1is en • The last node akis ex • For any two adjacent arcs ai= (n,m) and aj = (p,q), m=p. • A path is considered executable or feasible if there exists a test case which causes this path to be traversed during program execution, otherwise the path is unexecutable or infeasible. Software Testing and Reliability Aditya P. Mathur 2002

  32. Control flow graph-example Exercise: Draw a CFG for the following program and identify all paths.: 1. scanf (x,y); if (y<0) 2. pow=0-y; 3. else pow=y; 4. z=1.0; 5. while (pow !=0) 6. {z=z*x; pow=pow-1;} 7. if (y<0) 8. z=1.0/z; 9. printf(z); What does the above program compute? Software Testing and Reliability Aditya P. Mathur 2002

  33. en 1 scanf (x,y); if (y<0) pow=0-y; 2 3 else pow=y; 4 z=1.0; while (pow !=0) 5 7 if (y<0) {z=z*x; pow=pow-1;} 6 z=1.0/z; printf(z); 8 9 ex Control-flow Graph Software Testing and Reliability Aditya P. Mathur 2002

  34. Structure-based Test Adequacy • Based on the CFG of a program several test adequacy criteria can be defined. • Some are: • statement coverage criterion • branch coverage criterion • condition coverage criterion • path coverage criterion Software Testing and Reliability Aditya P. Mathur 2002

  35. Statement Coverage • The coverage domain consists of all statements in the program. Restated, in terms of the control flow graph, it is the set of all nodes in G. • A test T satisfies the statement coverage criterion if upon execution of P on each element of T, each statement of P has been executed at least once. Software Testing and Reliability Aditya P. Mathur 2002

  36. Statement coverage (contd.) • Restated in terms of G, T is adequate w.r.t. the statement coverage criterion if each node in N is on at least one of the paths traversed when P is executed on each element of T. Software Testing and Reliability Aditya P. Mathur 2002

  37. Statement Coverage (contd.) • Class exercise: • For the program for which you have drawn the control flow graph, develop a test set that satisfies the statement coverage criterion. • Follow the procedure for test assessment and improvement suggested earlier. Software Testing and Reliability Aditya P. Mathur 2002

  38. Statement Coverage-Weakness • Consider the following program: int abs (x); int x; { if (x>=0) x=0-x; return x; } Software Testing and Reliability Aditya P. Mathur 2002

  39. Statement coverage-weakness • Suppose that T= {(x=0)}. • Clearly, T satisfies the statement coverage criterion. • But is the program correct and is the error revealed by T which is adequate w.r.t. the statement coverage criterion? What do you suggest we do to improve T? Software Testing and Reliability Aditya P. Mathur 2002

  40. Branch (or edge) coverage • In G there may be nodes which correspond to conditions in P. Such nodes, also called condition nodes, contain branches in P. • Each such node is considered covered if during some execution of P, the condition evaluates to true and false; these executions of P need not be the same. Software Testing and Reliability Aditya P. Mathur 2002

  41. Branch coverage • The coverage domain consists of all branches in G. Restated, in terms of the control flow graph, it is the set of all arcs exiting the condition nodes. • A test T satisfies the branch coverage criterion if upon execution of P on each element of T, each branch of P has been executed at least once. Software Testing and Reliability Aditya P. Mathur 2002

  42. Branch coverage • Class exercise: • Identify all condition nodes in the flow graph you have drawn earlier. • Does T= {(x=0)} satisfy the branch coverage criterion? • If not, then improve it so that it does. Software Testing and Reliability Aditya P. Mathur 2002

  43. Branch Coverage-Weakness • Consider the following program that is supposed to check if the input data item is in the range 0 to 100, inclusive: int check(x); int x; { if ((x>=0 )&& (x<=200)) check=true; else check=false; } Software Testing and Reliability Aditya P. Mathur 2002

  44. Branch Coverage-Weakness • Class exercise: • Do you notice the error in this program? • Find a test set T which is adequate w.r.t. statement coverage and does not reveal the error. • Improve T so that it is adequate w.r.t. branch coverage and does not reveal the error. • What do you conclude about the weakness of the branch coverage criterion? Software Testing and Reliability Aditya P. Mathur 2002

  45. Condition Coverage • For example, in the check program the condition node contains the condition: • Condition nodes in G might have compound conditions. ((x>=0 ) && (x<=200)) • This is a compound condition which consists of the elementary conditions x>=0 and x<=200. Software Testing and Reliability Aditya P. Mathur 2002

  46. Condition coverage (contd.) • A compound condition is considered covered if all of its constituent elementary conditions evaluate to true and false, respectively, during some execution of P. • A test set T is adequate w.r.t. condition coverage if all conditions in P are covered when P is executed on elements of T. Software Testing and Reliability Aditya P. Mathur 2002

  47. Condition coverage (contd.) • Class exercise: • Improve T from the previous exercise so that it is adequate w.r.t. the condition coverage criterion for the check functionand does not reveal the error. • Do you find the above possible? Software Testing and Reliability Aditya P. Mathur 2002

  48. What might happen here? Branch coverage-weakness (contd.) • Consider the following program: 0. int set_z(x,y); { 1. int x,y; 2. if (x!=0) 3. y=5; 4. else z=z-x; 5. if (z>1) 6. z=z/x; 7. else 8. z=y; } Software Testing and Reliability Aditya P. Mathur 2002

  49. Branch Coverage-Weakness • Class exercise: • Construct T for set_z such that (a) T is adequate w.r.t. the branch coverage criterion and (b) does not reveal the error. • What do you conclude about the effectiveness of the branch and condition coverage criteria? Software Testing and Reliability Aditya P. Mathur 2002

  50. Path coverage • As mentioned before, a path through a program is a sequence of statements such that the entry node of the program CFG is the first node on the path and the exit node is the last one on the path. Is this definition equivalent to the one given earlier? Software Testing and Reliability Aditya P. Mathur 2002

More Related