1 / 19

Why Metrics in Software Testing?

Why Metrics in Software Testing?. How would you answer questions such as: Project oriented questions How long would it take to test? How much will it cost to test? Product oriented questions How bad/good is the product ? How many problems still remain in the software?

colleenwilliams
Télécharger la présentation

Why Metrics in Software Testing?

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Why Metrics in Software Testing? • How would you answer questions such as: • Project oriented questions • How long would it take to test? • How much will it cost to test? • Product oriented questions • How bad/good is the product ? • How many problems still remain in the software? • Test activities oriented questions • Will testing be completed on time? • Was the testing effective? • How much effort went into testing • All these Questions require some type of measurements and record keeping in order to answer properly.

  2. Some Basic Concepts on Measurement • What do we need before we can measure something? • Clear understanding and definition of the attribute/characteristic that we are trying to gauge • The metric that may be used to gauge that attribute • The methodology for performing the measurement. (often forgotten once we get the first two done ---- including yours truly.)

  3. Clarifying & Defining the Attribute to be Measured • Characterizing the attribute of interest • Size Attribute: • Physical height is a sizesub-attribute of many items. • Height of a building, person, tree - - - not a problem • Height of a ball or ocean ? - - - not comfortable? Why? • Physical weight is a sizesub-attribute of many items • What is the size attribute for software? What does it address? • The source statements - - - with screens? with db tables? • The storage space that the object code occupies in memory ? • Quality Attribute: • For a car ? - - - how fast it can accelerate? Number of times the car stalled? Number times the lights don’t work? • For software? - - - how many times we need to “re-boot”?, how good does the screen look? How many times do we need to call help-line? Or (# of times not Meeting customer requirements)

  4. 2. Metric for Gauging the Attribute • Metric – a unit used for describing or for measuring an attribute • Inches is a metric used for measuring the length attribute (simple metric) • Miles per hour is a metric for measuring the speed attribute (complex metric – requires 2 metrics) • Lines of code is a metric for measuring the size attribute of software (not a very good one) • Problems found per thousand lines of sourcecode is a metric for defect discovery rate attribute of software. (or is this for software quality attribute?

  5. 3. Conducting the Measurement • Once the attribute is defined and the associated metric is defined, the actual methodology to determine the extent of an attribute using that metric has to be spelled out. • How do you measure the length of a person using inches? • How do you measure the distance from earth to the moon using inches? • How do you measure the size of the computer program using bytes? • How do you measure the defects in a program using problems found during program testing? ( note: problems found may be counted in many ways - - - unique ones, accepted ones, etc.)

  6. Some General, Test Measurements • Time is used to measure the length of period expended for testing • Time to setup and conduct (run) a test or a set of tests • Units of measurement in minutes or hours • Time to design and document test cases • Units of measurement in minutes or hours • Keeping track of time gives us one parameter to help us plan for future testing; but time must be balanced with the “size” of the test. • 2 seconds to run a simple query • 5 seconds to run a complete purchase transaction with confirmation • “Size” of test is needed to make “time of test” more meaningful or conversely can amount of “test time” be used as a metric for size of test attribute?

  7. Size of Test • Test size attribute may use different metrics: • Amount of time to run test: (bit convoluted ?) • Small size : less than or equal to 3 seconds • Medium size: between 3 seconds and 1 minute • Large size: 1 minute or above • Number of lines of statements to document the test case: • Small size: less than or equal to 3 statements • Medium size: between 4 and 7 statements • Large size: 8 or more statements Any suggestions - - - - ? Number of test cases? --- or --- type of test such as unit test versus integration test ?----

  8. Quality : # of Problems • The attribute , Quality, is often measured with the metric of number of problems found; but number of problems alone does not tell the whole story - - - consider • Severity of problems • High • Medium • low • Type of problems • UI • Database • Network outage • Etc.

  9. Quality (cont.) • Both Severity and Type are important • # of problems found by severity • # of problems found by type • # of problems found when (when during development) • # of problems found when (months after release) • # of problems found where (UI,DB, Logic, Network, etc.) • Quality Information is relevant to both: • Software providers • Customers/users Why important to users? What would they do with it?

  10. Problem Find Rate The Weibull probability density curve: f(t) = (m/t) (t/2)m e –z where z = (t/c)m - for m= 1, the curve looks as dotted line - for m = 2, the curve looks as solid line and is called Rayleigh Problem Find Rate # of Problems Found per hour Time Day 1 Day 2 Day 3 Day 4 Day 5 Does severity of problem matter here? (it should , but not considered here)

  11. Problem Fix Rate Problem Fix Rate Problem Find Rate During Functional Test # of Problems Fixed per hour Problem Fix Rate During Functional Test Time Day 1 Day 2 Day 3 Day 4 Day 5 Would this fix rate present a problem ? Would you also want to keep a backlog # by day ?

  12. Problem Density Density Note: Just the # of problems found by area does not normalize the measurement; we need the per KLOC. 6 5 # of problems found per KLOC 4 3 2 1 Area Module 1 Module 2 Module 3 Module 4

  13. Test CoverageRate • Not all the planned test cases are actually run. • # of test cases executed / # of test cases planned • By functional areas • By test phases • # of source statements executed / total # of source statements • By functional areas • By modules

  14. Test Activity Effectiveness • Defect discovery and eradication activities occur at all phases of development. To see which is more effective one may use: • # of problems found / total # of problems found • By development phase (req. rev., design rev., func. test, system, etc.) • # of problems found / person-days of effort • By test activities (e.g. boundary value testing, branch testing, d-u testing, etc.)

  15. Fix Effectiveness • Not all problem fixes resolve the problems. • # of fixes that worked / total # of fixes • The first time • # of fixes that required more than 1 fix / total number of fixes

  16. Fix Cost • Fix cost is usually measured by amount of effort expended. • # of person-hours expended / fix • By severity • By areas • By phase type (including post-release) If the fix cost for post-release is higher than that of all of the pre-release phases, then that will be one reason for test and reviews.

  17. Problem Cost Comparison • Effort expended in discovering a problem and the effort expended in fixing that problem is the “test”cost during pre-release. • Effort expended in fixing a problem and releasing it to the customer is the “support” (problem resolution) cost during post-release. • Compare: (effort in people hours) effort expended / problem found and “fixed” (pre-release) .vs. effort expended / problem “resolved” (post-release) Post-release resolution usually cost more

  18. How “Big” is it (testing w/o fix) ? How would you answer this? • Assume --- # of test cases planned by size (or complexity): • large – 35 test cases • Medium – 200 test cases • small – 40 test cases • Assume --- average effort required to design and test • large – 1 person hour • Medium – 15 person minutes • small – 5 minutes • Then ---- “How Big is Testing?” may be answered • (35X60) + (200x15) + (40x5) = 5,330 person-minutes or 88.33 person-hours So, In this case --- how big is testing? - It is 275 test cases. - It is 88.33 person hours of effort.

  19. How Long Would it take? • Use the same example of 88.33 person-hours of test planning and execution effort. • You need to make some assumptions: • assume 2 testers of about equal ability • split the work effort evenly • 88.33people-hours/2 people = 44.17 hours • further assume that each person works 6 hours a day • 44.17 hours/ 6hours-perday = 7.3 days • So this will take 2 testers working 6 hours a day for 7.3 days

More Related