INFO 631 Prof . Glenn Booker

1. INFO 631Prof. Glenn Booker Week 2 – Reliability Models and Customer Satisfaction INFO631 Week 2

2. Reliability Models • Reliability Models are used here to treat software as though we expect predictable performance from using it on a regular basis • Hence this assumes we’re dealing with fairly stable requirements, and a well controlled environment INFO631 Week 2

3. Why use Reliability Models? • Determine objective quality of the product • Use for planning resources needed to fix problems INFO631 Week 2

4. Independent Variable in Reliability Growth Models • Typical scope of measurement (X axis) includes one of these: • Calendar time (days of testing) • Cumulative testing effort (hours of testing) • Computer execution time (e.g. number of CPU hours) INFO631 Week 2

5. Key Dependent Variables for Reliability Growth Models • Typical dependent variables (Y) include: • Number of defects found per life cycle phase, or total number ever • Cumulative number of failures over time • Failure rate over time • Time between failures INFO631 Week 2

6. Terminology • Reliability – probability that system functions without failure for a specified time or number of natural units in a specified environment • “Natural unit” is related to an output of a system • Per run of a program • Per hours of CPU execution • Per transaction (sale, shipment) INFO631 Week 2

7. Terminology • Availability – probability at any given time that a system functions satisfactorily in a given environment • Failure intensity – the number of failures per natural or time unit INFO631 Week 2

8. Software Reliability Modeling • Requires characterizing and applying • The required major development characteristics or goals • Reliability • Availability • Delivery date • Life-cycle cost (development, maintenance, training, etc.) INFO631 Week 2

9. Software Reliability Modeling • The expected relative use of the software’s functions (i.e. its operational profile) • Focus resources on functions in proportion to their use and criticality INFO631 Week 2

10. Operational Profile • Operational profile - a complete set of operations with their probabilities of occurrence • Operation = a major system logical task of short duration which returns control to the system when complete, a.k.a. a scenario INFO631 Week 2

11. Types of Reliability Models • Static and Dynamic • Static • Uses other product characteristics (size, complexity, etc.) to estimate number of defects • Good for module-level estimates (detailed) • See discussions of size and complexity measures INFO631 Week 2

12. Types of Reliability Models • Dynamic • Based on statistical distribution; uses current defect pattern to estimate future reliability • Good for product-level estimates (large scale) • Includes the Rayleigh and Exponential models INFO631 Week 2

13. Dynamic Reliability Models • Model entire development process • Rayleigh model; • Model back-end formal testing (after coding) • Exponential model • Both are a function of time or life cycle phase, and are part of the Weibull family of distributions “back-end” here refers to the later phases of the life cycle INFO631 Week 2

14. Define • PDF = Probability Density Function, is the number of defects which will be found per life cycle phase • CDF = Cumulative Density Function, is the total number of defects which will be found, as a function of life cycle phase INFO631 Week 2

15. Weibull Model Let: m = curve family, c = shape parameter, t = time Then: PDF = (m/t)*(t/c)^m*exp(-(t/c)^m) CDF = 1 - exp(-(t/c)^m) What can this look like? Lots of things! First, fix c=1.5 and look at various ‘m’ values INFO631 Week 2

16. M=1 M=0.5 INFO631 Week 2

17. M=2 Notice the Y axis range is changing M=1.5 INFO631 Week 2

18. M=5 M=3 INFO631 Week 2

19. For large ‘m’ values, Weibull looks like a normal distribution centered on ‘c’ M=9 M=7 INFO631 Week 2

20. Rayleigh Model • The history of defect discovery across the life cycle phases often looks like the Rayleigh probability distribution • Rayleigh model is a formal parametric model, used to produce estimates of the future defect count INFO631 Week 2

21. Rayleigh Model • Rayleigh model and defect origin/found analyses deal with the defect pattern of the entire software development process • Is a good tool, since it can provide sound estimates of defect discovery from fairly early in the life cycle INFO631 Week 2

22. Rayleigh Model Let: m = 2, c = scale parameter, t = time Then: PDF = (2/t)*(t/c)^2*exp(-(t/c)^2) CDF = Cumulative defect arrival pattern CDF = 1 - exp(-(t/c)^2) INFO631 Week 2

23. Rayleigh Model Assumptions 1. Defect rate during development is correlated with defect rate after release. 2. If defects are discovered and removed earlier in development, fewer will remain in later stages. In short, “Do it right the first time.” INFO631 Week 2

24. Rayleigh Model • The value of ‘c’ determines when the curve peaks • tmax = c/(2) is the peak • Area up to tmax is where 39.35% of all defects will be found (ideally) • Now look at influence of ‘c’ value on curve shape INFO631 Week 2

25. , c=1.5 INFO631 Week 2

26. Rayleigh Model Implementation • Various tools can model the Rayleigh curve • PASW/SPSS (using Regression Module) • SAS • SLIM (by Quantitative Software Management) • STEER (by IBM) INFO631 Week 2

27. Rayleigh Model Reliability • Statistical reliability relates to confidence interval of the estimate, which is in turn related to sample size • Small sample size (only 6 data points per project) means low statistical reliability, often underestimating actual later reliability • Improve this by using other models and comparing results INFO631 Week 2

28. PTR Submodel • A variation on the Rayleigh model can be used for predicting defects which will be found during integration of new software into a system • PTR is Program Trouble Report or Problem Tracking Report, a common mechanism for defect tracking • Follows the same idea as Rayleigh INFO631 Week 2

29. Reliability Growth Models - Exponential Model • Exponential model is the basic reliability growth model - i.e. reliability will tend to increase over time • Other reliability models include: Time Between Failure Models and Fault Count Models INFO631 Week 2

30. Exponential Model • Reliability growth models are based on data from the formal testing phase • After the software has been completely integrated (compiled & built) • When the software is being tested with test cases chosen randomly to approximate an operational (real-world usage) profile • Testing is customer oriented INFO631 Week 2

31. Exponential Model • Rationale is that defect arrival during testing is a good indicator of the reliability of the product when used by customers • During this testing phase, failures occur, defects are fixed, software becomes more stable, and reliability grows over time INFO631 Week 2

32. Exponential Model • Is a Weibull distribution with m = 1 • Let: c = scale parameter, t = time, l = 1/c CDF = 1 - exp(-t/c) = 1 - exp(-lt) PDF = (1/c)*exp(-t/c) = l*exp(-lt) • l is the error detection rate or hazard rate • This form also works for light bulb failures, computer electrical failures, etc. INFO631 Week 2

33. Typical Time Between Failure Model Assumptions • There are N unknown software faults at the start of testing • Failures occur randomly • All faults contribute equally to failure • Fix time is negligibly small • Fix is perfect for each fault INFO631 Week 2

34. Time Between Failure Models • Jelinski-Moranda (J-M) Model • Assumes random failures, perfect zero time fixes, all faults equally bad • Littlewood Models • Like J-M model, but assumes bigger faults arefound first • Goel-Okumoto Imperfect Debugging Model • Like J-M model, but with bad fixes possible INFO631 Week 2

35. Fault Count Model Assumptions • Testing intervals are independent of each other • Testing during intervals is reasonably homogeneous • Number of defects detected is independent of each other INFO631 Week 2

36. Fault Count Models • Goel-OkumotoNonhomogeneous Poisson Process Model (NHPP) • # of failures in a time period, exponential failure rate (i.e. the exponential model!) • Musa-Okumoto Logarithmic Poisson Execution Time Model • Like NHPP, but later fixes have less effect on reliability INFO631 Week 2

37. Goel Okumoto model: m(t) = a*(1 - e-b*t) l(t) = m’(t) = a*b* e-b*t where: m(t) = expected number of failures observed at time t l(t) = failure density a = expected total number of defects b = constant Cumulative Defects versus Cumulative Test Hours INFO631 Week 2

38. Fault Count Models • The Delayed S and Inflection S Models • Delayed S: Recognizes time between failure detection and fix • Inflection S: As failures are detected, they reveal more failures INFO631 Week 2

39. Mean Time to Failure (MTTF) • Mean Time to Failure is the average amount of time using the product between failures • MTTF = (total run time) / (number of failures) INFO631 Week 2

40. 0 Execution Time Line Failure Software Reliability Modeling:Time Between Failures • Time between failures is expected to increase, as failures occur and faults are fixed INFO631 Week 2

41. 1.0 Reliability Time Since Last Failure (t) Software Reliability Modeling: Time Between Failures Reliability, R(t) - probability of failure free operation for a specified period of time INFO631 Week 2

42. WARNING • Reliability models can be wildly inaccurate, particularly if based on little and/or irrelevant data (e.g. from other industries, or using bad assumptions) • Validate estimates with other models and common sense INFO631 Week 2

43. Reliability Modeling 1. Examine data on a scatter diagram. Look for trends and level of detail. 2. Select model(s) to fit the data. 3. Estimate the parameters of each model. 4. Obtain fitted model using those parameters. 5. Check goodness-of-fit and reasonableness of models. 6. Make predictions using fitted models. INFO631 Week 2

44. Test Compression Factor • Defect detection during testing is different from that by customer usage, hence the defect rates may change. • Result is that fewer defects are found just after product release • Or, testing is better at finding defects than customer usage INFO631 Week 2

45. Test Compression Factor • Hence for maintenance, use reliability models ONLY for defect number or rate, and look for field defect rate patterns to be different from those found during development (number of defects found drops after release, due to less effective customer “testing”) INFO631 Week 2

46. Customer Satisfaction INFO631 Week 2

47. Customer Satisfaction • Customer evaluation of software is the most critical “test” • Want to understand what their priorities are, in order to obtain and keep their business INFO631 Week 2

48. Total Quality Management • Expanded from just product quality to maintaining a long term customer relationship • 5x cheaper to keep an existing customer than find a new one • Unhappy customers tell 7-20 people, versus happy customers tell only 3-5 people INFO631 Week 2

49. Customer Satisfaction Surveys • Customer call-back after x days • Customer complaints • Direct customer visits • Customer user groups • Conferences INFO631 Week 2

50. Customer Satisfaction Surveys • Want representative sample of all customers • Three main methods • In person interviewsCan note detailed reactionsMay introduce interviewer biasExpensive INFO631 Week 2