A Guest Lecture on Testing-Based Software and System Reliability Evaluation (Part 2)

1. CSE 565 Software Verification, Validation, and Testing A Guest Lecture on Testing-Based Software and SystemReliability Evaluation (Part 2) Dr. Yinong Chen

2. Basic Input Domain Models Examples: MacWilliams73, Brown&Lipow75, Nelson78. Software reliability is defined as the probabilityR(N) = Prob{no of failures over N application runs} where N is the exposure period whose time unit is the number of application runs. Assuming that input cases are selected independently, then R(N) can be expressed as R(N) = (R(1))N = RN where, R R(1) is the expected reliability per application run. Now the question is how to estimate R.

3. Estimate R R, the reliability per test run, can be defined by the ratio of the number of test runs in which failures are observed and the total number of test runs when infinite number of different input cases are applied for test runs: R = 1 – F = 1 – Because of test time limit only a subset of the entire input domain can be applied to test the program in practice. Thus the reliability per test run, R, is usually estimated by

4. Example Test the program 10 000 time (test runs) Five (5) failures are observed F = 5 / 10 000 = 1 / 2 000 R = 1 - 1 / 2 000 = 0.9995 Reliability in N application runs are R(N) = RN

5. MacWilliams 73 and Brown&Lipow 75 With partition and profiling MacWilliams 73: the s input cases are selected randomly from the input domain. Brown&Lipow 75: the input domain is partitioned into m classes. If si input cases are selected from class Ci and fi failures are observed, the reliability can be calculated by where P(Ci) is a probability function reflecting the input profile in terms of classes.

6. Example The input domain is partitioned into 10 sub-domains. ˆ ˆ = - = 1 – 0.020938 = 0.979063 R 1 F

7. Case Study 1Anti-lock Braking System (ABS) • Requirement: • To obtain the maximum braking effect • Algorithm: • Define (or measure) the wheel diameter; • Measure the wheel rotations per seconds rps; • Compute the wheel velocity wv; • Measure the body velocity bv; • Error detection and action: • if (bv > wv), reduce braking force • else if (bv < wv), reduce acceleration force • else “no action”

8. Sample Code in C++ #include <iostream> using namespace std; const float mile_inch = 63360; const float pi = 3.1416; float wheel_diameter = 15; // inches float wheel_sensor() { float rps; cout << "get rotations per second: " << endl; rps = ReadWheelRotationSensor(); return rps; }

9. float wheel_velocity(float rps) { float wv; wv = (pi * wheel_diameter * rps * 3600)/mile_inch; return wv; } float body_velocity() { float bv; cout << "get miles per hour: " << endl; bv = ReadBodySpeedSensor(); return bv; }

10. void error_detection(float wv, float bv) { if (abs(bv - wv) < 0.01) cout << "no action" << endl; else if (bv > wv) cout << "reduce brake force!" << endl; else cout << "reduce acceleration force!" << endl; } void evaluation() { float rps, wv, bv; rps = wheel_sensor(); wv = wheel_velocity(rps); bv = body_velocity(); error_detection (wv, bv); } void main() { for (i = 1, I < 10000, i++) evaluation(); }

11. Apply the input domain reliability model • Write the random function that simulates: ReadWheelRotationSensor(); • Write the random function that simulates: ReadBodySpeedSensor(); • Test the program and collect date • Apply the input domain model to evaluate the reliability of the program

12. Case Study: Testing greatest common divisor program Source: Y. Chen, W.T. Tsai, Introduction to programming languages: Programming in C, C++, Scheme, Prolog, C#, and SOA, second edition, Kendall/Hunt Publishing Company, 2006, ISBN 0-7575-2974-7. Section 1.5.2.

13. greatest common divisor program #include <stdio.h> int gcd (int n0, int m0) { // n0  0, m0  0 int n, m; // n0 or m0  0 n = n0; m = m0; while (n != 0 && n != m) { if (n < m) m = m - n; else n = n - m; } return m; } void main() { int i, j, k; scanf("%d\n%d", &i, &j); //input k = gcd(i, j); // call gcd printf("%d\n", k); // output }

14. Random Test Case Generation Input Output k (i, j) = (6, 9) 3 (i, j) = (10, 5) 5 (i, j) = (0, 4) 4 (i, j) = (5, 7) 1 (i, j) = (8, 29) 1 The program produces correct outputs for all these different test cases. Can we claim that the program is correct?

15. Example: Input Domain Partition Analysis • Input Domain Analysis: • The program takes two integers as input. • The branches of the program are controlled by the relative values of the two integers. • Input Domain Partitioning: • Partition each integer input into three groups: < 0, = 0, and > 0. • For this program, < 0 is not allowed. The group that has only one value is called boundary value. • Considering the semantics of the program, prime numbers and nonprime numbers play a role in the program. Thus, the positive integers are further divided into prime and nonprime numbers.

16. Input Domain Partition Based on the analysis, we have a partition of: i: [0]; [2, 3, 5, 7, 11, …]; [4, 6, 8, 9, …] j: [0]; [2, 3, 5, 7, 11, …]; [4, 6, 8, 9, …] The combination of the two inputs generates following cases: (0, 0): This case is not allowed according to the specification. (0, 2), (0, 3), (0, 5), (0, 7), (0, 11), … (0, 4), (0, 6), (0, 8), (0, 9), … (2, 0), (2 2), (2, 3), (2, 5), (2, 5), … (2, 4), (2, 6), (2, 8), (2, 9), … (3, 0), (3, 2), (3, 3), (3, 5), (3, 7), … . . . (9, 0), (9, 2), (9, 3), (9, 5), (9, 7), …

17. Coverage Consideration: Branch Coverage   no n != 0?  yes  no n != m?  yes  no n < m?  yes m = m - n; n = n - m; return m;    Exit (0, 2),(0, 3),(0, 9),(0, 10):  (2, 2),(3, 3), (9, 9),(10, 10):  (2, 3),(2, 9), (2, 10): ... (3, 2),(9, 2),10, 2),(10, 3),(10, 9): ... (2, 0),(3, 0), (9, 0),(10, 0): ...

18. Testing the Program Using Test Cases Let's trace the program with (i, j) = (2, 0): (i, j) = (2, 0) (2-0, 0) (2-0, 0) ... A dead loop occurs – a design fault is found Inputs Output k (i, j) = (0, 2) 2 (i, j) = (2, 2) 2 (i, j) = (2, 3) 1 (i, j) = (3, 2) 1 (i, j) = (2, 0) ?

19. Apply the partition-based input domain reliability model • Modify the main program, so that it systematically generate (large number of) input cases from different sub-domain; • Test the gcd function and collect data • Apply the partition-based input domain model to evaluate the reliability of the program

20. SUMMARY SOFTWARE RELIABILITY MODELS • Basic concepts and terminology • Faults and failure rates • Reliability R(t) and availability A(t) • Software reliability models • Classifications • Time-domain models • Fault count model • Input-domain models • Fault seeding model • Sample code of real-time ABS software • Sample code of real-time ABS software • Partition-based input domain testing

21. Modeling Complex Systems (Software and Hardware) • A large system can be decomposed into smaller components. • Evaluate the reliability of the components; • Evaluate the reliability of the system based on known component reliabilities • Combinatorial Models • Markov Models

22. Markov Models Markov models are more generic than combinatorial models. They can handle repairs and much more complex situations. Assumption: • Any component may in one the two states: working or failed; • Probability of state transition depends only on the current state. ß Failure rates and repair rates are constants. ß Transition probability is proportional to the time that the component stays at a state. ß Exponential distribution of the reliability/availability

23. ß Construct differential equations (2) ß Solve the equations to obtain the probability in each state (3) (4) ß The reliability or availability is the sum of the probabilities of working states. Steps of Applying Markov Models A system consists of multiple components ß Construct state transition diagram (1)

24. System l 0 1 m A(t) = p0(t) Step 1: Construct state transition diagram Example 1: Simplex system with repair

25. Module A Module B Voter Input Output Module C Step 1: Construct state transition diagram Example 2: Reliability of TMR system with repair

26. b 4a 4b 00 (failed nodes, failed links) a 10 01 3a+4b 4a+3b 11 20 02 Step 1: Construct state transition diagram Example 3: A ring system with different node and link failure rates a and b. Assumethat the system fails if any two or more than components failed. Failed

27. Step 2: Construct differential equations = –l·p0 (t) + m·p1 (t) = l·p0 (t)–m·p1 (t) A(t) = p0(t) The question is how to obtain the probability of each state. p0 (t + Dt) = (1 –l · Dt)  · p0 (t) + m · Dt ·p1 (t) p1 (t + Dt) = l · Dt ·p0 (t) + (1 –m · Dt)  · p1 (t) Solve the differential equations to obtain (p0 (t), p1 (t)).

28. Step 2: Construct differential equations = –l·p0 (t) + m·p1 (t) p0 p1 -l m l -m = l·p0 (t)–m·p1 (t) = l 0 1 m

29. Step 3: Solve differential equations p0 p1 -l m l -m m l - l + m p0 ( ) t ( t ) = + e = l + m l + m l m - l + m p1 ( ) t ( t ) = + e l + m l + m • There are many different ways to solve differential equations • LaPlace Transformation • Tools like MatLab or Mathematica

30. m l - l + m ( ) t A ( t ) = + e l + m l + m p0 p0 ( ( t t ) ) = = Step 4: Find the Probabilities of Working States m l - l + m p0 ( ) t ( t ) = + e l + m l + m l m - l + m p1 ( ) t ( t ) = + e l + m l + m If m = 0, the probability at p0 represents the reliability m l - l + m = - l ( ) t t R ( t ) = + e e l + m l + m

31. Step 2: Construct differential equations (Find the pattern) a12 1 2 a21 a13 a23 a31 a32 a14 a41 3 a25 a52 a34 a35 a53 a43 a45 4 5 a54 p1 P2 p3 p4 p5 dp ( t ) aij = 2 dt … dp ( t ) 5 dt

32. Step 2: Construct differential equations (Find the pattern) In general, assume a STD has n states and is fully connected. Any state has n incoming and n outgoing transitions: aij 0 is the transition rate from state i to j. For i, j = 1, 2, ..., n, and i ≠ j.

33. Step 2: Construct differential equations (Find the pattern) where The probability in state j at t + Dt = the probability in state j at t+ incoming prob – outgoing prob Math manipulation: Divide Dt on both sides, let Dt 0

34. Let Dt 0 dp ( t ) ( ) ( ) ( ) n n n j = a - × a = a - × b å p ( t ) p ( t ) å å p ( t ) p ( t ) p ( t ) i ij j ji i ij j j dt j 1 1 1 = = = i i i ¹ ¹ ¹ i j i j i j ( ) ( ) n n a - a D å p ( t ) å p ( t ) t + D - D D D p ( t t ) t t t i ij j ji j 1 1 = = i i where ¹ ¹ i j i j = Step 2: Construct differential equations (More detail of the previous slide)

35. p1 p2 p3 … pn b1 a21 a31 a41 an1 a12 b2 a32 a42 an2 dp ( t ) = 2 a13 a23 b3 a43 an3 dt … dp ( t ) n bn a1n a2n a3n a4n dt where Step 2: Construct differential equations (found the pattern)

36. 3 l Example 1: Apply the Pattern -3 l m 0 æ ö ç T = -(2l+m) ÷ 0 ç ÷ è 0 ø 0 2l R(t) = p1(t) + p2(t)

37. 4a 4b 00 10 01 3a+4b 4a+3b 11 - 4 ( a + b ) 4 b 4 a 0 0 æ 0 ö ç ÷ - ( 4 a + 3 b ) 0 4 a + 0 3 b ç ÷ T = 0 - ( 3 a + 4 b ) ÷ ç 0 ç ÷ 0 0 è 3 a + 4 b ø Example 2 1 2 3 4 R(t) = p1(t) + p2(t) + p3(t)

38. SUMMARY • Basic concepts of reliability and reliability modeling • Hardware reliability models • Software reliability models • System reliability models consisting of multiple components • Combinatorial models • Markov models