Software Quality & Safety Assessment Using Bayesian Belief Networks

1. Software Quality & Safety Assessment Using Bayesian Belief Networks Joanne Bechta Dugan Susan Donohue, Ganesh Pai University of Virginia Department of Electrical and Computer Engineering University of Virginia

2. Problems Under Consideration • GETR: How does one decide that a software system is “good enough to release”? • SWQ-BBN: Can I combine process assessment and product assessment metrics to predict quality/reliability of a software system? Department of Electrical and Computer Engineering University of Virginia

3. Approach: Bayesian Belief Networks (BBN) • We use BBN models as the basis of both projects • BBN models effectively allow the combination of quantitative and qualitative assessment (that is, measures and expert judgment) in the same model Department of Electrical and Computer Engineering University of Virginia

4. GETR Approach (with S. Donohue) • For the GETR (Good Enough to Release) project, we are developing a BBN model of the decision process • What evidence is used, and how is it weighed • Determining conditional probabilities from expert opinion (to get probability parameters for the model) • GETR is building a mathematical framework based on BBN to understand and facilitate the decision making process Department of Electrical and Computer Engineering University of Virginia

5. How can we investigate and document the decision process that is used to go from... to… Is the system good enough to release? I have an acceptable level of belief that the system will operate as specified. Test Results Personal and Team CMM Quality Assurance Formal Methods Requirements Review Prototype Performance Risk Assessment Code Inspection Engineering Judgment for a computer-based system GETR Decision Department of Electrical and Computer Engineering University of Virginia

6. Department of Electrical and Computer Engineering University of Virginia

7. Quantifying Judgment for BBN Department of Electrical and Computer Engineering University of Virginia

8. Quantifying Judgment for BBN (QJ BBN) • Conditional probabilities (NPT entries) are generated as a function of the contribution of evidence to support a premise. For example, • Acceptable results from testing supports the conclusion that verification is acceptable. • Unacceptable documentation supports the premise that the artifact quality is unacceptable. • Evidence can overlap, be disjoint or synergistic. • Proofs of coherence of functions used in QJ methodology help assure rational decisions. • Importance and sensitivity analysis can help guide decision makers in seeking new evidence. • BN model provides a record of evidence analysis. Department of Electrical and Computer Engineering University of Virginia

9. Is the tool “fit for use”? Is it “good enough” to share? Is the tool appropriate for use in a given domain? What evidence is available for review? How much influence does certain evidence have on the approval process? Application to NASA: Seal of Approval Process (SOAP) for PRA tools Department of Electrical and Computer Engineering University of Virginia

10. Department of Electrical and Computer Engineering University of Virginia

11. SWQ BBN Approach (with G. Pai) • For the SWQ BBN project, we are developing techniques to build a BN to model the software development process and the products (artifacts) • BBN model represents causally related phases and activities within the phases. • Measurements or expert opinion can be used to determine probability parameters for the model. • Model can be used to assess the process/product with respect to reliability (defect density) or other quality attribute Department of Electrical and Computer Engineering University of Virginia

12. Department of Electrical and Computer Engineering University of Virginia

13. Candidate BBN for design phase Department of Electrical and Computer Engineering University of Virginia

14. Posterior Evidence Hypothetical illustrative example • Hypothetical priors • Model result • Medium defectcontent • Actual values dependenton the mappingbetween node statesand range values • E.g. Vlow, Low, Medium, High, Vhigh 0-20, 20-40, 40-60, 60-80, 80-100 • Model results  Defect content would lie in 40 – 60 range Department of Electrical and Computer Engineering University of Virginia

15. Feedback to the designer greater value • Network itself can provide feedback • Propagation of evidence • In this case: knowledge of high specification quality, observation of high defect content • Change in distribution indicates potential problem area Department of Electrical and Computer Engineering University of Virginia

16. Application to IV&V(joint work with Titan (Khalid Lateef)) Use IVV process for use case analysis, construct BBN from process model • Relevant process parameters and inputs represent parent nodes • Child nodes of BBN represent features desired from the requirements specification Department of Electrical and Computer Engineering University of Virginia

17. Department of Electrical and Computer Engineering University of Virginia

18. Example analysis • Probabilities reflect either measurement or analysts’ beliefs • The state ‘true’ is less than 95%  Not mature enough. Department of Electrical and Computer Engineering University of Virginia

19. Technology Readiness Level Department of Electrical and Computer Engineering University of Virginia

20. Data / Case Study Availability GETR case study domain: “lightweight” V&V for in-house developed analytical tools being considered for release to other centers or research groups. Identified case studies – RAP (JPL), SIAT (IV&V), and MATT (IV&V) SWQ BBN case study domain: Case study of system development, including artifacts & defect data. OO or ODC would be great Working with Khalid Lateef to develop case study for OO requirements analysis Department of Electrical and Computer Engineering University of Virginia

21. Barriers to Research or Application Case studies Department of Electrical and Computer Engineering University of Virginia