Jeffrey M. Voas Chief Scientist

1. Four Mega-Challenges Facing The Commercial “Software Quality Movement”:Definition, Composition, Certification, and Commercialization and Return-On-Investment Jeffrey M. Voas Chief Scientist

2. Definitional Problem

3. Ask the CIO • Q: Are you interested in software quality? A: What do you mean by that? • Q: Are you interested in quality software? • A: Yes.

4. What is Software Quality? Subjective term that produces confusion among most software engineering professionals

5. IEEE • “Totality of features of a software product that bears on its ability to satisfy given needs.” [Source: IEEE-STD-729] • “Composite characteristics of software that determine the degree to which the software in use will meet the expectations of the customer.”[Source: IEEE-STD-729]

6. Reliable/ Accurate (integrity) Secure/ private Timeliness Three High-Level Attributes Quality Software

7. Reliable/ Accurate (integrity) Secure/ private Timeliness High-Level Attributes Quality Software Problem: Intuitive, but not formal

8. Lower-Level Attributes Quality Software Reliable/ accurate Secure/ private Timeliness reliability performance privacy availability security confidentiality fault tolerance fault tolerance testability intrusion tolerance Non-functional attributes (“ilities”) Functional attributes

9. Position Statement Software Quality (or Quality Software) must be viewed/defined as some combination of: (1) the degree to which the functional requirements are met, as well as, (2) the degree to which the non-functional requirements are met. reliability performance privacy availability security confidentiality + fault tolerance fault tolerance testability intrusion tolerance Non-functional attributes (“ilities”) Functional attributes

10. Position Statement Software Quality then is some combination of the following functional and non-functional attributes: Reliability [R], Performance [P], Safety [Sa] Fault Tolerance [F], Security [Se], Availability [A] Testability [T], and Maintainability [M]

11. However …. Software Quality can also be viewed as some combination of the previous attributes PLUS: Scalability, Usability, Sustainability, Survivability, Interoperability, Extensibility, Reusability, Readability, etc.

12. Eight in an Equation? Q = aR + bP + cF + dSa + eSe + fA + gT + hM where a, b, c, d, e, f, g, and h are units of quantitative or qualitative measures of a particular attribute.

13. Key Problems • The equation cannot be linear, since the units of measure for each attribute cannot be standardized (the apples and oranges problem). • dSa = Q – (aR + bP + cF + eSe + fA + gT + hM) • Most “ilities” are not quantifiably measurable. • Reliability, Availability, and Performance are measurable (via testing).

14. For Example …Maintainability • Size, defect density, amount of testing, T, R, cohesion, coupling, documentation, complexity, depth of inheritance, number of objects, testing infrastructure, mean-time-to-repair, experience of maintenance personnel as well as their domain knowledge, existence of impact analysis tools, etc., all impact M. • Q: So how can you assign a single numerical score for M?

15. Security • The level of security of an information system is a function of the partially unknown threat space, that changes by the minute. • Q: So how can you assign a single numerical score for Se? • A: You can assess, for a bounded set of anticipated threats, how the system will respond to those, e.g., 100 known threats, 50 mitigated. • A: Or you could measure the percentage of patches that are installed based on the number that need to be, and then test to make sure those installed work. Such information could also be used to give security “a score” but once again, that is only a score based on known threats and available patches.

16. It is more difficult to directly measure the quality of software than to achieve quality. The “Culprit” Phenomenon

17. So Where Does That Leave Us? Without a Numerical Quality Equation, But With a Way to Discuss the Attributes of Quality Software,and Therefore With a Means for Industry to Define Quality Goals

18. Composition Problem

19. Two Software Components x y

20. With Attributes x y x has the following properties: (aR, bP, cF, dSa, eSe, fA, gT, hM) y has the following properties: (iR, jP, kF, lSa, mSe, nA, oT, pM)

21. What Have You Got? y x Then f(xy) will inherit some level of Quality from the individual components. Is that level of quality an integer? Probability? An n-tuple of values? Color coded (green red yellow)? Key Point: The Composite Quality must represent something from which predictions of future behavior can be made.

22. Key Problems • It is hard enough to know, with any precise accuracy, what the composite reliability score will be as a result of the a and i values (let alone for the non-functional attributes). • But an even greater challenge exists here. For example, the security mechanisms in component y could thwart the performance that is built into component x. • Attributes are only reasonable to talk about within the context of a system, i.e., it is not reasonable to talk about them and attempt to measure them as standalone component properties. Their eventual target environments must weighed into their individual assessments.

23. Environment Quality Operational environment Reliable/ accurate Secure/ private Timeliness reliability performance privacy availability security confidentiality testability fault tolerance fault tolerance intrusion tolerance Non-functional attributes (“ilities”)

24. So Where Does That Leave Us? In Search of a Calculus or Calculi for Predicting How a Composite System Will Behave in the Future in a Specific Environment

25. Product Certification and Software Engineering Standards To Aide the Composition Problem Standardized Parts?

26. What is a Standard? Ideally, it is a line in the sand from which a certificate of compliance can be written.

27. Pros Any bar or hurdle is better than no bar or hurdle

28. Cons Possibly the developers would have done more to improve quality but now feel they have a license to do less.

29. Premise for SW Product Certification Commercially built software should be tagged with some guarantee (or at least a “warm fuzzy”) as to how good the software should be. Problem: Software Of Unknown Pedigree (SOUP) Goal of Product Certification: SO(Known)P

30. Three Schools of Thought All SE standards incorporate one or more of these perspectives Processes People Products

31. 1. Process: Clean Pipes, Dirty Water? Certifying that you know how to do things correctly does not mean that you do them correctly!

32. 2. People The IEEE Computer Society has developed a program to certify software engineering professionals. This program provides formal recognition of professionals who have successfully achieved a level of proficiency commonly accepted and valued by the industry.

33. 3. Product: The Software Itself 0% confidence 100% confidence Spectrum of possibilities as to what a certificate proclaiming that some “quantified” level of quality has been built in could state --- it could say anything in the range between “Nothing” (e.g., “here is a piece of software”, etc.) to “This software will always work perfectly under all conditions” (i.e., a 100% guarantee of perfection).

34. But Problems Exist With Standards • Vague: Develop software that only does "good" things • Common sense "dos" and "don'ts" - Very watered done by voting time • Disclaimers by publishing organizations • Profitable to organization that publishes them • Used only if mandated • Return-on-investment is unknown • Thwart intellectual creativity • "Protectionist" legislation • Paperwork • 2167A: ~400 English words per Ada code statement • "Old news" before being ratified • Relating one to another is very hard • Hundreds in existence • Cannot be easily tested for compliance • Mis-certifications are possible

35. Different interpretations • Lack of fairness during certification judgment • So much legacy code exists that complies with no standards and therefore get excluded in heterogeneous systems, making it’s impact to the system unknown.

36. Example of “Standards” Confusion • Suppose you have the following logical expression: (A and B) or (B and C) or (A and C) where A, B, and C are Boolean variables • To meet verification requirements for Level A software in RTCA DO178-B, you need to know the number of conditions in this statement Condition: A Boolean expression containing no Boolean operations How many conditions are there? 3, 4, 6, or 9 [Source: “Challenges in Software Aspects of Aerospace Systems”, K. Hayhurst & C.M. Holloway, Presented at the 26th Software Engineering Workshop, Greenbelt, MD, November 28, 2001]

37. The FAA Says … Distribution of Responses from 39 FAA Certification Authorities 41.0 35.9 17.9 5.1 [Source: “Challenges in Software Aspects of Aerospace Systems”, K. Hayhurst & C.M. Holloway, Presented at the 26th Software Engineering Workshop, Greenbelt, MD, November 28, 2001]

38. And the Answer is … 6 [Source: “Challenges in Software Aspects of Aerospace Systems”, K. Hayhurst & C.M. Holloway, Presented at the 26th Software Engineering Workshop, Greenbelt, MD, November 28, 2001]

39. Explanation (A and B) or (B and C) or (A and C) has 6 conditions • The full definition for condition is not contained in the glossary entry for that term • Part of the definition is given in the entry for decision Decision: A Boolean expression composed of conditions and zero or more Boolean operators. A decision without a Boolean operator is a condition. If a condition appears more than once in a decision, each occurrence is a distinct condition. [Source: “Challenges in Software Aspects of Aerospace Systems”, K. Hayhurst & C.M. Holloway, Presented at the 26th Software Engineering Workshop, Greenbelt, MD, November 28, 2001]

40. So Where Does That Leave Us? In Need of More Precise, Less Vague, and Repeatable Processes, for Grading The Quality of Software

41. Commercialization and ROI Issues

42. Commercialization Issues • Proven technology? (empirical vs. anecdotal) • Prototypes? Are they Maintainable/Extensible or Trashware? • Scalable? Theoretical or Practical? Maturity? • Automated? Is it a solution or standalone? • What languages/architectures does it support? Fad/Lifetime? • Difficult to learn? Ease of use? Time-to-market enabler or disabler? • Client base: commercial or government? Number of site? • Evolutionary (leap frog-able) vs. Revolutionary?

43. Compatible with existing technology (e.g., Microsoft)? • Point of Origination: University? Small business?, Large Corporation?, Government? • Competing foreign technologies? • Process, People, and Product - oriented? • Which attribute(s) does it address? Measurement or design? If design, how much of that attribute can it offer? • SOUP or SOKP? • Does this technology self-certify Quality after use?

44. Final Thoughts

45. 1. Quality is a Recipe As with food, ingredients of different types (liquids, powders, vegetables, meats, etc.) can all be mixed together. What food tastes like is a function of the ingredients and their proportions. Quality Software can be viewed/defined in a similar manner. And certain ingredients overpower others.

46. 2. Standards Beat Chaos • Software engineering standards are completely necessary despite their limitations. They usually are a “good rule of thumb”, but not an absolute process for achieving perfection. • Virtually any standard beats development chaos • What is Missing in SE Standards? • Not technology! • How to Implement, • How to gain regulatory approval given uncertain knowledge as to how judgment will be rendered, • Fairness in the certification processes, • And ROI (most are anecdotes, not statistical studies)

47. 3. Only Product Certification Can Address The Composition Problem • Until the non-functional attributes of software components can be graded, and assumptions about the target environments of those components can be nailed down (HUGE PROBLEM), a priori certification of the quality of any software component is suspect. • Research is needed into how to compose both functional and non-functional attributes.

48. 4. Attributes Need to Be Pre-Defined • Requirements should prescribe at some level of granularity as to what the weights are for various “ilities”, as well as how much of each “ility” is desired. • But HOW? • Ignoring the non-functional attributes is not an option for high assurance and trustworthy systems! Make an attempt to discuss them with the client even if quantification is not possible. Just get the issue on the table!

49. 5. Weighting is Important w1R w2P w3F w4Sa w5Se w6A w7T w8M in order to not over-design any attribute into the system. For example, for an e-commerce application, w4 would probably equal 0.0 and w7 would also be less than something like w2

50. 6. Tradeoffs How much will you spend for increased reliability knowing that doing so will take needed, financial resources away from security or performance or …?