INFO 630 Evaluation of Information Systems Prof. Glenn Booker

1. INFO 630Evaluation of Information SystemsProf. Glenn Booker Week 1 – Statistics foundation INFO630 Week 1

2. Syllabus • This class focuses on understanding the types of measurements which can support a software development or maintenance project, including many key business metrics • We will use the statistics program PASW (was SPSS) to manipulate data and generate graphs early in the course INFO630 Week 1

3. Course overview • The course has three main objectives • How measurements are made (the statistical foundations); week 1 • How to choose what to measure (GQ(I)M approach and Ishikawa’s tools); week 4 • The rest of the course is devoted to understanding different types of measurements used for software development and maintenance INFO630 Week 1

4. My background • DOD and FAA contractor for 18 years • Use a Systems Engineering approach - because software doesn’t live in a vacuum! • Mostly work with long-lived systems, so maintenance issues get lots of attention • Metrics focus on supporting decision making during a project INFO630 Week 1

5. Week 1 overview • Identify the need for better measurement in software projects • Discuss software life cycles • Discuss process and quality models (ISO 9000, CMMI, 6s, etc.) • Define measurement scales and basic measures • Introduce key statistical concepts: R2, 95% confidence interval, t-statistic INFO630 Week 1

6. Software Crisis • For every six new large-scale software systems put into operation, two others are canceled • Average software development project overshoots its schedule by 50% • Three quarters of all large scale systems are operating failures that either do not function as intended or are not used at all INFO630 Week 1

7. Software Crisis • Most computer code is handcrafted from raw programming languages by artisans using techniques they neither measure or are able to repeat consistently • There is a desperate need to evaluate software product and process through measurement and analysis • That’s why we have required this course! INFO630 Week 1

8. Software Life Cycles • Some measurements are based on traditional software development life cycles, such as the waterfall life cycle • They can be adapted to other life cycles INFO630 Week 1

9. Conceptual Development RequirementsAnalysis Architectural (High Level) Design Detailed(Low Level)Design Coding &Unit Test SystemTesting Waterfall LifeCycle Model INFO630 Week 1

10. Waterfall Model • Conceptual Development includes defining the overall purpose of the product, who would use it, and how it relates to other products • Requirements Analysis includes definition of WHAT the product must do, such as performance goals, types of functionality, etc. INFO630 Week 1

11. Waterfall Model • Architectural Design, or high level design, determines the internal and external interfaces, component boundaries and structures, and data structures • Detailed Design, or low level design, breaks the high level design down into detailed requirements for every module INFO630 Week 1

12. Waterfall Model • Coding is the actual writing of source code, scripts, macros, and other artifacts • Unit Testing covers testing the functionality of each module against its requirements • System Testing can include “string” or component tests of several related modules, integration testing of several major components, and full scale system testing INFO630 Week 1

13. Waterfall Model • After system testing, there may be early release options, such as alpha and beta testing, before official release of the product • Early releases test the ability of your organization to deliver and support the product, respond to customer inquiries, and fix problems INFO630 Week 1

14. Prototyping Life Cycle • When requirements are very unclear, an iterative prototyping approach can be used to resolve interface and feature requirements before the rest of development is done • Do preliminary requirements analysis • Iterate Quick Design, Build Prototype, Refine Design until customer is happy INFO630 Week 1

15. Prototyping Life Cycle • Then resume full scale development of the system using some other life cycle model • It’s critical to do quick development cycles during prototyping, or else you’re just redeveloping the whole system over and over INFO630 Week 1

16. Spiral Life Cycle • Used for resolving severe risks before development begins, the spiral life cycle uses more types of techniques than just prototyping to resolve each big risk • Then another life cycle is used to develop the system • That’s a key point – the spiral life cycle isn’t used by itself! INFO630 Week 1

17. Iterative Life Cycle • Many modern techniques, such as the Rational Unified Process (RUP) advocate an iterative life cycle • RUP has four major phases, defined by the maturity of the system rather than traditional life cycle activities • Inception, Elaboration, Construction, and Transition INFO630 Week 1

18. Iterative Life Cycle • Like the spiral, iterative life cycles are driven by the need to resolve key risks, but here they are resolved all the way to implementation • Much more focus on early implementation of the core system, then building on it with each iteration INFO630 Week 1

19. Cleanroom Methodology • The Cleanroom methodology is a mathematically rigorous approach to software development • Uses formal design specification, statistical testing, and no unit testing • Produces software with certifiable levels of reliability • Very rarely used in the US INFO630 Week 1

20. Life Cycle Standards • The IEEE Software Engineering Standards are one source of information on many aspects of software development and maintenance • The standard ISO/IEC 12207, “Software Life Cycle Processes” has collected all major life cycle activities into one overall guidance document You can download ISO/IEC 12207 – see IEEE instructions on my web site INFO630 Week 1

21. Who cares… • …about statistics and measuring software activities? • The main models for guiding a software project, ISO 9000 and the Capability Maturity Model Integration (CMMI), both recommend use of statistical process control (SPC) techniques to help predict future performance by an organization • Six Sigma is all about SPC INFO630 Week 1

22. Process Maturity Models • Quality standards and goals are often embodied in process maturity standards, to guide organizations’ process improvement efforts • The primary software standard is the Software Engineering Institute’s (SEI’s) Capability Maturity Model Integration (CMMI) INFO630 Week 1

23. CMMI • Describes five maturity levels: • 1. Initial; all processes are ad hoc, chaotic, not well defined. Do your own thing. • 2. Repeatable; a project follows a set of defined processes for management and conduct of software development INFO630 Week 1

24. CMMI • 3. Defined; every project within the organization follows processes tailored from a common set of templates • 4. Managed; statistical control over processes has been achieved • 5. Optimizing; defect prevention and application of innovative new process methods are used INFO630 Week 1

25. Other CMM’s • CMMI is based on the original CMM for Software (SW-CMM) • The latter led to many other variations before the models were “integrated” in 2000 • The index of all SEI reports was ftp://ftp.sei.cmu.edu/public/documents/sei.documents.pdf • (File missing as of Sept. 2010) INFO630 Week 1

26. Malcolm Baldrige • The Malcolm Baldrige National Quality Award (MBNQA) is a US-based quality award created in 1988 by the Department of Commerce • Includes a broader scope, such as customer satisfaction, strategic planning, and human resource management INFO630 Week 1

27. ISO 9000 • The international standard for quality management of an organization is ISO 9000 • Now applies to almost every type of business, but was first used for manufacturing • Hence it includes activities like ‘calibration of tools’ INFO630 Week 1

28. ISO 9000 • ISO 9000 is facility-based, whereas CMMI is organization-based • A building gets ISO 9000 certification, but a project or organization gets CMMI • Was revised and republished in 2008 • Previous editions were 1987, 1994, and 2000 • “ISO 9001:2008 and ISO 14001: 2004 are implemented by over a million organizations in 175 countries.” – from here INFO630 Week 1

29. Six Sigma • Six Sigma was founded by Motorola • It’s best known for its Black Belt and Green Belt certifications • It focuses on process improvements needed to consistently achieve extremely high levels of quality INFO630 Week 1

30. Enter Measurement • Measurement is critical to all process and quality models (CMMI, ISO 9000, MBNQA, etc.) • Need to define basic concepts of measurement so we can speak the same language INFO630 Week 1

31. (Software or otherwise) Resources (People!) Technology and Tools Product Processes Engineering in a Nutshell INFO630 Week 1

32. Engineering in a Nutshell • So in order to create any Product, we need Resources to use Tools in accordance with some Processes • Each of these major areas (Product, Resources, Tools, and Processes) can be a focus of measurement INFO630 Week 1

33. Measurement Needs • Statistical meaning - need long set of measurements for one project, and/or many projects • Could use measurement to test specific hypotheses • Industry uses of measurement are to help make decisions and track progress • Need scales to make measurements! INFO630 Week 1

34. Measurement Scales • The measurement scales form the French word for black, noir (as in “film noir”) • Nominal (least useful) • Ordinal • Interval • Ratio (most useful) • ‘NOIR’ is just a mnemonic to remember their sequence INFO630 Week 1

35. Nominal Scale • A nominal (“name”) scale groups or classifies things into categories, which: • Must be jointly exhaustive (cover everything) • Must be mutually exclusive (can’t be in two categories at once) • Are in any sequence (none better or worse) INFO630 Week 1

36. Nominal Scale • Common examples include • Gender, e.g. “This room contains 19 people, of whom 10 are female, and 9 male” • Portions of a system, e.g. suspension, drivetrain, body, etc. • Colors (5 blue shirts and 3 red shirts) • Job titles (though you could argue they’re hierarchical) INFO630 Week 1

37. Ordinal Scale • This measurement ranks things in order • Sequence is important, but the intervals between ranks is not defined numerically • Rank is relative, such as “greater than” or “less than” • E.g. grades, CMM Maturity levels, inspection effectiveness ratings INFO630 Week 1

38. Interval Scale • An interval scale measures quantitative differences, not just relative • Addition and subtraction are allowed • Only examples: common temperature scales (°F or C), or a single date (Feb 15, 1962) • Interval scale measurements are very rare!! • A zero point, if any, may be arbitrary (90 °F is *not* six times hotter than 15 °F!) INFO630 Week 1

39. Ratio Scale • A ratio scale is an interval scale with a non-arbitrary zero point • Allows division and multiplication • E.g. defect rates (defects/KSLOC), test scores, absolute temperature (K or R), anything you can count, lengths, weights, etc. • The “best” type of scale to use, whenever feasible INFO630 Week 1

40. Scale Hierarchy • Measurement scales are hierarchical:ratio (best) / interval / ordinal / nominal • Lower level scales can always be derived if data is from a higher scale • E.g. defect rates (a ratio scale) could be converted to {High, Medium, Low} or {Acceptable, Not Acceptable}, which are ordinal scales INFO630 Week 1

41. Why Are Scales Important? • The types of statistical analyses which are possible depend on the type of scale used for the measurements • In statistics, this is roughly broken into parametric tests (for interval or ratio scaled data) or non-parametric tests (for nominal or ordinal scaled data) • Some tests are more specific about the data scale(s) needed, e.g. a test might require ordinal data INFO630 Week 1

42. Basic Measures - Ratio • Used for two exclusive populations • Everything fits into one category or the other, never both • Ratio = (# of testers) : (# of developers) • E.g. tester to developer ratio is 1:4 INFO630 Week 1

43. Proportions and Fractions • Used for multiple (> 2) populations • Proportion = (Number of this population) / (Total number of population) • Sum of all proportions equals unity • E.g. survey results • Proportions based on integer units; whereas fractions are based on real numbered units • E.g. what proportion of students answered A, B, C, or D on an exam question? INFO630 Week 1

44. Percentage • A proportion or fraction multiplied by 100 becomes a percentage • Only cite percentages when N (total population measured) is above ~30 to 50; always provide N for completeness • Why? Statistical methods are meaningless for very small populations INFO630 Week 1

45. Rate • Rate conveys the change in a measurement, such as over time, dx/dt. • Rate = (# observed events / # of opportunities)*constant • Rate requires exposure to the riskbeing measured • E.g. defects per kSLOC = (# defects)/(# of kSLOC)*1000 INFO630 Week 1

46. Data Analysis • Raw data is collected, such as the date a particular problem was reported • Refined data is extracted from one or more raw data, e.g. the time it took that problem to be resolved • Refined data is analyzed to produce derived data, such as the average time to resolve problems INFO630 Week 1

47. Models • Focus on select elements of the problem at hand and ignores irrelevant ones • May show how parts of the problem relate to each other • May be expressed as equations, mappings, or diagrams, such as • Effort = a + b*SLOC • where SLOC = source lines of code • Effort = a*(SLOC)b • May be derived before or after measurement (theory vs. empirical) INFO630 Week 1

48. Exponential Notation • You might see output of the form +2.78E-12 • This example means +2.78 * 10-12 • A negative exponent, e.g. –12, makes it a small number, 10-12 is 0.000000000001 • The leading number, here +2.78, controls whether it is a positive or negative number INFO630 Week 1

49. Precision • Keep your final output to a consistent level of precision, e.g. don’t report one number as “12” and another as “11.862512598235678563256091” • Pick a reasonable level of precision (“significant digits”) similar to the accuracy of your inputs • Wait until the final answer to round off INFO630 Week 1

50. Graphing • A typical graph shows Y on the vertical axis, and X on the horizontal axis • Y is the dependent variable, and X is the independent variable, since you can pick any value of X and determine its matching value of Y iff (if and only if) Y is a function of X • PASW/SPSS will sometimes ask for X and Y, other times independent and dependent variables INFO630 Week 1