INFO 630 Evaluation of Information Systems Prof. Glenn Booker

INFO 630Evaluation of Information SystemsProf. Glenn Booker Week 4 – GQ(I)M and Ishikawa’s Tools INFO630 Week 4

Why Care About Measurement? • To quote Albert Einstein, "Not everything that counts can be counted and not everything that can be counted counts." • We are seeking to identify things 1) which can be counted, and 2) which count toward achieving our goals INFO630 Week 4

Reasons for Measurement • Measurements are required by all major process and quality models (CMM, ISO 9000, etc.) for four good reasons: • To Characterize or understand the current status of activities or products • To compare that understanding to our objectives, and Evaluate whether the current status is “good” or “bad” INFO630 Week 4

Reasons for Measurement • To Predict future performance, based on past trends • To form the basis for measuring Improvement • You won’t know if you improved if you don’t know where you started! INFO630 Week 4

Where do we get Information? • Metrics are often used to support decision making (the Evaluate step in the previous slide) • Decisions should be based on quantified information • To get that information, we calculate measures from raw data called data elements • The data elements each come from a data source INFO630 Week 4

How do we Choose What to Measure? • A commonly used method for selecting measurements is called GQ(I)M for Goal, Question, Indicator, and Measurement • It is based on GQM work by Victor Basili(first reported circa 1988-89) INFO630 Week 4

How do we Choose What to Measure? • The GQ(I)M method uses ten steps to describe measurements systematically • The steps don’t have to be followed in the order presented • Their main purpose is to help ensure that measurements have been fully thought out, so you can start at the top, or the bottom, or the middle • We’ll discuss them in a top-down approach, just to be linear INFO630 Week 4

1. Identify Business Goals • These are big, vague, lofty desired accomplishments or objectives for the organization: • Reduce cycle time • Improve customer satisfaction • Develop detailed process history • Respond to changing business environment INFO630 Week 4

1. Identify Business Goals • Reduce overhead • Improve competitive position • Increase market share • Improve product quality • Think of what you’d find cited in a company’s annual report – those are often high level business goals INFO630 Week 4

2. Identify Desired Knowledge • Break down each goal into products, resources, and activities (processes) needed to meet that goal • Think of questions like: • What activities do I manage or execute? • What do I want to achieve or improve? • To meet this, I will need to … INFO630 Week 4

3. Identify Subgoals • Set Subgoals (objectives) for each of the entities you manage • What do you want to know about the results of step 2? What kind of information is important to you? • How big, fast, expensive, complex, or much time will a process, product, tool, or resource take? INFO630 Week 4

4. Identify Entities and Attributes • Formulate questions to identify entities (document, product) created by your process, and the attributes of them you are interested in (size, quality, duration, cost) • Entities in this sense are often part of the inputs, outputs, or process associated with an activity INFO630 Week 4

4. Identify Entities and Attributes • Don’t get too detailed at this point - just identify the type of information desired (what are you studying), and the subject of that information (what about it do you want to know) • Then once the entities have been described, find the particular characteristics of interest (attributes) INFO630 Week 4

<Process name> Input(receives) Output(produces) Consists of…Holds... (products) entities entities entities (traits) attributes attributes attributes 4. Identify Entities and Attributes INFO630 Week 4

5. Define Measurement Goals • Form structured statements of the measurement goals for each attribute • This step is the heart of defining a metric in the form of a sentence • Two types of goals: • Active goals: reduce or improve something • Passive goals: identify, assess, understand INFO630 Week 4

5. Define Measurement Goals • Lower maturity organizations start with passive goals, then work on active goals • First measure existing trends, before predicting improvements • Description of a measurement goal includes the target entity, a purpose, a perspective, the environment and constraints INFO630 Week 4

5. Define Measurement Goals • It should include the quantity to be measured, the active or passive verb, describe the independent variable(e.g. time), and for active goals, quantify a desired amount or level • DO NOT report traits of individual people (fear of judgment) unless it’s a known part of their job INFO630 Week 4

5. Define Measurement Goals • Also need to balance how often measurements are made • More frequent measurement gives finer control, but excessive measurement wastes time and slows the process being measured • Can measure per release, per component, or some basis other than time INFO630 Week 4

5. Define Measurement Goals • Examples of measurement goals: • Passive: “Measure the number of requirements which changed each month.”“Identify the voluntary turnover rate per month for programmers and software engineers.” • Active: “Reduce the defect rate of developed code over time to under 20 defects/KLOC”“Improve the percent of satisfied customers after 30 days of product use to 95% or more” INFO630 Week 4

5. Define Measurement Goals • The general format is:<verb> <measure> <qualifier(s)> [objective] • Where the <qualifiers> indicate the scope or time frame of the measurements (a.k.a. independent variables), and [objective] is only given for active measurement goals INFO630 Week 4

6. Quantify Questions and Indicators • Pose questions to address your measurement goals (quantifiable ones, if addressing active goals) • Active: “Can we resolve customer emergencies, on average, in under 24 hours?” • Passive: “How many requirements do we have at the end of the Requirements Definition phase?” INFO630 Week 4

6. Quantify Questions and Indicators • Sometimes the search for a meaningful metric starts with a question, and that leads to filling out the GQ(I)M from the middle • Identify indicators to show the results effectively, such as: • Pie charts • Bar graphs • Scatter plots, etc. (see Ishikawa later) INFO630 Week 4

7. Identify Data Elements • Identify the data elements and equation(s) needed to prepare (calculate) each indicator • E.g. “defect rate by module each month” needs a table of defects found, for the last month, with the module each came from • Determine the source for each data element INFO630 Week 4

8. Define Measures • Describe how the data elements will be used to produce the indicators (i.e. what is the equation for the measure) • Even the most obvious measure could be defined many ways, so state the definition of each measure clearly • E.g. does your ‘turnover rate’ include all project employees, or only salaried ones? INFO630 Week 4

8. Define Measures • Define exactly what you mean by each measure - use a checklist if needed to show what is and isn’t included in its definition • Include rules, assumptions, constraints, and environment • Cite source if an unusual measure is used; or if you made it up, explain why INFO630 Week 4

An Aside • The first eight steps of the GQ(I)M approach define traceability from the business goals to the exact definition of each measure, and the way it is calculated from its data elements • The last two steps focus on the broader issue of planning a measurement approach for a project INFO630 Week 4

9. Identify Measurement Implementation • Analyze what measures are currently collected (if anything) and how they’re being used • Diagnose how well the current measurements meet your goals – ask ‘what’s missing?’ • Act on implementing new measurements, possibly in a phased approach based on priorities INFO630 Week 4

10. Prepare Measurement Plan • Take all of the aforementioned information and create a complete plan to identify and implement measurement for your organization or project • This is generally called a Metrics Plan or a Measurement Plan INFO630 Week 4

Summary of Core Steps • Goal (the big picture this is describing) • Subgoal (objective; why collect this metric) • Question(s) (answered by this metric) • Indicator (how display metric) • Measurement (the actual metric and its definition) • Data Elements (used to calculate various metrics) • Source (of each data element) INFO630 Week 4

Indicators • Indicators are the means used to present measures, such as charts, graphs, etc. • (yes, ‘indicator’ is an odd term for it, but pretend it makes sense) INFO630 Week 4

Indicators • To choose a good indicator, consider: • The Amount of Data to be presented for each interval (e.g. one measure at a time, or five different survey responses at once), and • The Number of Intervals to be shown, such as time units, modules of code, etc. INFO630 Week 4

Indicators • Different indicators are better at different Amount or Number characteristics • Consider also how your data will be presented - in color or B/W, live or printed? • Will your audience see pristine originals, or will it be copied a zillion times? INFO630 Week 4

Indicators • For graphs: • The X-axis of a graph (the horizontal line) is the independent variable • Is often a <qualifier> you choose before the measurement, such as time, severity, etc. • If X is time, describe how often measurements are made (weekly, monthly, every release, etc.) • The Y-axis of a graph (the vertical line) is the dependent variable; the thing you are measuring (the Measure) INFO630 Week 4

Pie Chart • The lowly pie chart is good for presenting a limited amount of information attractively • % of customers satisfied and not satisfied • % of defects by severity at this moment • Amount of Data: Shows a few data points (2-10) • Number of Intervals: One - it generally shows only one moment in time, or one set of measurements INFO630 Week 4

Ishikawa’s Seven Basic Tools • Developed circa 1950 for manufacturing production • Used widely in manufacturing quality control • Focuses on project level concerns; is this batch good enough to accept? • Not very useful for research; has little theoretical basis INFO630 Week 4

Ishikawa’s Seven Basic Tools • From a software perspective, these tools are often best for managers – focuses on identifying process and/or quality control issues • Not typically helpful for individual developers INFO630 Week 4

1. Check sheet • Used to gather data easily, consistently, and in a standard format • Helps to define key parts of a process, and make sure they are all performed • Examples include code inspection checklist, detailed test procedures • A check sheet used to help the quality of a process or product is a “checklist” INFO630 Week 4

2. Pareto Diagram • Used to identify problem areas - where to fix first; what are the biggest fires to put out • Defects tend to cluster in portions of code e.g. plot % of defects by the type of defect (logic, data definitions, etc.), and plot the total % of defects above it • Must list categories in descending order of frequency Helps look for the few components where most defects reside X axis must be a nominal variable INFO630 Week 4

3. Histogram • A bar chart is used to break down data by an ordered category (e.g. defect severity, satisfaction rating, etc.) • Can choose to put bars next to each other, or stack them. Stack when they add to a constant (e.g. 100%), or when the total is also a useful measure • Can show limited time spans, e.g. a few time intervals INFO630 Week 4

3. Histogram Stacked and cluster bar graphs for the same data set INFO630 Week 4

Example shows how many customers were in age ranges of 0-18, 19-30, 31-40, 41-50, 51-60, and 61+ 3. Histogram • A true histogram uses ranges of X axis values to show, for example, the number of events or data points in that range of values INFO630 Week 4

4. Run Charts • Purpose is to plot something important versus time • Often compare values to a desired or target value, especially at higher process maturity levels (CMM Level 3 and up) and for comparing to active goals • Special case: The “S” curve plots (cumulative % completion of something) versus time Usually uses a line chart INFO630 Week 4

5. Scatter Diagram • Is used to plot two measures against each other to see if there’s a correlation between them • E.g. Defect rate per module versus module complexity, or productivity versus experience • When we say ‘plot Blah versus Ick,’ Blah is the Y axis, and Ick is the X axis INFO630 Week 4

5. Scatter Diagram • If there’s enough data, can try to add curve fitted lines – we’ll cover this in the regression analysis discussion • This and control charts are the most powerful types of graphs for understanding your data • Many estimation rules and equations are derived from scatter diagrams INFO630 Week 4

6. Control Chart • Is a key Statistical Process Control tool • Hard to apply to software development • Specifications poorly defined • Each project takes a long time • Too many uncontrolled process variances • Process-quality relationship not well defined • Too many processes used • Rapidly changing technology • There are many varieties of control chart INFO630 Week 4

6. Control Chart • Control charts are often used at very high levels of process maturity (CMMI levels 4 & 5) INFO630 Week 4

6. Control Chart • Pseudo-control charts include: • The ‘u’ chart for defect rates by component, BMI, etc. versus time • The ‘p’ chart for percentages, such as inspection effectiveness or customer satisfaction rating • Upper and Lower Control Limits (UCL and LCL) are +/- 3 sigma from the mean (average) • Can add a warning limit at +/- 2 sigma INFO630 Week 4

7. Cause and Effect Diagram • A.k.a. the Fishbone chart • Is the least used Ishikawa tool, in my experience • Is not an Indicator per se, just a tool for capturing thoughts about a problem • Used with brainstorming to trace the causes of some outcome or result (good or bad) • Ask “what causes that” or “what influences that” to determine the major types of causes, then break them into more detailed events INFO630 Week 4

7. Cause and Effect Diagram • Sample fishbone diagram to analyze causes of Incorrect Deliveries Note the choices of types of causes (communication, skills, transport, procedures) vary from one problem to the next INFO630 Week 4

INFO 630 Evaluation of Information Systems Prof. Glenn Booker