Evaluating Health Promotion Programs

1. Evaluating Health Promotion Programs

2. Questions for the Planner • Did the program have an impact? • How many people stopped smoking? • Were the participants satisfied with the program? • Should we change anything about the way to program was offered? • What would happen if we just changed the time we offer the program?

3. Evaluating Health Promotion Programs • Evaluation: An Overview • Evaluation Approaches, Frameworks, and Designs • Data Analysis and Reporting

4. Evaluation: An Overview • There are two general categories of evaluation: Informal Evaluation, Formal Evaluation • Informal Evaluation: an absence of breath and depth; no procedures and formally collected evidence • e.g., consulting colleagues about a program concern, making a program change based on participant feedback. • Adequate when making minor changes in programs

5. Evaluation: An Overview • Formal Evaluation: “systematic well-planned procedures” • Most valuable procedure in health program evaluation • Controlled for a variety of extraneous variables that could produce evaluation outcomes that are not correct

6. Characteristics of Formal and Informal Evaluation • CharacteristicFormal Informal • Degree of freedom Planned activities Spontaneous activities • Flexibility Prescribed proces or protos Flexible procedures or protocols • Information Precision of information Depth of information • Objectivity Objective scores of measure Subjective impressions • Utility Maximal comparability Maximal informativeness • Bias Potential narrowed scope Subjective bias • Setting Controlled settings Natural settings • Inference Strong inferences Broad inferences Source: Willams and Suen (1998).

7. Important Points • The evaluation must be designed early in the process of program planning • Begins when the program goals and objectives are being developed • Must also answer questions about the program as it is being implemented • Must involve a collaborative effort of program STAKEHOLDERS (i.e., those who have a vested interest in the program)

8. Important Points • Understand that evaluation can be a political process • Judgment often “carries the possibilities of criticism, rejection, dismissal, and discontinuation (Green & Lewis, 1986, p. 16) • Need to have ethical considerations for the individuals involved • Institutional Review Board or Human Subject Review Committee

9. Define EVALUATION • In general terms: “Evaluation is a process of reflection whereby the value of certain actions in relation to projects, programs, or policies are assessed” (Springett, 2003, p. 264) • Applies to health promotion “The comparison of an object of interest against a standard of acceptability” (Green & Lewis, 1986, p. 362)

10. Standards of Acceptability Defined: “the minimum levels of performance, effectiveness, or benefits used to judge the value” (Green & Lewis, 1986) and are typically expressed in the “outcome” and “criterion” components of a program’s objectives. Standard of Acceptability Example Mandate (policies, statues, laws) Of regulating agencies Priority population health status Values expressed in the local community Standards advocated by professional Organizations Norms established via research Norms established by evaluation of previous programs Comparison or control groups Percent of children immunized for school; % of priority population wearing safety belts Rates of morbidity and mortality compared to state and national norms Type of school curriculum expected Passing scores, certification, or registration examinations Treadmill tests or percent body fat Smoking cessation rates or weight loss expectations Used in experimental or quasi-experimental studies

11. Types of Evaluation • Process Evaluation: “Any combination of measurements obtained during the implementation of program activities to control, assure, or improve the quality of performance or delivery. Together with preprogram studies, makes up formative evaluation” (Green & Lewis, 1986, p. 364). • Getting reactions from program participants about the times programs are offered or about program speakers are examples. Such measurements could be collected with a short questionnaire or focus group.

12. Types of Evaluation • Impact Evaluation: Focuses on “the immediate observable effects of a program, leading to the intended outcomes of a program intermediate outcomes” (Green &Lewis, 1986, p.363). • Measures of awareness, knowledge, attitudes, skills, and behaviors yield impact evaluation data.

13. Types of Evaluation • Outcome Evaluation: Focuses on “an ultimate goal or product of a program or treatment, generally measured in the health field by morbidity or mortality statistics in a population, vital measures, symptoms, signs, or physiological indicators on individuals” (Green & Lewis, 1986, p. 364). • Outcome evaluation is long-term in nature and takes more time and resources to conduct than impact evaluation.

14. Types of Evaluation • Formative Evaluation: “Any combination of measurements obtained and judgments made before or during the implementation of materials, methods, activities or programs to control, assure or improve the quality of performance or delivery” (Green & Lewis, 1986, p. 362). • Examples include, but are not limited to, a needs assessment or pilot testing a program.

15. Types of Evaluation • Summative Evaluation: Any combination of measurements and judgments that permit conclusions to be drawn about impact, outcome, or benefits of a program or method” (Green & Lewis, 1986, p.366).

16. Comparison of Evaluation Terms Planning → Start of Implementation → End of Implementation Formative Summative Process Impact Outcome Source: McKenzie et al. (2005, p. 295)

17. Purpose for Evaluation • Basically, programs are evaluated to gain information and make decisions • Six Specific Reasons: • To determine achievement of objectives related to improved health status • To improve program implementation • To provide accountability to fenders, community, and other stakeholders • To increase community support for initiatives • To contribute to the scientific base for community public health interventions • To inform policy decisions

18. The Process for EvaluationGuidelines • Planning • Review the program goals and objectives • Meet with the stakeholders to determine what general questions should be answered • Determine whether the necessary resources are available to conduct the evaluation; budget for additional costs • Hire an evaluator, if needed. • Develop the evaluation design • Decide which evaluation instrument(s) will be used and, if needed, who will develop the instrument • Determine whether the evaluation questions reflect the goals and objectives of the program • Determine whether the questions of various groups are considered, such as the program administrators, facilitators, planners, participants, and funding source • Determine when the evaluation will be conducted; develop a time line

19. The Process for EvaluationGuidelines • Data Collection • Decide how the information will be collected; survey, records and documents, telephone interview, personal interview, observation • Determine who will collect the data • Plan and administer a pilot test • Review the results for the pilot test to refine the data collection instrument or the collection procedures • Determine who will be included in the evaluation (e.g., all program participants, or a random sample of participants) • Conduct the data collection

20. The Process for EvaluationGuidelines • Data Analysis • Determine how the data will be analyzed • Determine who will analyze the data • Conduct the analysis, and allow for several interpretations of the data

21. The Process for EvaluationGuidelines • Reporting • Write the evaluation report • Determine who will receive the results • Choose who will report the findings • Determine how (in what form) the results will be disseminated • Discuss how the findings of the process or formative evaluation will affect the program • Decide when the results of the impact, outcome, or summative evaluation will be made available. • Disseminate the findings

22. Practical Problems or Barriers in Evaluation • Planners either fail to build evaluation in the program planning process to do so too late (Solomon, 1987; Valente, 2002; Timmreck, 2003). • Resources (e.g., personnel, time, money) may not be available to conduct an appropriate evaluation (Solomon, 1987; NCI, 2002; Valente, 2002). • Organizational restrictions on hiring consultants and contractors (NCI, 2002). • Effects are often hard to detect because changes are sometimes small, cone slowly, or do not last (Solomon, 1987; Glasgow, 2002; Valente, 2002). • length of time allotted for the program and its evaluation (NCI, 2002). • Restrictions (i.e., Policies, ethics, lack of trust in the evaluators) that limit the collection of data form those in the priority population (NCI, 2002).

23. Practical Problems or Barriers in Evaluation • It is sometimes difficult to distinguish between cause and effect (Solomon, 1987). • It is difficult to separate the effects of multi-strategy interventions (Glasgow et at., 1999), or isolating program effects on the priority population from “real world” situations (NCI, 2002). • Conflicts can arise between professional standards and do-it-yourself attitudes (Solomon, 1987) with regard to appropriate evaluation design. • Sometimes people’s motives get in the way (Solomon, 1987; Valente, 2002). • Stakeholders’ perceptions of the evaluation’s value (NCI, 2002). • Intervention strategies are sometimes not delivered as intended (i.e., type III error) (Glasgow, 2002), or are not culturally specific (NCI, 2002; Valente, 2002).

24. Who Will Conduct the Evaluation • Internal evaluation (within a department) • External evaluation (outside a department) • Persons with credibility and objectivity • Clear about the evaluation goals, design, and accurately reporting the results regardless of the findings.

25. Evaluation Approaches, Frameworks, and Designs • Seven Major Evaluation Approaches • 1. Systems Analysis Approaches • Efficiency-based: determining which are the most effective programs using inputs, processes, and outputs. • Economic evaluations ate typical strategies used in system analysis approaches – Comparison of alternative courses of action in terms of both costs and outcomes • Two common cost analyses in health promotion • Cost-benefits: how resources can be best used – yielding the dollar benefits received from the dollars invested in the program • Cost-effectiveness: quantifies the effects of a program in monetary terms. Indicates how much it costs to produce a certain effect (e.g., years of life saved, # of smokers who stop smoking). • Check out the handout by McKenzie et al. (2005) on “cost-benefit and cost-electiveness as a part of the evaluation of health promotion programs.”

26. Evaluation Approaches, Frameworks, and Designs • Evaluation Approaches • 2. Objective-Oriented Approaches • Specify program goals, and objectives, and collect evidence to determine if the goals and objectives have been reached

27. Evaluation Approaches, Frameworks, and Designs • Evaluation Approaches • 3. Goal-Free Approaches • Focus on all outcomes, including unintended positive or negative side effects. • 4. Management-Oriented Approaches • Focus on identifying the meeting the informational needs of managerial decision makers

28. Evaluation Approaches, Frameworks, and Designs • EvaluationApproaches • 5.Consumer-Oriented Approaches • Focus on developing evaluative information on “products.” (e.g., using checklists and criteria to allow an evaluation of the “product.” • 6. Exercise-Oriented Approaches • Reply “primarily on the direct application of professional expertise to judge the quality of whatever endeavor is evaluated

29. Evaluation Approaches, Frameworks, and Designs • EvaluationApproaches • 7. Participant-Oriented Approaches • A unique one. Focus on a process in which involvement of participants (stakeholders in that which is evaluated) are central in determining the values, criteria, needs, data, and conclusions for evaluation.

30. Framework For Program Evaluation • Six Steps must be completed in any evaluation, regardless of the setting (CDC, 1999) • Step 1: Engaging stakeholders • Step 2: Describing the program • Step 3: Focusing the evaluation design • Step 4: Gathering credible evidence • Step 5: Justifying the conclusions • Step 6: Ensuring use and sharing lessons learned

31. Engage stakeholders Ensure use and share lessons learned Describe the program Standards Utility Feasibility Propriety Accuracy Focus the evaluation design Justify conclusions Gather credible evidence Framework for Program Evaluation

32. Framework For Program Evaluation • Four Standards of Evaluation (CDC, 1999) • Utility standards ensure that information needs of evaluation users are satisfied • Feasibility standards ensure that the evaluation is viable and pragmatic • Propriety standards ensure that the evaluation is ethical • Accuracy standards ensure that the evaluation produces findings that are considered correct

33. Selecting an Evaluation Design • Critical to the outcome of the program • Few perfect evaluation designs • Because situation is ideal, and there are always constraining factors, such as limited resources • Challenge is to devise an optimal evaluation as opposed to an ideal evaluation

34. Selecting an Evaluation DesignAsking These Questions • How much time do you have to conduct the evaluation • What financial resources are available? • How many participants can be included in the evaluation? • Are you more interested in qualitative or quantitative data? • Do you have data analysis skills or access to computers and statistical consultants?

35. Selecting an Evaluation DesignAsking These Questions • In what ways can validity be increased? • Is it important to be able to generalize your findings to other populations? • Are the stakeholders concerned with validity and reliability? • Do you have the ability to randomized participants into experimental and control groups? • Do you have access to a comparison group?

36. Selecting an Evaluation DesignA Four-Step Model (Dignan, 1995) Orientation to The situation Resources, constraints, And hidden agendas Step 1 Dependent variables Independent variables Confounding variables Defining the problem Step 2 Step 4 Plans for Qualitative Measurement Data collection Data analysis Reporting of results Basic design decision Step 3 Quantitative Combination of both

37. Evaluation Design • True Experimental Design • Pretest-posttest design: Exp. vs. Control • R O1 X O2 • R O1 O2 • Posttest-only Design: Exp. vs. Control • R X O • R O • Time series Design: Exp. vs. Control • R O1 O2 O3 X O4 O5 O6 • R O1 O2 O3 O4 O5 O6

38. Evaluation Design • Quasi-Experimental Design • Pretest-posttest design: Exp. vs. Control • R O1 X O2 • R O1 O2 • Time series Design: Exp. vs. Control • R O1 O2 O3 X O4 O5 O6 • R O1 O2 O3 O4 O5 O6

39. Evaluation Design • Non-Experimental Design • Pretest-posttest design: Experimental only • R O1 X O2 • Time series Design: Exp. vs. Control • R O1 O2 O3 X O4 O5 O6

40. Internal Validity • Defined: the degree to which the program caused the change that was measured. • Factors that can threaten internal validity • History, maturation, testing, instrumentation, statistical regression selection, mortality, etc.

41. External Validity • Defined: the extent to which the program can be expected to produce similar effects in other populations – known as generalizability. • Factors that can threaten external validity • Social desirability, expectancy effect, Hawthorne effect, placebo effect, etc.

42. Data Analysis and Reporting • Data Management • Data Analysis • Univariate data analyses • Examines one variable at a time • Bivariate data analyses • Study two variables simultaneously • Multivariate data analyses • Study three or more variables simultaneously

43. Applications of Data Analyses: CASE #1 Program Goal: Reduce the prevalence of smoking Target Population: The seventy smoking employees of Company XYZ Independent Variable: Two different smoking cessation programs Dependent Variable: Smoking cessation after one year Design: R A X1 O1 R B X2 O1 where: R = Random assignment; A = Group A; B = Group B; X1 = Method 1; X2 = Method 2; O1 = self-reported smoking behavior Data collected: Nominal data: quite yes or no Smoking Employees A B Quit 24% 33% Did not quite 76% 67% Data analysis: A chi-square test of statistical significance for the null hypothesis that there is no difference in the success of the two groups

44. Applications of Data Analyses: CASE #2 Program Goal: Increase the AIDS knowledge Target Population: The 1,200 new freshmen at ABC University Independent Variable: A 2-hour lecture-discussion program Dependent Variable: AIDS knowledge Design: O1 X O2 where: O1 = pretest; X = 2-hour program at freshman orientation; O2 = posttest Data collected: Ratio data; scores on 100-point-scale test Test Results Pretest Posttest Number of students 1,200 1,200 Mean score 69.0 78.5 Data analysis: A dependent t-test of statistical significance for the null hypothesis that there is no difference between the pre- and posttest means on the knowledge test

45. Applications of Data Analyses: CASE #3 Program Goal: Improve breast cancer self-examination skills Target Population: All women employees in the Valley River Shopping Center Independent Variable: A 2-hour training on self-examination Dependent Variable: score on breast cancer self-exam skills test Design: A O1 X O2 B O1 O2 where: A = all women in VRC; B = all women in the Cascade shopping center; O1 = pretest scores; X = a 2-hour program on skill training; O2 = posttest scores Data collected: Ratio data; scores on 100-point skills test Test Results VCR Cascade (n=142) (n=131) Pre 62 63 Post 79 65 Data analysis: An independent t-test of statistical significance for the null hypothesis of (a) no difference in the pretest scores of the 2 groups; and (b) no difference between the posttest means of the 2 groups.

46. Interpreting the Data (1) • Determining whether objectives have been achieved; • Determining whether laws, democratic ideals, regulations, or ethical principles have been violated; • Determining whether assessed needs have been reduced;

47. Interpreting the Data (2) • Determining the value of accomplishments; • Asking critical reference groups to review the data and to provide their judgments of successes and failures, strengths, and weakness; • Comparing results with those reported by similar entities or endeavors; • Comparing assessed performance levels on critical variables to expectations of performance or standards; • Interpreting results in light of evaluation procedures that generated them

48. Interpreting the Data (3) • Distinguish between program significance (practical significance) and statistical significance • Program Significance: the meaningfulness of a program regardless of statistical significance. • e. g., 70 vs. 69 (out of 100 points) • Statistical Significance: determined by statsistical testing • e.g., at 5% type I error level

49. Evaluation Reporting • Designing the Written Report • Abstract/executive summary • Introduction • Methods/procedures • Results • Conclusions/recommendations

50. Evaluation Reporting • Presenting Data – Guidelines • Use graphic methods whenever possible • Build the results and discussion around tables and figures • Provide instructions on how to read them