PM 536 Program Evaluation & Research

1. PM 536Program Evaluation & Research • Course Outline, Objectives & Expectations • “Health Promotion Program Evaluation” • Program Evaluation is Fun • Program Evaluation is Difficult

2. Course Materials 1) Text (Everything you always wanted to know about program evaluation, but were afraid to ask) 2) Supplemental readings (Some case studies and other methodology points) 3) Lab materials Labs SAS/SPSS/STATA 4) Overhead copies (6 slides/page)

3. What is Evaluation • Systematic application of research procedures that assess the conceptualization, design, implementation, and effectiveness intervention programs Did it work How effective was it What does it mean

4. The Evaluation Field Evaluation has had an explosive growth since WWII as social programs have expanded Evaluation has emerged as a distinct field of inquiry boasting its own professional association and own journals. Three stages of evaluation: Shadish, Cook and Leviton (1991) 1. Defining scientific rigor (Campbell & Stanley; Scriven, etc.) 2. Practicalities of evaluation in the field (Weiss, Wholey, Stake) 3. Integrate lessons to construct evaluation theory that is appropriate and rigorous (Cronbach, Rossi)

5. Evaluation can be Threatening Acknowledge that evaluation can be “value” laden. In reality evaluation is rarely used to terminate someone’s job However, evaluation is used to decide on where to focus resources and this causes concern Evaluation can be empowering

6. Why Do We Evaluate Programs • Determine whether the program met its objectives • Force programmers to set objectives • Determine how the program worked (if it did) • Determine who was influenced by the program • Often required (Fed. Gov’t Acct. Act, GRPA) • Provides research opportunities • Plan for future programs • Interesting

7. Evaluation is Difficult Control groups hard to create a. Hard to restrict programs (mass media) b. People move c. People talk Programs are often culturally-specific Control variables are hard to specify Cost is often perceived to unjustifiable

8. Evaluation Framework • Formative - pre-intervention planning • Process - intervention implementation • Summative - post-intervention impact analysis

9. Old Figure 1-1 Communication Campaign Evaluation Framework Formative Research Design Stage Analysis Stage Observations Depth Interviews Focus Group Discusions Key Informants Community Mapping Secondary Analysis Determine Impacts Determine Outcomes Design & Test Messages Design & Test Instruments Design & Test Monitoring Methods Implement Program/Campaign Needs Assessment/ Identification Collect Follow-up Data Share Findings with Stakeholders Collect Baseline Data Disseminate Findings Conduct Program & Effects Monitoring

10. Figure 1-1. Program Evaluation Framework Needs Identification and Assessment Formative: Observations, Interviews, FGDs, and Community Mapping Design: Design and Test Messages, Instruments, and Monitoring Methods Data Collection: Collect Baseline Data Monitor Implementation of Program Collect Follow-up Data Analysis: Determine Impact and Outcome Interpretation: Share Findings with Stakeholders and Interpret Results Disseminate Findings

11. Process Research Summative Research Figure 1-2: Campaign Evaluation Framework Formative Research Activities Focus Groups Depth Interviews Secondary Analysis Participant Observation Activities Implementation Monitoring (viewer logs, broadcast schedule) Effects Monitoring (sales data, visitation data or interviews) Activities Analyze Survey Data Key Informant Interviews Objectives, Understand: Barriers to Action Appropriate Language Constellation of Factors Objectives, Understand: Frequency of Broadcasts Potential Audience Reach Preliminary Data on Effects Objectives, Understand: Level of Effect Degree of Efficiency Design Program Launch Program Program Ends

12. Formative - pre-intervention planning • To develop messages • Determine barriers to behavioral adoption • Determine baseline benchmarks

13. Process - intervention implementation • Determine the “reach & freq” • Make adjustments to implementation • Determine Impacts of program (esp. for long programs)

14. Summative - post-intervention impact analysis 1) determine impact 2) determine efficiency 3) future planning

15. Impact vs. Outcomes • Immediate impact • Long-term outcomes • Many programs designed to change outcomes in an overall model of behavior, but waiting until those outcomes are realized is unrealistic. • Hence indicators are developed to assess programs for immediate impact.

16. Evaluators must be versatile • Program changes • Evaluation changes (usually for the worst) • Programs and evaluations change • stakeholders change • the priorities and resources change • Program Stages - Evaluation should be tailored to the program stage such that more resources devoted to evaluating a pilot project or innovative program than an on-going one.

17. Chapter 2: Health Promotion Interventions (What Do We Evaluate) • Provider Training - educational materials such as manuals, flyers • CBD/CBO • Community Mobilization • Social Marketing • Mass Media Advertising - Promotional campaigns

18. Interventions (cont.) • Entertainment-education such as dramas, Soap operas, theater, film, music (songs, music videos, concerts) • Interactive Health Communication • Multi-media or community-wide Programs

19. Figure 2-1: Intervention Impact SpaceTradeoff Between Impact and Reach One-One Counseling Effective Programs Outreach Impact: Effect on Audience Street Theater community-wide programs Community Mobilization Soap Opera Ineffective Programs Mass Media Promotion Reach: Percent of Audience Exposed to Message

20. Tailoring • Objectives • Program Stage • Impact Models

21. Conditions Needed to Show Impact (4) • Situational: Population can adopt behavior Promoting behaviors that cannot be adopted is frustrating Be sure creating a health promotion program, be sure it is appropriate

22. Conditions for impact (cont.) • Programmatic: Program delivers appropriate message, appropriately 1) reach 2) appeal 3) tailored 4) appropriate channels 5) messages match needs 6) coordinate supply

23. Programmatic conditions (cont) 7) logos & slogans 8) source credibility 9) source trustworthy 10) replication 11) call to action 12) continuous involvement

24. Programmatic conditions (cont.) 13) Maximize IPC 14) Feedback 15) Promise a reward 16) Use fear cautiously

25. Impact Conditions (cont.) • Theoretical: Researchers and programmers understand the population and have an articulated theory of behavior change • Methodological: Researchers appropriately measured population characteristics, mediators and outcomes

26. Chapter 3: Behavior Change Theory Theories • Diffusion of Innovations • Hierarchy of Effects • Attitudes Intentions and Beliefs • Social Learning Theory (Self-Efficacy) • Health Belief Model • PRECEDE/PROCEED Perspectives • Networks • Ecological Perspectives • Social Marketing • Framework vs. Theories

27. Theories Diffusion of Innovations • "Diffusion is the process by which an innovation is communicated through certain channels over time among the members of a social system” (Rogers, 1995). • Innovativeness • Stages: Knowledge, Persuasion, Decision • Implementation, Confirmation • Characteristics of the Innovation (Radical vs. Continuous; Perceived attributes) • Mathematical Curves and Properties

28. Figure 3-1: Typical Diffusion of Innovations Curves

29. Figure 3-2: Knowledge Attitude and Practices Curves

30. Figure 3-3: Campaigns/Programs Can Accelerate Behavior Change

31. More Theories Hierarchy of Effects • Expanded diffusion stages to 12/16 steps to behavior change that are more measurable Posits rate of progress through steps Provides expected rates for each step (for calculating expected effects)

32. Hierarchy Model

33. More Theories Prochaska’s Stage Model • Specifies cognitive behavioral change stages Attitudes Intentions Motivations and Beliefs Social Learning Theory and Self-efficacy • Bandura (1985) individuals imitate models who are similar to the individual and likeable, and the behavior must be demonstrated and then rewarded

34. More Theories Health Belief Model • HBM states that individuals change health-related behaviors if their perceptions of: Risk, Severity, Susceptibility, Cost-effectiveness, Cues) PRECEDE • presents categories of factors that inhibit and facilitate behavior change: PREdisposing, Enabling, Reinforcing

35. Theories, Frameworks, Theories • Frameworks guide implementation of programs and describes steps in the process. • Theories explain behavior change and predict human behavior- they explain how and why change occurs • Models are particular embodiments of theories • Hypotheses are testable statements derived from theories or models.

36. Frameworks/Theories/Models (2) • Thus, the framework guides program steps, the logic model of what is going to happen • The theory explains why or how you expect the behavior to occur • The model depicts the relationships expected between key variables.

37. Perspectives • Perspectives are points of view that direct programmers and researchers to variables or activities that affect the program, but • They do not provide a proscribed prediction on behavior change, rather they • Specify variables or activities that affect the outcome and should be included in the models

38. Perspectives Ecological • Ecological Perspective (emphasizes analysis and intervention at multiple levels: individual, interpersonal, institutional, community and public policy) • Individual behavior is conditioned on environmental factors • Ecological levels include interpersonal, family, community, organizational, societal, policy • Ecological levels may vary depending on behavior

39. Figure 3-4: Ecological Models Specify Multilevel Influences on Behavior Society Policy Community Interpersonal Individual

40. Perspectives (cont.) Social Network Analysis • Relations rather than attributes; and Structural positions are important • People have many different networks and they can both constrain or promote health behavior • A community of scholars (INSNA) has been formed to study social networks • Methods and techniques can be simple or quite complex

41. A Typical Social Network Data come from one class asking students to name others whom they think would make good leaders

42. Perspectives (cont.) • Social Marketing • “Why can’t we sell health like we sell soap?” • Product • Price • Place • Promotion

43. Supply vs. Demand Perspective • Behavior change and diffusion have both supply and demand characteristics: • Supply refers the components of a product or service that involve making the service product available for consumption. • For example, the supply-side of family planning services is the provision of contraceptive counseling and devices used to regulate fertility.

44. Supply vs. Demand • Demand refers to the characteristics of the consumer population in terms of their desires and preferences for the service or product. • For example, the degree that a given population chooses to use a contraceptive such as pill is their demand for the pill. • Supply and demand factors interact in a dynamic way

45. Figure 3-5: Supply and Demand Variables and Interventions Supply Demand Knowledge, attitudes, and preferences of the public Examples Target audience knows of the importance of vaccines Audience members believe that vaccines are important Provision, accessibility, and quality of services and products Examples Number of locations to get vaccines Politeness of trained service providers Integrated/comprehensive programs address both supply and demand

46. Use & Selection of Theory • Theory helps define behavior and specify the mechanism thought to influence it • Thus it provides the target concepts for the programs • Program design • Variable measurement • Setting Goals/objectives • Theory vs. program failures

47. Theory for Program Design • Theory can be used to design programs. • For example, if studies show that adolescents are most influenced by their peers, programs should use peers to implement the program • Theory says that people smoke because “it’s cool” programs should be designed to show that it’s not cool

48. Theory forDefining Setting Goals & Objectives • Goals are general statements about what is to be achieved (Decrease unwanted fertility, Reduce smoking, Increase seatbelt use) • Objectives are operational, specifications of goals and should be SMART: (Specific, Measurable, Achievable, Realistic, Timely)

49. Hints: 1. Use strong active verbs 2. State only one purpose or aim 3. Specify a single end-product or result 4. Specify timeframe Examples: • Reduce homelessness by 10 percent • Decrease fertility to from 3.6 to 3.1 • Reduce smoking by 35 percent

50. Absolute vs. Relative Objectives • Absolute objectives specify that everyone (or no one) will have the condition - Percentage Point Change • Relative objective establish some proportionate improvement - Percentage Change • Modifications to objectives are often necessary