Introduction to Program Evaluation: Theory and Practice

1. Introduction to Program Evaluation: Theory and Practice Huey T. Chen, PhD Professor, Department of Public Health Director, Center for Evaluation and Applied Research Mercer University Atlanta, Georgia, USA E-mail: chen_h@mercer.edu

2. Overview • History and basic concepts of program evaluation • Campbellian validity typology • Program theory and theory-driven evaluation • Chinese philosophy, politics, and evaluation

3. Evaluation Examples Chinese history: • God of Agriculture • Confucius Modern societies: • Cooking contest • Talent show • Beauty contest • Product evaluation • Personnel evaluation • Accounting …… Program evaluation

4. Brief History of Program Evaluation • President Lyndon Johnson, initiated “War on Poverty” and “Great Society” in 1960s • U.S. Government began to taking major responsibility of general welfare of its citizens • Decision makers requires these federal programs to be evaluated. • American Evaluation Association was established in 1986. • European countries also emphasizes evaluation. Swaziland …….

5. What is Program Evaluation? Systematically gather empirical information of an intervention program on what, who, how, and why questions for assessing a program’s planning, implementation, and/or outcomes in order to serve stakeholders’ program accountability and/or improvement purposes. Questions? Intervention programs: Education, public administration, public health, criminal justice, welfare, job safety, environmental policy, and so on. Stakeholders:

6. Example of an Intervention Program: A NGO-Based HIV Prevention Program in South China Background: HIV prevalence was high in southern boarder provinces. Injection drug users lacked knowledge and skills in preventing HIV and did not know their HIV status Intervention: HIV counseling and testing Target population: Injection drug users (IDUs) Implementing organization: Wuzhou Women’s Foundation, 2006-2007 Goals: Reducing IDUs’ needle/syringe sharing and high risk sexual behaviors

7. Basic Evaluation Concepts • Process evaluation and outcome evaluation • Responsiveness vs. objectivity • Internal evaluators vs. external evaluators • Formative evaluation vs. summative evaluation R. Stake: “When the cook tastes the soup, that is formative; when the guests taste the soup, that’s summative.” • An evaluation could be both formative and summative.

8. Basic Concepts (continued) Basic Evaluation Types • Program outcome • Intended outcomes • Unintended outcomes

9. Major Issues in Outcome Evaluation Campbellian validity typology Internal validity: Does an intervention affect the outcomes? External validity: Are the effects generalizable? Trade-off between internal and external validity Prime priority of internal validity in research or evaluation Threats to internal validity: Contaminations, rival hypotheses, confounded factors

10. Threats to Internal Validity in Research Designs Pre-experimental designs One-group pretest and posttest design O1 X O2 Threats to internal validity (sources of contamination) • Maturation • Alternative historical events • Regression toward the mean (participants are extremer) • Testing • Instrumentation • Attrition

11. Threats to Internal Validity in Research Designs Quasi-Experimental Designs: • Nonequivalent comparison group design Intervention G: O1 X O2 _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ Comparison G: O1 O2 *Threats to internal validity: Self-selection bias • Interrupted time-series design O OOOOO X O OOOOO *Threats to internal validity: alternative historical events

12. Threats to Internal Validity in Research Designs Randomized controlled trials (RCTs) R Intervention G: O1 X O2 R Control G: O1 O2 R: Random assignment O: Observation X: Intervention * Maximizing internal validity

13. Campbellian Validity Typology Rigor in research designs • Best designs: RCTs or other experiments (pure effects) • Next to the best: Quasi-experiments (some contaminations) • Weak designs: Pre-experiments (many contaminations) Evidence-based interventions Interventions with evidence of RCTs. The popularity of the typology • The typology has been introduced in every evaluation text books.

14. Program Theory and Theory-Driven Evaluation Limitations of the traditional evaluation Black box evaluation Intervention Outcome Method-Driven Evaluation No conceptual framework is needed in doing evaluation Limitations Provide no information on how and why Provide no insightful information for understanding or improving a program

15. Program Theory and Theory-Driven Evaluation Chen (2005): A set of stakeholders’ Implicit and explicit assumptions on what actions are required to solve a problem and why the problem will respond to the actions. Theory-driven evaluation: Using program theory to guide evaluation design and practice

16. Illustrate the importance of program theory In order to encourage police officers to actively patrol streets and fight crime, the police chief of a major city announced a new policy using mileage shown on odometers to measure police officers’ performance. His change model: New Policy Increase Patrols (as measured by odometers) Reduce Crimes

17. Action Model PROGRAM THEORY Associate organizations and community partners Intervention and service delivery protocols Implementing organizations Ecological context Target populations Implementers Change Model Intervention Determinants Outcomes

18. Example of Assessing a Change Model Petrified Forest National Park’s perseveration program Placed signs to make tourists aware of preservation efforts of keeping the Petrified Forest intact: “Your heritage is being vandalized everyday by theft losses of petrified wood of 14 tons a year, mostly a small piece at a time” Signs Increased Enhancing Reducing awareness tourists’ theft of of a wide-spread morality for petrified problem protecting wood heritage

19. Example of Assessing a Action Model: Evaluating a School-Based Anti-Drug Abuse Program in Taiwan Drug abuse among middle school students had worsened The Ministry of Education launched a national anti-drug abuse program to deal with the problem Teachers were trained to identify students abusing drugs and provide counseling Schools were required to file monthly reports on the number of active drug abusing students to the ministry

20. # of active cases *3850 *1625 *501 *440 *374 *260 *353 *55 1 2 3 yr

21. Theory-Driven Process EvaluationApplication Procedures • Conducted working group meetings with key officials at the Ministry of Education to develop an action model • Conducted working group meetings with representatives of teachers to develop their version of the action model • Combined both groups (key officials, teachers) to create a new version for feedback • Used mixed methods (site visits, survey, participant observation, focus group meetings, interviews, record checking) to collect implementation data

22. Action Model as Planned vs. as Implemented

23. Action Model (cont)

24. Action Model (cont)

25. Action Model (cont)

26. # of active cases *3850 *1625 *501 *440 *374 *260 *353 *55 1 2 3 yr

27. New Developments in Outcome Evaluation Integrated evaluation perspective: Effectuality Viability Transferability Viability evaluation Assess whether an intervention is viable in the real world

28. Viability Evaluation: A New Type of Real-world Evaluation Assessing an intervention’s: • Practicality • Suitability • Affordability • Evaluability • Helpfulness: Real-world effects (quantitative and qualitative evidence)

29. Example of an Intervention Program: A NGO-Based HIV Prevention Program in South China Background: HIV prevalence was high in southern boarder provinces. Injection drug users lacked knowledge and skills in preventing HIV and did not know their HIV status Intervention: HIV counseling and testing Target population: Injection drug users (IDUs) Implementing organization: Wuzhou Women’s Foundation, 2006-2007 Goals: Reducing IDUs’ needle/syringe sharing and high risk sexual behaviors

30. Example of Viability Evaluation Wuzhou HIV prevention program Suitability: Women’s Federationwas able to manage the project well.They recruited 226 IDUs in four months. Practicality: WF staff was able to deliver the services well, butthe protocol required slight modifications. Affordability: HIV testing kits Evaluability: Procedures and outcome were well-established and measurable.

31. Viability Evaluation (continued) Helpfulness: Pretest-posttest data showed IDUs substantially: Increasing condom use reducing needle/syringes sharing Qualitative data showed: Clients were satisfied with the services and expressed their reduction of HIV risks Evaluation findings: • Viability: High • Transferability: Strong leadership required

32. Chinese Philosophy, Politics, and Evaluation in Chinese Society • Chinese philosophy, moral courage, and intervention • Moral courage and evaluation • What kinds of political contexts or moral courage disfavor or favor in conducting evaluation?

33. Conclusions • The state of the art of program evaluation: Blooming, but beginning a paradigm shift. • Theory-driven evaluation and the integrated evaluation perspective will play a key role in the new development. • Program evaluation is useful for China’s efforts in further improving people’s well-being.

34. Questions and Answers