The Importance of Assessing Program Implementation Fidelity

1. The Importance of Assessing Program Implementation Fidelity Finn-Aage Esbensen, Ph.D. University of Missouri-St. Louis Law Enforcement Planners Meeting St. Louis, MO September 2009 This research is funded by the National Institute of Justice, Office of Justice Programs, U.S. Department of Justice, Award # 2003-JN-FX-0003 (October 2003 – June 2009) and Award # 2006-JV-FX-0011 (July 1, 2006 – December 2012). The opinions, findings, and conclusions or recommendations expressed in this presentation are those of the author and do not necessarily reflect the views of the Department of Justice.

2. Overview • Comprehensive process evaluations are essential components of outcome evaluations. • How and why to assess fidelity of program implementation.  • Results from two NIJ funded evaluations will be described. • Both programs are law enforcement, in-school prevention programs.

3. Program Implementation • Program Fidelity • Dosage- is enough of the program actually implemented to expect the anticipated effect. • Program Adherence- to what extent are program components actually taught or delivered. • Quality of Delivery- if delivered, what is the quality of the delivery?

4. Why Document Program Fidelity? • It Compliments Outcome Analyses: • One can confidently tie outcome effects to the program/policy under study. - Or – • When no effect is found • (1) the program, as designed and implemented, has no effect • Often Assumed • (2) the program, as designed, is not well suited for the given setting • (3) implementation failure

5. Research on Program Fidelity • Research to date: “about half of school-based prevention activities are of such poor quality that they cannot reasonably be expected to make a difference…” (Gottfredson et al., 2000: 7). • Low levels of program effect …are more likely attributable to improper implementation practices than inept program content (DuBois et al. 2002…)

6. Common Hurdles to Program Fidelity • The idea that “something is better than nothing.” • Reality: Half-hearted implementation of programs designed for high-risk youth fair no better than no implementation at all (Kovaleski et al., 1999). • Insufficient “buy in” on the part of providers. • Effect: Program providers deviate from the program, and administer the lesson(s) differently than intended (Ruiz-Primo, 2005).

7. Case Studies • Teens, Crime, and the Community and Community Works (CW) • National Crime Prevention Council • Victimization reduction program • G.R.E.A.T. – Gang Resistance Education and Training • Bureau of Justice Assistance • Gang prevention program

8. Common Program Characteristics • Utilize law enforcement officers (or probation officers) • School-based and general prevention model • Incorporate interactive and active learning strategies • Emphasize skills building • Require implementers to be trained • CW is a two-day session • G.R.E.A.T. is one week for experienced officers or two weeks for officers who have not previously taught

9. Program components • Skill-based curriculum consisting of: • A 31-Lesson Curriculum • Core program consists of 11 lessons that, if taught with fidelity, requires approximately 30 hours of class time • The use of community resource people (CRP) to deliver the program • A group service learning project, referred to as Action Projects (AP) • Taught by teacher &/or SRO/JPO

10. CW Program Goals • Increase teen awareness of their vulnerability to crime • Educate teens on how to protect themselves and their communities • Motivate teens to take action in their communities • Help teens understand the costs of crime and what can be done to prevent it • Foster community relations by bringing community resource people into the sessions • Bond youth to the community through increased self-esteem • Increase teen empathy toward crime victims

11. G.R.E.A.T. Program • Taught by uniformed law enforcement officer • Skills building program • 13 lessons that take 13-14 hours to complete • Goal-setting; Decision-making; Communication skills; Conflict resolution • Student workbook • G.R.E.A.T. project

12. G.R.E.A.T. Program Goals • Prevent youth from becoming involved in gangs, violence, and delinquency • Develop positive relationships between youth and law enforcement

13. Evaluation Designs • Similar design for evaluation of both projects • Purposive selection of cities (G.R.E.A.T.) or schools (CW) • Observation of trainings • Observation of program delivery • Surveys of teachers, officers, &/or implementers • Outcome component (longitudinal survey of students)

14. CW Site Selection • 15 middle school programs in 9 cities/towns in 4 states • 98 classes – 49 CW & 49 Comparison • 2 – 10 classes per school (half CW, half comparison) • 1,686 students • 34 to 229 students per school • 6th – 9th grade

15. G.R.E.A.T. Site Selection • 31 schools – • 4 to 6 in each of 7 cities (representative of districts) • 195 classes – • 102 G.R.E.A.T. & 93 Control • 3 - 12 classes in each school (half receive G.R.E.A.T.) • 24 - 35 classes per city • 3,820 Students • Between 67 and 186 students per school • 6th or 7th (5 schools) grade

16. Training: Research Questions • Are trainees adequately prepared to teach the program after completion of training? • Are trainees provided the opportunity to practice program delivery? • Are teaching strategies modeled during training? • Are trainers professional and knowledgeable? • Do trainers adhere to suggested time frames? • Is the time allotted to training adequate?

17. Community Works Training • Observed 7 training sessions (6 two-day, 1 one-day refresher training). • Typical Training Scenario • 3 trainers (1 primary, 2 assistants) • 25-30 trainees • Trainees represented a mixture of professions • (SROs, teachers, juvenile justice workers, …)

18. Community Works Training • Trainers • Appeared comfortable when speaking from a script. • When asked questions, trainers sometimes provided vague or inaccurate information • e.g., misdemeanor vs. felony • Emphasized the program was “adaptable and flexible” • Is it a program or a resource?

19. Community Works Training • Teaching Strategies • Training staff made use of many group activities and teaching strategies. • e.g., icebreakers were modeled frequently • Activities were seldom linked to program content, even though a clear link exists.

20. Community Works Training • Program Content • Training provided attendees with an overview of the program, as well as key components of the program • More time is needed to allow for more modeling of program delivery. • More time needs to be devoted to explaining the program as a whole, not just individual lessons • (e.g., skills building approach). • Little time was devoted to explaining the substantive content of the lessons. • (i.e., not all in attendance had a law background)

21. Community Works Training • Program Content (Continued) • Trainers did not seem to alter trainings according to their audience. • (e.g., English teachers may need more attention paid to substantive knowledge, while SROs may need more time devoted to teaching strategies.) • Trainers did not provide remedies to common problems faced when implementing the program (e.g., substitutes, short classes).

22. Community Works Training • Time Management • Posted timelines for trainings were seldom followed • (e.g., long lunch, leave early). • Demonstration of a 45 minute lesson lasted 90 minutes. • Coincidently, this lesson was never implemented correctly in the classroom, per our observations. • When trainees voiced concern over the time needed to implement the program, trainers stated that one can either “use” the curriculum or “implement” the program; both could be used effectively (Flexibility).

23. Community Works Training • Based on observation of training, what are your expectations for program implementation fidelity?

24. G.R.E.A.T. Training • Observed 7 G.R.E.A.T. Officer Trainings (GOTs) (5 one week sessions and 2 two week session) • Typical Training Scenario (8+ trainers) • Seminar Supervisor & logistical support from Regional Administrator and staff • 6 trainers (Team Leaders) • Gang Specialist and Education Specialist • 30 - 36 trainees • Trainees primarily local law enforcement officers

25. G.R.E.A.T. Training • Training consists of multiple teaching blocks, including: • Transition from law enforcement to prevention • Gangs • Adolescent development and instructional issues • Issues of G.R.E.A.T. concern • Team leader models each lesson (although a few lessons are taught as an overview)

26. G.R.E.A.T. Training • Team leaders typically adhere to training rules and structure (e.g., time frames) • Trainers stress importance of program adherence, including historical development and program revision, skills-based program in which each lesson builds on prior lessons • Trainers model lessons so that trainees have experience of observing different teaching styles and different personal approaches to teaching standard curriculum

27. G.R.E.A.T. Training • Team leaders are “debriefed” by Supervisor after each modeling session • Group discussion of lesson content and/or delivery issues • Trainers utilize multiple teaching strategies • Team leaders meet with team of 5 – 6 officers for more in-depth discussion of program

28. G.R.E.A.T. Training • Trainees (officers) are required to develop one lesson overview and present to group • Trainees are subsequently required to model their lesson to their team • Each trainee has experience with one lesson and observes five others modeled by their team members • Trainees have considerable exposure to program content in this manner

29. G.R.E.A.T. Training Concerns • Trainings sometimes adjust schedule to accommodate gang or education specialists – suggest consistency to enhance “flow” • Consider more emphasis on classroom management issues • Gang block has tendency to regress to discussion of stereotypical media images of gangs • Given observations of training, what are expectations for program implementation fidelity?

30. Observation of Program Delivery • Observed more than 100 CW classrooms • Observed more than 500 G.R.E.A.T. classrooms

31. SESSION 1 CHECKLIST Creating a Community Vision Part 1 (55 minute estimate) Instructions: Please place a check mark after each area that is covered in the session. Leave the line blank when certain areas are not covered. Also, please note the approximate time spent on each part and steps within each part. After the checklist is completed, please complete the subsequent qualitative session evaluation. Start Time: ________ Step A: Warm up (15 minute estimate) 1. Introduction and welcome _______ a. teens introduce themselves and state their favorite activity ______ b. name tags distributed and used _______ 2. Icebreaker: Shapes (kids with same shapes get together and form shape) ________ 3. Overview of program and today's session _________ a. teens put together puzzle messages (small group activity) _______ Actual time spent on Step A: ______ Step B: Creating a Community Vision (30 minute estimate) 1. Brainstorm what does it mean to be safe? _________ 2. Defining community ________ a. help teens to come up with a definition similar to this: "a group of people that have something important in common" ______ b. teens consider what makes a school a community ______ c. ask teens what things we have in common to make school a community _____ 3. Think about what makes a community safe and secure _______

32. CW Observation Results • Overall, out of the 110 class periods observed, only 18 (16%) lessons were found to be delivered in the manner intended by program developers. • Common Deviations • Skipping Steps in the Lesson • Poor Time Management • Covering Multiple Parts in One Class Period

33. CW Observation Results • The program, as designed, is not well suited for implementation in the given setting. • One school classified as high fidelity. • Three schools classified as medium fidelity. • Could conceivably link program outcome to program delivery • Eight schools classified as low fidelity. • If program effect found, it could not be attributed to program delivery.

34. G.R.E.A.T. Classroom Observations • Purpose: determine extent to which program is delivered as intended with regard to: • Dosage • Adherence to intended lessons • Quality of instruction • 518 observations of G.R.E.A.T. program delivery • 33 different officers • Each lesson observed at least 30 times

35. G.R.E.A.T. Observations • Time frame adherence • Did not appear to be a problem if there were no behavioral problems • When problems occurred, generally due to external factors (shortened day, substitute teacher, fire drill) • Topical area • Overall, officers covered all lessons

36. G.R.E.A.T. Observations • Lesson Adherence • Overall close adherence • Some officers skipped around, teaching components out of order • Overall quality • Generally good quality • External factors most notable in terms of affecting lesson quality (time of day, substitute teacher, some particularly “bad” classrooms)

37. G.R.E.A.T. Overall Program Fidelity • 27officers (80 classes) implemented with good fidelity (adhered to lesson components, topical areas, time frame, and “quality” of instruction) • 3officers (11 classes) had lowfidelity (implemented much of lesson but with low quality) • 3 officers (11 classes) failed to implement (did not cover lesson components and poor quality)

38. Implementer Surveys • A complimentary technique to address program fidelity is through implementer surveys. • Allows us to document lessons taught, CRP used, and how Action Projects were implemented. • Note:Research shows that this technique tends to overestimate degree of program fidelity.

39. CW Implementer Surveys • Impressions of the Curriculum • Overall positive assessment. • 82% agreed that the program raised awareness of programs that assist victims of crime. • Most agreed that the lessons were relevant. • 59% felt the materials were appealing to students. • This may signal that materials should be modified to make them more appealing.

40. CW Implementer Surveys • Program Fidelity • Programs were categorized as implementers and non-implementers based on the following standards: 1) At least 70% of the “core 11” lessons taught 2) At least 20 hours of program delivery time 3) CRP had to be used 4) AP had to be at least planned and initiated • Based upon the implementer surveys, 6 of the 14 schools could have been classified as having implemented the program.

41. G.R.E.A.T. Officer Survey • Survey of G.R.E.A.T. officers in 7 cities (anonymous questionnaires) • 137/205 = 67% response rate

42. G.R.E.A.T. Officer Attitudes • Program Goals (% agreeing or strongly agreeing) • G.R.E.A.T. teaches skills to avoid gangs and violence (80%) • G.R.E.A.T. has improved police/youth relations (90%) • G.R.E.A.T. strengthened police/community relations (90%) • G.R.E.A.T. has reduced community gang problem (29%) • G.R.E.A.T. has reduced community crime problem (29%)

43. G.R.E.A.T. Officer Attitudes • Program • G.R.E.A.T. is appropriate for age and comprehension levels (83%) • G.R.E.A.T. lessons address risk factors for gangs and delinquency (71%) • Enough time to cover important topics (72%)

44. G.R.E.A.T. School Personnel Survey • Purpose: obtain opinions about school-based prevention programs in general and G.R.E.A.T. in particular. • All G.R.E.A.T. grade-level teachers and administrators at 31 participating schools (n=231/373, 62%).

45. G.R.E.A.T. School Personnel • Program Assessment (% who agreed or strongly agreed) • 88% in favor of having G.R.E.A.T. in their school • 90% - curriculum age appropriate • 82% - gives students skills to avoid gangs and violence • 86% - improved students attitudes about police • 52% - reduced gang participation at school • 46% - reduced gang participation in community

46. G.R.E.A.T. School Personnel • G.R.E.A.T. Classroom Teachers • 80% - officer adequately trained in content • 70% - officer adequately trained in classroom management • 81% - students responsive to officer • 78% - officer punctual (or rescheduled if needed • 48% - officer incorporated teacher into lesson • 14% - officer strayed from lesson plan • 16% - officer had difficulty controlling class

47. G.R.E.A.T. Bottom Line • Generally favorable opinions expressed by officers and school personnel • However, it is noteworthy that both officers and teachers more positive about mediating effects than on program objectives

48. Summary Comments • Evidence-Based Prevention Programming has become a reality. • Funding, Adoption in Schools/Communities • Attention to program fidelity is necessary. • Tale of Two Programs • CW- Implementation Failure and No Program Effect • Is this a bad program, or one that was not implemented correctly? • With G.R.E.A.T., we have program fidelity and preliminary analyses indicate positive program effect.

49. Thank you for listening.