Evaluation of a Statewide Prevention Program: An Illustration of a Research-Practitioner Model

1. Evaluation of a Statewide Prevention Program: An Illustration of a Research-Practitioner Model Marcia Hughes, Ph.D. Tim Black, Ph.D. Center for Social Research University of Hartford

2. Purpose • Show usefulness and importance of bringing evaluation research and practice together in what we refer to as a research-practitioner model. • Using the stages of research and program development of a statewide home visitation model in Connecticut, will illustrate how well-designed research has been used to inform and refine program practices.

3. Overview of the talk • Why use a research-practitioner model? • Connecticut’s Home Visitation Program: chronology & evolution • Highlights of findings from national research • Highlights of research-practitioner stages of research and program development • NFN standing in comparison with national models/research • Conclusions

4. Why Use a Research-Practitioner Model? • Increasing awareness of overall gap between research and practice: different cultural practices, different rules of evidence, distinctive languages. • Open-ended, complex problems vs. closed-systems problems. • Evaluation Research that is communicated to and used by practitioners and policymakers. • Research-practitioner model: Conduct meaningful evaluation work that informs program practices, strengthens organizational framework, and drives stages of program development.

5. Healthy Families CT (1995) Healthy Families Initiative (2000) Nurturing Families Network (2003) Connecticut’s Home Visitation Program: chronology & evolution • The charge of Children’s Trust Fund: Prevention • Adopted a National model: Healthy Families America • From 2 sites in 1995 to 42 sites in every geographic region • Urban focus: Hartford (2005) & New Haven (2007) “go to scale” • Statewide infrastructure: All 29 birthing hospitals, ob-gyn clinics

6. Goals and Services of Home Visiting • Provides high-risk families intensive parent education and support in the home and also helps to link families with needed resources and assistance for up to five years. • Primary goal: prevent child abuse and neglect; ensure child safety and well being • Ensure child health and wellness • Improve parenting attitude and behavior and improve relationship between mother and child • Improvement in mother’s developmental trajectory and life course: education, employment, and self-sufficiency

7. Nurturing Connections Number of first time mothers screened each year

8. Nurturing Home Visiting Participation Program Participation by Year Since 1998

9. A statewide infrastructure Program sites are located in every geographic area of CT and in all 29 hospitals, pre-natal and ob-gyn clinics. 2005: Harford goes from 2 to 10 program sites 2007: New Haven goes from 2 to 8 program sites

10. Is NFN home visitation making a difference? Interpretation of Pre-Post Evaluation Design • Research over the past eleven years overall have yielded positive results. • A good job of identifying & recruiting a high-risk population • Making improvements in parenting capacities, attitudes & behaviors. • A good job of reducing child physical abuse • An excellent job of linking families to services in community • Mothers who remain in the program for one or two years often achieve educational and employment goals

11. Home Visiting at the National Level: Research (RCTs) show mixed results Interpretations of the research • Research design in evaluating programs (Olds, 2007) • Who delivers the service (Olds et al., 2002) • Program implementation and quality assurance (Duggan et al., 2007; Duggan et al., 2004) • Staff training & supervision (Gomby, 2007; Halperm 1993; Hiatt et al., 1997) • Who is being targeted (Dumont et al., 2008) • Programs need an explicit “change theory” (DeLarosa et al., 2005) • Change theory is not comprehensive enough. (Hebbeler et al., 2002) • Families w/ multiple problems (Chaffin, 2004; Duggan et al., Gomby, 2007

12. National Research on program impact on child abuse and neglect (Gomby, 2005) • Population being served (low SES, often young, first-time mothers with low coping skills at program entry) • Onset of services (prenatal, birth, families with children under 3 years of age) • Case management services and curriculum that targets specific issues and outcomes and includes parent-child activities • Intensity of services (consistently and frequently engage mothers and families) • Performance monitoring to ensure fidelity to the model • Staff training & supervision to address risk factors

13. Research-Practitioner Model Center for Social Research & Children’s Trust Fund • Pre-post design and analysis of outcome data • Process evaluations: interviews, focus groups, surveys & ethnographic field work • “Reflections on a program” (1996) • Cultural Broker Model (1998-2000) • Study Circles (2001) • Continuous Quality Improvement (2002 and ongoing) • Life Stories of Vulnerable Families in Connecticut (2002-2003) • Expanded analysis of child abuse & neglect reports (2004 and ongoing) • Urban Focus: neighborhood analysis (Hartford in 2005 and New Haven in 2007) • Research team Analysis of NFN in comparison with National models • Clinical trial of In-home Cognitive Behavioral Therapy (3-year study starting in 2009)

14. Bridging Research and Practitioners: researchers, practitioners, policymakers • Multiple Research Methods • Quarterly reports and annual site visits • Roles of project manager and data base managers: • Provide technical assistance to program staff • emphasize relationships with providers • Research Team Meetings • CQI Team

15. Process Evaluation of Home Visitation (Black & Markson, 2001) • Ethnographic analyses of program practices; developed & articulatedtheoretical rationale for a paraprofessional model. • Following needs were identified: • Protocols for conducting home visits • Organizational channels for home visitors to communicate perspectives on family needs and to participate in statewide decision-making • Strategies for addressing specific challenges to delivering home visiting services (3 malleable risk factors) • Strategies for effective supervision

16. Study Circles (Diehl, 2001) • Participatory Action Study: promote dialogue among practitioners; discover common voices, and propose solutions to practice and policy concerns identified in the process evaluation. • Outcome of study circles: • Changes and additions to staff training. • Changes to parenting education curricula. • Changes in organizational structure of sites (solidifying the role of the clinical supervisor • Development of the Continuous Quality Improvement (CQI) team with broad representation that includes staff at all levels.

17. Cultural Broker Model (Black & Markson, 2001) • Interaction between home visitor & family member is the heart of the program: • Success is often dependent on personal and cultural experience with the population she is serving • Two important patterns to identification. • Professionalism is embraced by the home visitors • Home visitors reject professionalism as a viable role.

18. Quality of clinical supervision • Interaction between home visitor & clinical supervisor is equally important • In order for home visitor to buy into the professional model, clinical supervisor has to be willing to learn from the home visitor’s experience and knowledge of the families and communities • Role of clinical supervisor is also articulated and included as central component to paraprofessional model: • Managing feelings and reactions to families • Frequency and scheduling of supervision • Joint home visits • Group supervision • Professional development/education

19. Continuous Quality Improvement Team (2002 and ongoing) Major Policy that have been developed • Creative Outreach Policy • Family Engagement • Looking for the Little Steps • Clinical Supervision • Length and Content of Visits • Working with Significant Others • Families with Cognitive Delays • Families with Multiple Children Policy • Families with Acute Problems Policy • Professional Development Policy

20. Life Stories of Vulnerable Families in Connecticut (Black, Erdmans, & Dickinson, 2004) • Typologies of families receiving services:Analysis of 171 interviews with program participants conducted in 2002 and 2003 identified patterns of vulnerability among mothers in Connecticut who were identified as at risk for poor parenting. • Roles of the Home Visitor: Mothers described their program experiences and perspectives.They articulated what they valued most about their relationships with their home visitors.

21. Connecticut’s Vulnerable FamiliesN=171

22. Roles of the Home Visitor Life story interviews with NFN mothers in 2002 and 2003 provided an opportunity for mothers to describe their program experiences and perspectives. They articulated what they valued most about their relationships with their home visitors.   • Home Visitor as “Baby Expert” • Home Visitor as Advocate • Home Visitor as Friend • Home Visitor as “fictive kin”

23. Annualized Rates of Abuse and Neglect Annualized Rates of Child Maltreatment in the NFN Program

24. Expanded analysis of child abuse & neglect reports (2004 and ongoing) • Analyzed variation in rates between program sites (Damboise & Black, 2004) • Compared maltreatment rates with rates reported in studies of similar high-risk mothers not receiving home visitation (Murphy et al., 1985; Stevens-Simones et al., 2001) • Compared rates with rates documented by research on similar home visitation programs across the country (range from 1% -8%) • Analysis of reports: issues surrounding domestic violence, substance use are prevalent among perpetrators, as well as high prevalence of a parent with mental illness, and cognitive delay

25. NFN Compared at National level √ • Who is delivering services √ • Program implementation and quality assurance √ • Staff training and supervision √ • Who is being targeted √ • Programs need an explicit “change theory” √ • Problem is the change theory itself √ • Home visiting is not addressing the Big 3 ? • Research design

26. Conclusions and some of the challenges • Program change and development largely resulted from carefully designed research. • Changes and new initiatives often addressed/mirrored critical areas identified in national research. • Research-practitioner model lead to increased public awareness, funding and support for program expansion • Now ready to do more rigorous evaluation: RCT • Challenges: limits due to state system; don’t always get cooperation from program staff; learn to be accepting of new language; requires program leaders be tolerant at same time are vulnerable