No Place Like Home

1. No Place Like Home Cross-Site Evaluation Training

2. General Staff Survey • Please open [enter link] on your computers • Read the consent page, and if you agree, please click “Next” to begin the survey • It should take approximately 15 minutes to complete

3. NPLH Summary • Evaluation of three child welfare agencies’ use of family meetings with families receiving in-home services • Process • What does practice look like? • Tools: focus groups, observations, surveys • Outcomes • Placements (number and type), re-reports, recurrence of maltreatment, etc. • Satisfaction and service provision • Tools: administrative data, surveys

4. Your role… • Is critical! • To provide data and feedback about your experience working in child welfare and with family meetings • To recruit families to participate, so that they can provide data and feedback about their experience • To participate in the feedback loop – ongoing communication

5. What’s in it for…?

6. Human Subjects • What is the CITI program? • History and Ethical Principles • The Belmont Report

7. Human Subjects • Defining Research with Human Subjects • Federal Regulations • Assessing Risks • Basic Institutional Review Board (IRB) Regulations and Review Process

8. Human Subjects • Informed Consent • Risks to you personally are minimal • There are potential benefits to children, families, and to staff in the agency • Your participation is voluntary • Privacy and Confidentiality • All survey information is confidential and used only for the study • Survey data is maintained by Kempe Center NPLH evaluation team • Individually identifiable information will NOT be shared outside of the Kempe Center evaluation team

9. Human Subjects • Records-Based Research • Research with Protected Populations – Vulnerable Subjects • Unanticipated Problems and Reporting Requirements

10. Evaluation Basics • Evidence-Based Practice • Confidentiality vs. Anonymity • Bias • Design Types • Experimental • Quasi-Experimental • Process vs. Outcome evaluation • How vs. What • Qualitative vs. Quantitative methods • Depth vs. Breadth

11. Process Evaluation Outcome Evaluation Did the program meet its process objectives? What type and amount of services were provided? Can the program be causally linked to outcomes? Did the program meet its objectives and goals? Was the program implemented as planned? Did the program make a difference? Who was served? Generally, outcome evaluations use quantitative research methods and designs to establish causality (i.e., experimental* and quasi-experimental) Generally, process evaluations use qualitative research methods like interviews and focus groups. Typically, the best research designs combine both process and outcome evaluations and rely on mixed methods. A cost analysis links benefits to costs for comparison of effectiveness in policy-relevant terms. Example evaluation questions from: Handler, A. (2002). Role of evaluation in policy development and implementation. Retrieved from http://www.uic.edu/sph/mch/evaluation/class.ppt Process vs. Outcome Evaluation

12. True Experimental Design

13. Evaluation Design - TX Right now FBSS worker makes FGC referral FGC 1-2 months post-FGC FBSS Case closure/transfer to CVS

14. Evaluation Design - SD Right now IFA worker makes FGC or FGC/TDM referral FGC/TDM; IFA and Ongoing worker attend 1-2 months post-FGC Ongoing case closure/transfer to out-of-home services

15. Evaluation Design - CO Right now Intake or FAR worker makes 1st FSRT or FUM referral 2nd FUM/FGC 1-2 months post-FUM/FGC Ongoing case closure/transfer to out-of-home services

16. NPLH Surveys • General Staff Survey • Caregiver Survey • Meeting Fidelity Survey • Participant • Facilitator/Coordinator • Case-Specific Questionnaire • Follow-Up Survey • Caregiver – Intervention • Caregiver – Control • Meeting Participant

17. Surveys that are administered TO you • General Staff Survey • All staff who have a role in the evaluation (case workers, coordinators/facilitators, supervisors) • Coordinator-Facilitator Survey • Coordinators/Facilitators only • Case-Specific Questionnaire • Case-workers only

18. General Staff Survey

19. General Staff Survey • Who? You! • Case workers (FBSS, Ongoing…) providing in-home services • Family meeting facilitators/coordinators • Supervisors of both • When? Now • How? Via Survey Monkey

20. Coordinator-Facilitator Fidelity Survey

21. Coordinator-Facilitator Fidelity Survey • Who? Meeting coordinator/facilitator • When? After the family meeting (FUM/FGC/TDM) • How? Via Survey Monkey

22. Case-Specific Questionnaire

23. Case-Specific Questionnaire • Who? Case workers • When? At case closure OR transfer from in-home services to out-of-home services (foster care, kinship care, etc.) • Transfer does NOT mean voluntary placements • It does mean a formal or legal change in custody • How? Via Survey Monkey • We will send you a ‘tickler’ or reminder

24. Surveys that are distributed BY you • Caregiver Survey • By case workers • At face-to-face meetings with family • At point of referral for a family meeting • Participant Fidelity Survey • By coordinators/facilitators • At family meetings

25. What you ARE responsible for • Informed Consent • Will families agree to be part of the pilot? • If so, they have rights which are outlined in the Information Sheet, including the right to change their mind at any time. • If not, no penalty will come to them. • Distribute • Paper and pencil surveys with • CaregiverSurvey – self-addressed and stamped envelope with a flyer with a phone number to call for assistance • Meeting Participant – large self-addressed, stamped envelope for all surveys in the center of the room/table

26. What you are NOT responsible for • Explaining the content of surveys to families • Helping families/meeting participants fill out surveys • People with questions can call the number provided on the Information Sheet (have extra copies available for them to keep) • Other household members or family supports, etc. can help. • In order to maintain confidentiality, **

28. Caregiver Survey • Who? Parents or legal guardians • When? • Face-to-face meeting • At time of referral for family meeting • How? • Paper and pencil • Self-addressed and stamped envelope • Distributed by IFA/FBSS/FAR/Intake Caseworker

29. Participant Fidelity Survey

30. Participant Fidelity Survey • Who? Any meeting participant who agrees to fill it out • When? AT the family meeting • Section 1: Before the meeting starts • Section 2: After the meeting ends • How? Paper and Pencil • Meeting participants will fill out paper-pencil surveys and place them in a large envelope • The coordinator/facilitator will mail them to the evaluation team

31. Surveys administered by evaluation team • How? Paper and pencil; via mail • 3 follow-up survey versions with the following components: • Meeting Follow-Up • Who? All meeting participants who filled out the initial Meeting Participant Fidelity Survey • Caregiver Follow-Up • Who? All (intervention and control group) parents • Family Satisfaction • Who? All (intervention and control group) parents AND meeting participants

32. NPLH Survey Summary • Case workers: • Complete: • General Staff Survey (1 time) • Case Specific-Questionnaire (for every study case) • Distribute: • Caregiver Survey (to every study case) • Coordinators/Facilitators: • Complete: • General Staff Survey (1 time) • Coordinator-Facilitator Fidelity Survey (for every study family meeting) • Distribute: • Meeting Participant Fidelity Survey (for every study family meeting)

33. Coordinators/Facilitators Meeting Log • [placeholder]

34. THANK YOU! If, throughout the life of this project, you have any questions please contact: Heather Allan, NPLH Project Coordinator Email: heather.allan@childrenscolorado.org Phone: 720.336.8283