Consistent Protocol, Unique Sites: Seeking Cultural Competence in a Multisite Evaluation

1. Consistent Protocol, Unique Sites: Seeking Cultural Competence in a Multisite Evaluation Carolyn Sullins, Ph.D. Ladel Lewis, Ph.D. candidate The Kercher Center for Social Research Western Michigan University

2. National Study • Comprehensive Community Mental Health Services for Children and Their Families Program: “Systems of care.” • incorporates a broad, flexible array of effective services and supports for a defined, multi-system population that is organized into a coordinated network… is culturally and linguistically competent, builds meaningful partnerships with families and youth at service delivery, management and policy levels, and has supportive policy and management infrastructure. (Pires, Lazear, & Conlan, 2008).

3. Systems of Care • 144 sites have been or are in the process of being evaluated. Each 1-2 year cohort: +/- 30 sites. • Each SoC has distinct: • geographic location and scope (e.g., statewide, county wide, city-wide, tribal) • Ages of the youth served • Mental health issues facing the targeted youth • Racial, ethnic, and cultural factors

4. Core Values • Child centered and family driven • Community based • Culturally competent

5. Common Variables • Presenting issues of the youth • Youth’s level of functioning (strengths and weaknesses) • Family strengths and barriers • Types of services family and youth are receiving • Satisfaction with services • Cultural competence of services • Youth and family input into services

6. Barriers to Evaluation Implementation • “One size fits all” battery of questions • HSIRB mandates re language on consent forms • Ensuring an adequate sample size • Ensuring retention in a mobile population

7. Barriers to Participation • History of racist abuse by researchers (E.g., Tuskegee syphilis study) • Misinterpretation of data, or no access to results • Sensitive or stigmatized topics even more difficult  • Families overwhelmed or embarrassed

8. Local System of Care:“Kalamazoo Wraps” Ages 7-17 Diagnosed with a Severe Emotional Disturbance Caucasian: 49.1% (including White Latino/a: < 5%) African-American/Multiracial: 50.9%

9. Academic/Professional vs. Local Population Perspectives • Informed consent • Voluntary participation • Confidentiality and its exceptions

10. Evaluation Work Group Parents, various social service workers, eval staff • Reviewed consent forms for clarity • Gave opinions to HSIRB re child abuse reporting • Parent input re: communication among clinicians, families, and interviewers • Interpretation of data • Reporting of results

11. Youth Group • We came to their group • Help re local language • Helped us make it more comfortable for participants • Info that later helped us interpret data

12. Academic Culture vs. Local Communities’ Culture Does confidentiality mean… • No interviews in public places, even if that’s what participants request? • Kicking Grandma out of the room? • Pretending you don’t see participant in public?

13. Problems with Interviews • Some found it emotionally draining • Too long and redundant • Questionnaires with overlapping questions • Categories of services – national vs. local terms • Keeping in touch with families every 6 months

14. Easing Stressful Processes • “Evaluating System of Care – not you” • Yet up front about sensitive, personal questions • Non-judgmental attitude for better rapport, retention, AND accuracy. • Balance – we can’t act as friends or counselors! • If SoC not working, or not working with all groups of people, we need to know.

15. Interviewers: Pay Attention – So Participants Will Too! • Offer breaks, gum, stress balls, etc. to participants • Coloring books, DVDs for young kids • Redundant questions from multiple surveys: propriety and accuracy trump methodological “letter of law.”

16. Keeping in Touch with Families • Family address tracking form • Birthday and holiday cards with coupons • Incentives for families to contact us • Annual dinner as a “thank you” • Results in bimonthly newsletter, website, and other venues

17. Recruitment/Retention Results

18. Recruitment Rates (N=224) Chi Square (1, N=224) = .466, p = .495

19. 6 Month Interviews (N = 145) Chi Square= (1, N= 145) = .665, p = .415

20. 12 Month Interviews (N=119) Chi Square= (1, N=119) = .511, p = .561

21. 18 Month Interviews (N=91) Chi Square= (1, N=91) = .071, p = .834

22. 24 Month Interviews (N=52) Chi Square= (1, N=52) = 1.055 , p = .402

23. Conclusion • Fulfilling national evaluation requirements within a local context is a constant balancing act. • With enough input from local stakeholders at each stage, it can be achieved!

24. Questions? Comments? • Please contact carolyn.sullins@wmich.edu • Or Ladel_lewis@yahoo.com • For more info, please see also • http://www.wmich.edu/sociology/kzoowraps.html