Making it Last … The Marriage of Social Marketing and Evaluation

1. Making it Last…The Marriage of Social Marketing and Evaluation Social Marketing and Evaluation Webinar for the 2006 Newly Funded CommunitiesJuly 10, 2007

2. Social Marketers Are From Mars Evaluators Are From Venus

3. Social Marketers Are From Mars… • Like simplicity • Comfortable with generalities • Goal-oriented • Like to see quick results • Prefer short, straightforward presentations • Risk-takers

4. Evaluators Are From Venus • Like complexity • Deal in absolutes • Process-oriented • Like to take time, explore all angles • Prefer lengthy, abstract reports • Risk-averse

5. Mars and Venus Are Both Planets in the Same Universe Want to help children: • Stay in school • Stay at home • Become more productive members of society at a cost that helps sustain programs in communities

6. Challenges and Barriers to a Happy, Productive Relationship • Evaluators • Want to focus on long-term outcomes • Want data communicated accurately • Uncomfortable passing along early results Social Marketers • Want interim data to report • Want easy to communicate info • Have difficulty interpreting/ identifying alone

7. Why Cooperate At All? • Presentation of outcomes is key to sustainability • Increase the public’s knowledge and awareness of an issue, problem or solution • Influence perceptions, beliefs and attitudes that may change social norms • Increase your organization’s visibility as a resource on children’s mental health • Enhance your organization’s leadership position • Strengthen relationships • Form a united front for projecting and reinforcing messages • Create policy change • Increase demand or support for community-based, youth/family-driven, and culturally competent mental health services • Truly make a difference in the lives of children and families in your community

8. Why Cooperate At All? • Evaluators have the information: • Descriptive/Outcome Data • Local Wraparound Fidelity Index • Service Experience Data • System of Care Assessment • Data Profile Reports • Continuous Quality Improvement Reports

9. Why Cooperate At All? • Social marketers know how to present it: • Audience Research • Communication Strategies and Tactics • Packaging Data

10. Can This Marriage Be Saved?

11. Understanding Is the Key • What are roles? • What is the language? • What are the desired outcomes?

12. Role of the Evaluator • Understand program goals • Analyze ongoing/post-program needs • Design evaluation that captures pertinent data • Conduct evaluation; refine where necessary • Monitor other data sources; compare and contrast • Communicate regularly with staff across program, particularly social marketers • Report interim and final results • Empower families and youth to be persuasive advocates

13. Role of the Social Marketer • Understand program goals • Devise strategic communications plan • Identify and profile key audiences • Monitor changes and trends in attitudes and behavior • Share challenges and successes with audiences • Revisit and refine plan • Communicate regularly with staff across program, particularly evaluators

14. Role of Family and Youth • Distinguish if the data reflects their community • Help social marketers package the data, making it accessible and relevant • Inform message development • Complement data products with descriptive testimonials • Prevent dissemination of potentially misleading data • Provide the bridge from numbers to lives • Promote retention and recruitment for studies

15. What Can You Offer Each Other? Social Marketers • Understanding of audience needs • Suggestions for simpler language • Clearer communication of statistical info • Evaluators • Accurate, timely data on varied topics • Interpretation of results • Monitoring and analysis of data from outside sources

16. If We’re From Different Planets, Do We Need an Interpreter? Social Marketers • Audience segmentation • Editorial board • Actuality • Evaluators • Informed consent • Program theory model • Longitudinal statistical analysis

17. Breaking Down the Language Barrier • Eliminate jargon where possible • Define terms • Ask questions

18. Making Beautiful Music Together

19. National Example • Collaboration between the National Evaluation Team and the Children’s Campaign has resulted in many products including: • Fact sheets • PowerPoint presentations • Awareness Day data

20. Challenges • To balance audience-friendly language with accurate data

21. Lessons Learned • Dialogue can start from either team • Begin working together at the very beginning of development • Collaborate on the development of a prototype • Create a timetable which includes time for multiple revisions • Learn about the other team's development process • Pose evaluation questions several different ways for better understanding • Keep the dialogue going!

22. Questions?Please press *7 on your phone to unmute your line. When you are finished, please press *6 to remute.

23. Local Example: Family Voices Network of Erie County • We are in our 3rd Year – grant funded in October 2004. • We have an informal process for communicating with evaluation– we’re located in the same building and maintain regular contact with the families. Sometimes this is a dual role between social marketing and evaluation staff. • We share data with our parents and youth.

24. Family Voices Network of Erie County:SocialMarketing & Evaluation Team • Many times the social marketer and the evaluator wear dual hats. • We review data with our parents and youth and listen to their suggestions on how to make data family-friendly. • We inform the policy-makers of our findings based on the data collected bi-weekly at our Management Team meetings.

25. Family Voices Network of Erie County:Project Samples • Community Assessment for Family Involvement: • Knowledge & Effectiveness Survey • Children’s Mental Health Awareness Day: • NCMH Awareness Survey • Wraparound Bulletin: • Our quarterly newsletter

26. Family Voices Network of Erie County:Project #1: Knowledge and Effectiveness Survey • The 3 purposes of our Community Survey: • The purpose of this survey is to identify community knowledge and attitudes and to provide marketing assessment and direction including stigma evaluation. • The social marketing team presented the concept to the evaluation team and they worked together to develop a survey that would serve as a baseline of where our community stands today. By doing so, we have a measuring point to see improvements over time. • Between the evaluation team and the social marketing team, we identified our audience to be surveyed and worked together on identifying what we wanted to measure.

27. Family Voices Network of Erie County:Project #1: Knowledge and Effectiveness Survey

28. Family Voices Network of Erie County:Project #1: Knowledge and Effectiveness Survey • We then rolled-up the data into four categories: • Families • Care Coordinators • Supervisors • Service Providers (vendor network)

29. Family Voices Network of Erie County:Project #1: Knowledge and Effectiveness Survey • We asked our families to tell us what was important to them from the survey categories. They identified: • Wraparound process/care coordination • The family role • Moving from system to natural supports • Making a referral • Child & Family Team (CFT) process

30. Family Voices Network of Erie County:Project #1: Knowledge and Effectiveness Survey • Next Steps: • We will work with the evaluation team to create communications plans that address the needs of the family in the priority order they have identified. • The evaluation team will provide us with other data from care management and other survey to assist in our plan development. • You can view the survey form on our Web site:http://www.familyvoicesnetwork.org//en/files/ • Forms/Knowledge-Effectiveness-Survey.pdf

31. Family Voices Network of Erie County:Project #1: Knowledge and Effectiveness Survey

32. Local Example: Monterey County’s System of Care • Mostly agricultural county, in many areas 60-75% Hispanic • Lead Agency: Monterey County Health Department, Children’s Behavioral Health (about 125 staff) • Social marketing and evaluation are led by outside consultants

33. To achieve optimal outcomes from social marketing & evaluation efforts both need to be recognized as integral parts of all other SOC activities. And ensures that programs have achievable outcome measures & a clear identity & communication plan. Ensures that social marketing activities have a built-in evaluation component; This allows the social marketer to influence data collection from a creative and audience perspective; Program staff, social marketing, evaluation, families, youth & collaborating agencies work together from the very beginning – starting with the development of the social marketing plan, program logic models and design of outcome measures.

34. Monterey County’s System of Care:Strategy One: Bring Everyone to the Table Family & Youth Behavioral Health Probation Data Management Cultural Competency Social Marketing Evaluation Social Services The Local Evaluation Team Members of the Local Evaluation Team bring diversity & expertise to the monthly and “as needed” work sessions.

35. Monterey County’s System of Care:Strategy Two: Develop a Local Evaluation Framework • 13 local evaluation measures • Developed local survey with families and youth • State and national data • Program specific outcomes • Data from partnering agencies How do we bring partners together to discuss shared outcomes? Monterey County utilizes a Comparison of Results Approach. Seeks to answer the questions: How are we doing? Have outcomes for families and youth improved this year in comparison to last year?

36. Monterey County’s System of Care:Strategy Three: Develop a “Content Matrix” to Share Evaluation Findings

37. Monterey County’s System of Care:Example: Bilingual Newsletter Reaching Families

38. Monterey County’s System of Care:Strategy Four: Develop a Distribution Plan and Implement it!

39. Monterey County’s System of Care:Example: Score Card

40. Monterey County’s System of Care:Score Card 2005- At a Glance • Purpose:To highlight system-wide (shared) accomplishments of Monterey County’s System of Care in a user-friendly way. To show why systems of care are important to our community and why it is important to continue to fund them. • Primary Target Audience:Policy-makers, Board of Supervisors, Children’s Council, Governance Council, Community Stakeholders and Staff(Language: Eng) • Distribution:Briefing to Children’s Council and media on Children’s Mental Health Summit Day; Mailing

41. Monterey County’s System of Care:Summary of Strategies • Strategy One:Bring everyone to the table • Strategy Two:Develop a local evaluation framework • Strategy Three:Develop a content matrix to share evaluation findings • Strategy Four:Develop a distribution plan to share findings and implement it! • Strategy Five:Ask for feedback and actively involve new and existing partners in the process

42. Monterey County’s System of Care:Lessons Learned & Accomplishments • Scorecard is a visually appealing and useful tool to explain SOC values and principles. • Highlighting of shared programs and accomplishments generated goodwill – and momentum (many new partners this year!) Also- triggered more collaboration in other areas! • Ongoing struggle – who is our target audience? Translate into Spanish? Weighing cost against level of distribution. • Need to make the process youth- and family friendly – training. Goal in Monterey County: Bring more youth to the table. Key lesson learned to new communities (from a social marketing perspective):Have the social marketer involved early on in the development stages of the evaluation framework and survey tool. The social marketer can provide input into how questions and outcomes can be translated into meaningful data to varies target audiences.

43. Making It Last • Have early conversations about goals/objectives • Inform each other of progress/challenges • Be a resource for trends, new developments, community news • Present information together for clarity and accuracy

44. Resources • Caring for Every Child's Mental Health Campaign www.mentalhealth.samhsa.gov/child/default.asp(Features an order form for Children's Campaign products such as Children’s Coloring Books, Annual Reports to Congress, etc.) • Subscribe to the listserve 4-THE-CHILD subscribe@yahoogroups.com • Family Leadership in Systems Evaluation Tip Sheets • Engaging New Families in Evaluation & Language Qualifying Family Preference in Hiring • Federation of Families for Children’s Mental Health • The World of Evaluation: Three Short Courses

45. Questions?Please press *7 on your phone to unmute your line. When you are finished, please press *6 to remute.

46. Presenter email addresses: • Deborah Porter, deborahaporter@wobblee.com • Lisa Rubenstein, lisa.rubenstein@samhsa.hhs.gov • Jana Sczersputowski, jana@yoursocialmarketer.com • Tisha Tucker, alyce.l.tucker@macrointernational.com