Being Participatory and Action-Oriented

1. Being Participatory and Action-Oriented Conducting Community Needs Assessments and Service Evaluations in Your Military Family Resource Centre Toronto Military Family Resource Centre June 23, 2004 Rich Janzen Maria Hatzipantelis Centre for Research and Education in Human Services

2. A Presentation By: Centre for Research and Education in Human Services 73 King St. West Suite 202 Kitchener, Ontario N2G 1A7 519.741.1318 www.crehs.on.ca • Evaluation • Applied Research • Training • Proposal Development • Facilitation and Planning • International Consulting Centre for Research and Education in Human Services

3. Purpose of this Session • To give a brief overview of the Good Practice and Resource Guide describing a participatory action approach to evaluation/ needs assessment. • To learn in more depth about methods of information gathering. Laying the Foundation Acting on Findings Research Planning Gathering and Analyzing Information Centre for Research and Education in Human Services

4. Session Overview • Introduction • Phase 1: Laying the Foundation • Phase 2: Research Planning • Phase 3: Gathering and Analyzing Information • Phase 4: Acting on Findings • Zooming in on Methods Centre for Research and Education in Human Services

5. Icebreaker • How experienced do you feel in carrying out evaluations/needs assessment? • How stressed are you when thinking about doing an evaluation/needs assessment in your Centre? • How meaningful have past evaluations/needs assessments been to you? very ………………………………………………….. not all INTRODUCTION Centre for Research and Education in Human Services

6. “It is good to have an end to journey towards, but it is the journey that matters in the end” INTRODUCTION - Segaki Centre for Research & Education in Human Services

7. Why this Guide? • Intended to help staff, board/advisory committee members and volunteers of your Canadian / Military Family Resource Centres (C/MFRCs) conduct community needs assessments and evaluation of services within your CF community. • A tool to help you be responsive to your community. • A tool to help you to be accountable to DMFS. INTRODUCTION Centre for Research and Education in Human Services

8. Who Helped Shape It? Advisory Group Peter Gabor DMFS Consultant University of Calgary in Lethbridge Laurie Johnson Member and Past-Chair Winnipeg Military Family Resource Centre Genella Macintyre Executive Director Shilo Military Family Resource Centre Major Rob Morrow Research Officer – Social Policy Directorate of Human Resources Research and Evaluation Pam Willis/Eva Marks MacIsaac Regional Representatives Directorate Military Family Services Alla Ivask Senior Manager, Policy & Program Development Directorate Military Family Services Maureen Kellerman Program Co-ordinator Directorate Military Family Services INTRODUCTION Centre for Research and Education in Human Services

9. Whom Did We Interview? To increase usefulness of the Guide and to provide examples of what others have done. • Calgary MFRC • Dundurn MFRC • Kingston MFRC • Centre de la famille Valcartier • Halifax MFRC INTRODUCTION Centre for Research and Education in Human Services

10. PHASE I STEPS PHASE IV STEPS 1. Clarifying Stakeholder Roles 2. Understanding the Benefits of Participatory Action Research 3. Describing Your Context 4. Determining the Purpose of Your Research Laying the Foundation 10. Sharing Your Findings 11. Implementing What You Learned Acting on Findings Research Planning PHASE II STEPS Gathering and Analyzing Information 5. Determining the Main Research Questions 6. Choosing Your Methods, Tools and Sampling Strategy 7. Planning Your Analysis Before You Begin PHASE III STEPS 8. Gathering Your Information 9. Analyzing Your Information How is the Guide Organized? INTRODUCTION Centre for Research and Education in Human Services

11. What’s Covered Within Each Step? • What does this step mean? • How do you do this step? • How do you do this step in your C/MFRC? • What else can help you? (MFRC stories and other evaluation resources) INTRODUCTION Centre for Research and Education in Human Services

12. Why Do Research? • Evaluation • Increase accountability to various stakeholders • Service planning and improvement • Marketing INTRODUCTION • Needs Assessment • Inform service planning • Increase accountability • Highlight opinions of residents Centre for Research and Education in Human Services

13. STEP #1 Clarifying Stakeholder Roles • Identify the groups that have a stake in the services and their evaluation/needs assessment. • Choose representatives from the stakeholder groups you identified to form a research steering committee. • Build a common understanding of expectations among steering committee members. • Ensure that meetings are accessible to all members. • Ensure that every member has an equal voice. • Create a way to include other people from the various stakeholder groups in the early planning phase. LAYING THE FOUNDATION Centre for Research and Education in Human Services

14. STEP #2Understanding the Benefits of Participatory Action Research • Explore people’s assumptions of what is good research. • Designate a portion of your first steering committee meeting to talk about and record what people see as the benefits of using a participatory action approach to research. • Constantly revisit this discussion as you plan and implement your evaluation/needs assessment. LAYING THE FOUNDATION Centre for Research and Education in Human Services

15. 3. Understanding Cause and Effect 4. Understanding Complex Social Dynamics Judge Quality Test Theory Provoke Change Mobilize the Marginalized The History of the Evaluator’sRole 2. Describing Interventions 1. Measuring Outcomes LAYING THE FOUNDATION In Order To …. Centre for Research & Education in Human Services

16. STEP #3Describing Your Context • Identify the people and other resources that will support and limit your research. • Review and become acquainted with all relevant service documentation and potential sources of information. • Identify the audience for the evaluation/needs assessment. • Visualize your services and their goals by developing a logic model. LAYING THE FOUNDATION Centre for Research and Education in Human Services

17. Key Questions A Logic Model Tries to Answer • What is it that we really do? • What do we hope changes because of our program? • How do we know that we’ve achieved what we set out to do? • Does it make sense? LAYING THE FOUNDATION Centre for Research & Education in Human Services Centre for Research & Education in Human Services

18. STEP #4Determining the Purpose of Your Research • Brainstorm with your steering committee what main achievement you hope the evaluation/needs assessment will accomplish. • Summarize your discussion by writing a one–two sentence purpose statement that all steering members can agree on. LAYING THE FOUNDATION Centre for Research and Education in Human Services

19. STEP #5 Determining the Main Research Questions • Brainstorm with your steering committee about the main questions that you hope your evaluation/needs assessment will answer once completed. • Summarize this discussion by writing a list of research questions that: 1) are clearly worded, 2) fulfill the evaluation purpose, 3) reflect the interests of all stakeholder groups, 4) provide useable information, and 5) are consistent with the MFSP service policies in Parameters for Practice. • Identify indicators that are relevant to your service(s) and linked to your research questions. RESEARCH PLANNING Centre for Research and Education in Human Services

20. Linking Purpose Statement with Research Questions and Indicators RESEARCH PLANNING Centre for Research and Education in Human Services

21. STEP #6Choosing Your Methods, Tools and Sampling/Recruitment Strategy • Understand the importance of “triangulation” (using multiple methods from multiple perspectives) • Review the different options that provide sources of information. • Chose and create your information gathering tools. • Develop a sampling and recruitment strategy. • Be serious about ethics. • Put it all together in a workplan. RESEARCH PLANNING Centre for Research and Education in Human Services

22. Research Design Planning Chart RESEARCH PLANNING Centre for Research & Education in Human Services

23. STEP #7Planning Your Analysis Before You Begin • Assess the soundness of your research plan. • Trust your original design and questions. • Assess the feasibility of various alternatives. • Build in flexibility – leave time for reflection. RESEARCH PLANNING Centre for Research and Education in Human Services

24. Warning… • 4 out of 5 statistics are misleading. RESEARCH PLANNING Centre for Research & Education in Human Services

25. STEP #8Gathering Your Information • Pre-test, pilot test and fine tune your research tools. • Look for ways to involve others. • Monitor your progress to ensure quality control. • Identify and deal with unexpected problems. GATEHRING/ANALYZING INFORMATION Centre for Research and Education in Human Services

26. STEP #9Analyzing Your Information • Organize the data in a way that makes sense. • Review the original research questions to give you a reference point in your analysis. • Begin summarizing and coding the data. • Identify overarching themes and relations between the various parts of your data. • Begin drafting your evaluation/needs assessment report. • Go out and gather feedback in order to verify your findings. GATEHRING/ANALYZING INFORMATION Centre for Research and Education in Human Services

27. STEP #10Sharing Your Findings • Write an evaluation report. • Consider alternative formats of sharing your findings. ACTING ON FINDINGS Centre for Research and Education in Human Services

28. STEP #11Implementing What You Learned • Consider tips for making evaluation/needs assessment findings more useful. • Discuss how you can create a culture of learning within your organization. ACTING ON FINDINGS Centre for Research and Education in Human Services

29. More About Methods Topics we’ll cover… • Appropriate mix of methods • Quantitative vs. qualitative methods • Designing a methodology • Doing interviews and focus groups • Ensuring trustworthy data collection ZOOMING IN ON METHODS Centre for Research and Education in Human Services

30. An Appropriate Mix of Methods Consider….. • A Balance of Breadth and Depth • A Balance of Exploration and Confirmation • Efficiency • Triangulation of Stakeholders and Methods • Accessibility ZOOMING IN ON METHODS Centre for Research & Education in Human Services

31. Quantitative vs. Qualitative Methods • Can use one to complement the other • Breadth versus depth • Depends on the question, resources available and your perspective on how deep and broad an understanding is needed ZOOMING IN ON METHODS Centre for Research & Education in Human Services

32. ZOOMING IN ON METHODS Centre for Research & Education in Human Services

33. Types of Quantitative Methods: • Discussion • What’s the “instrument?” • Pros and Cons • What have you used? ZOOMING IN ON METHODS Centre for Research & Education in Human Services

34. Types of Qualitative Methods: • Discussion • What’s the “instrument?” • Pros and Cons • What have you used? ZOOMING IN ON METHODS Centre for Research & Education in Human Services

35. Develop Methods to Answer This Research Question ... • Do Canadians identify more strongly with Tim Horton’s coffee or with hockey? ZOOMING IN ON METHODS Centre for Research & Education in Human Services

36. Interviews as a Method Why use an interview? Interviews are used when the people you want to gather information from seem unlikely to respond to a written survey, when the topic may be difficult or sensitive enough that a person may require encouragement to answer the questions or when the researcher is uncertain as to what is most important to the respondents. Interviews also provide the advantage of being able to follow-up on points made by the respondent. Interviews can be done with people individually, or in groups (focus groups). What is a focus group interview? Focus groups are used as a means to gather specific information from a small group discussion. The questions used to begin focus group discussions are usually open-ended questions. The discussion is usually amongst 7-10 people. Why use a focus group interview? Because focus group interviews provide an opportunity for people to respond to the ideas of others, they allow for richer and better thought-out ideas. Discussion tends to prompt participants to reflect more thoughtfully about the reasons for their opinions than they would otherwise. Focus groups also allow for the collection of a large amount of information in a short time. What is the role of the focus group leader? The focus group leader often acts as a research tool him/herself rather than using a survey or questionnaire as a research tool. The quality of the interview (and the data which will be later analyzed) depends on the quality of the interviewer’s listening skills and verbal and non-verbal behaviour. It is important to listen and observe very carefully, to be conscious of your own posture, to make eye contact and to acknowledge what people are saying. It is also the role of the focus group leader to ensure that participants feel safe enough to talk and stay on topic (while allowing for fluid conversation) and to probe for answers. At the same time, the focus group leader tends to remain in the background while the participants share ideas and opinions. ZOOMING IN ON METHODS Centre for Research & Education in Human Services

37. Tips for Doing Focus Groups Prior to Interview: • Recruit 10-12 people to participate in the focus group (send them a copy of the questions and consent form). • Call each participant one day prior to interview to remind them of the time and location. • On the day of the interview, set up refreshments, chairs, flip chart, markers and tape recorder. • Have all consent forms signed before you begin the questions. • Ask for the group’s permission to use the tape recorder. • Go over ground rules for the group. • Check tape recorder to ensure that it is working properly. • During the Interview: • Use the facilitator’s protocol as your guide. • Record answers on both the flip chart and on paper while you ask questions and facilitate. • Make sure that actual quotes will be recorded (and quotation marks used). After the Interview: • Thank participants (give them honouria). • Let participants know how they will receive feedback of the evaluation findings. • Label all notes and flipcharts with the date, type of interview and project name. • Send a thank you card a week later? • Invite participants to your feedback session. ZOOMING IN ON METHODS Centre for Research & Education in Human Services

38. Criteria for Judging the Trustworthiness of an Evaluation Trustworthiness is the expanded version of "reliability and validity checking" used in a quantitative evaluation Criteria include: • accurate recording in the field • prolonged engagement - knowing the context takes time • peer checks - members of a research team • debrief and challenge one another • saturation - qualitative information is rich and detailed enough to ensure that key themes have not been missed ZOOMING IN ON METHODS Centre for Research & Education in Human Services

39. Criteria for Judging the Trustworthiness of an Evaluation (continued) • Well established "audit trail" to track where you drew information from to reach conclusions • Triangulation - findings are supported by multiple methods and multiple stakeholders • Endorsement of participants - findings are fed back and validated by stakeholders • Clear program description and statement of objectives • Links between exploration and confirmation ZOOMING IN ON METHODS Centre for Research & Education in Human Services

40. THANK YOU ! Centre for Research and Education in Human Services 73 King Street West, Suite 202 Kitchener, Ontario N2G 1A7 Phone: (519) 741-1318 Fax: (519) 741-8262 E-mail: general@crehs.on.ca Web page: www.crehs.on.ca