How to Evaluate Community Projects Workshop Workshop Facilitators: Lisa Hall & Kathryn Crowther

1. How to Evaluate Community Projects Workshop Workshop Facilitators: Lisa Hall & Kathryn Crowther York Consulting LLP

2. What is Evaluation? • Many similar definitions and explanations of "what evaluation is“ • Each definition, and each evaluation is slightly different, • Common Elements: • Specific type of research activity – similar to applied research • Focus on practical implications • Extent to which aims are realised • Consider the effects of activity • Judgement of worth or value “….the systematic application of social research procedures for assessing the conceptualization, design, implementation, and utility of ... programs." Rossi and Freeman: Evaluation: A systematic approach, Newbury Park, Calif: Sage Publications, 1993

3. Why Evaluate? Benefits of Evaluation • Ensure delivery as planned e.g. volunteer numbers • Evidence outcomes and impacts achieved • Support identification of good practice • Contribute to wider interest and thinking around community cohesion • Evidence success (or otherwise) of new ways of working “Can be a very positive reflective learning experience!”

4. Hypothetical Example • Quantitative Data: Output - 5 people engaged in a Conflict Awareness Training Course with 5 sessions undertaken over 5 weeks; cost £500. • Qualitative Data: By speaking with the course attendees, information is uncovered which details that they have used their skills in the community centre where they volunteer. • As a result people who use the venue where the course attendee volunteers now feel more safe and secure, and believe that if a situation involving conflict should arise then they feel confident that the volunteer can resolve it efficiently. • Possible Long-term/ “knock-on” Impacts?

5. Challenges in Evaluation • Often linked to Process, Implementation and Execution Issues • “Evaluation as a burden” • Purpose and Rationale • Evaluation Culture • Support and Communication • Insurance Policy Approach • Methodological issues – e.g. engagement, approach • Utility; Feasibility; Propriety; and Accuracy.

6. Different types of Evaluation • Impact Evaluations: • Beyond immediate results to identify longer-term and unintended program effects. • Counterfactual analysis • Can help to answer key questions for evidence-based policy making including what works, what doesn’t, where, why and for how much? • Summative Evaluations: • study the immediate or direct effects of the program on participants. • Usually ex-post and they are concerned with the results of an intervention. • consider the total impact and the unintended or unforeseen effects of an intervention, but combined with the overall objectives

7. Different types of Evaluation cont.. • Process Evaluations: • describe and assess the structures, procedures, operations, materials and activities within a scheme or program. • Examination of materials as programme develops: check appropriateness of the approach and procedures. • Main purpose: establish how project works in practice and achieves what it does. • Project Evaluations: • Look at the project as a whole: success against outputs, targets, milestone, outcomes and financial spend • Investigating the impacts on participants, communities and wider beneficiaries. • Establish lessons learnt, best practice and sustainability

8. Case StudyCohesive Communities Oldham and Rochdale ~ Families Project: Research Brief • Neighbourhood Background and Project Aims • Activities will include: • Three day long sessions run by volunteers with families, young people, children and older people; • The day sessions will involve a wide range of volunteer run activities including dance, sport, music, healthy eating and lifestyle advice. • Project Objectives are: • to develop a better understanding of each other and each other’s neighbourhoods; • to investigate common concerns and interests in order to raise awareness, change perceptions and support the formation of links/friendships; • to enable the development of longer-term relationships between families and residents from different neighbourhoods;

9. Cohesive Communities Oldham and Rochdale ~ Families ProjectBreakout Questions Who would you involve in the evaluation? 2. What methods would you use to evaluate the project? 3. What should the design of the evaluation consider? 4. What challenges/issues may you face in conducting the evaluation (e.g. engaging participants, consent), how would you overcome these?

10. How to Evaluate Community Projects Breakout Session Workshop Facilitators: Lisa Hall & Kathryn Crowther York Consulting LLP

11. Cohesion Counts Evaluation Overview • Impact Evaluation Report covers: • Good Relations Oldham: Conflict Awareness Training and Bridge Building Project • Groundwork: Multi Sports and Dance-Uni • Spiral Dance – Folkways in Falinge • And: • Philosophy for Communities • First Choice Homes Oldham Linking Project • Limeside and Clarkwell Linking Communities Project • Five Overarching Aims focused on Satisfaction (Aim 1); Belonging (Aim 2); Perceptions (Aim 3); and Relationships (Aims 4 & 5). “Transforming neighbourhoods to improve peoples’ lives”

12. Cohesion Counts Evaluation: Responses to Breakout Session Questions 1 & 2 • Who would you involve in the evaluation? • Stakeholders, project deliverers, community members and beneficiaries. • Different perspectives and focus • What methods would you use to evaluate the projects? Focus Groups One-to-one Consultations Questionnaires Secondary Data Monitoring Forms Video Diaries/Reporting Discussion Facilitators – Shoe Box Exercise Monthly Update Emails Mood Boards Scenario Testing Blogs/Forums

13. Cohesion Counts: Mood Board Examples

14. Cohesion Counts Evaluation: Responses to Breakout Session Question 3 • What should the design of the evaluation consider? • Focus of Activity Impact of Cohesion Counts Projects on the five Overarching Aims: • Aim 1: to improve residents’ satisfaction with the neighbourhood in which they live; • Aim 2: to improve resident’s sense of belonging to their street, neighbourhood and Borough; • Aim 3: to improve residents’ perceptions of living in communities mixed by age, tenure, property types, areas of the neighbourhood, and social and ethnic backgrounds; • Aim 4: to improve relationships within neighbourhoods between residents mixed by age, tenure, property types, areas of the neighbourhood and social and ethnic backgrounds; and • Aim 5: to encourage residents to build relationships with people from different backgrounds from themselves through meeting and talking in a variety of places.

15. Cohesion Counts Evaluation: Responses to Breakout Session Question 3 cont… • What should the design of the evaluation consider? • Key Factors: • Approach: Real-Time Evaluation • Size of Project: Small with clear overall aims • Priority/Importance: Impact against Overarching Aims • Innovation of Project: Cohesion and “Softer Outcomes” • Evaluation Techniques available • Understand issues associated with certain techniques • Critically assess different methods • Communication, flexibility and adaptability

16. Cohesion Counts Evaluation: Responses to Breakout Session Question 4 • What challenges/issues may you face in conducting the evaluation how would you overcome these? • Individual/project/programme priorities • Communication • Engaging participants • Consent • Consultation Fatigue • Appropriate Evaluation Methodologies • “Innovative” and “Engaging” • Interpreting and Analysing Results • Facilitation skills • Confidence!

17. How to Evaluate Community ProjectsSummary • Things for delegates to consider in completing their own evaluations or when writing evaluation briefs for external evaluators; • Understanding and Developing Aims and Objectives • Understanding and Developing Evaluation Methodology • Primary and Secondary Research • E.g. Face to face, telephone, postal, case studies, focus groups, mood boards, scenario testing, interactive media, observation • Reporting and Disseminating • Budgets, Timescales, Focus, Referral • Ensuring Evaluations are good quality and robust; • Managing the Process • Collecting Evidence • Interpreting the Data

18. How to Evaluate Community ProjectsSummary cont.. • Sign posting to further build knowledge of and confidence in evaluation; • United Kingdom Evaluation Society (UKES) http://www.evaluation.org.uk/resources.aspx • Institute of Community Cohesion (ICoCo) http://www.cohesioninstitute.org.uk/home • Department for Communities and Local Government (DCLG); http://www.communities.gov.uk/corporate/ • Joseph Rowntree Foundation; http://www.jrf.org.uk/ • Cohesion Counts Website: • any questions/issues: • lisa.hall@yorkconsulting.co.uk or kathryn.crowther@yorkconsulting.co.uk

19. How to Evaluate Community Projects Workshop Facilitators: Lisa Hall & Kathryn Crowther York Consulting LLP