What Works, and Why: Issues and Methods in Evaluation in the Non-Profit Sector

1. What Works, and Why:Issues and Methods in Evaluation in the Non-Profit Sector Edward T. Jackson Carleton University, Ottawa Presented to the Future Fund Evaluation Workshop, Ontario Trillium Foundation, Toronto April 28, 2010

2. Overview • Context • Issues in Evaluation • Results-Based Management • Participatory Strategies • Social Return on Investment • Next Steps • Resources

3. Context I: Ontario Economy – Challenges • Restructuring driven by globalization (BRICs), high dollar • Shrinking of the manufacturing sector • A return to a branch-plant, staples-driven economy • Reduced number of Ontario multi-nationals • Decrease in corporate and individual tax revenues to governments

4. Context II: Ontario Economy – Opportunities • Enabling New Ontarians, Aboriginal Ontarians and young people to fill upcoming labourshortagesin all sectors • Accelerating the development of good jobs in the green economy (i.e. energy, water, etc) • Building new, large-scale Ontario-based anchor companies in key sectors with a focus on value-added production, global markets and local suppliers • Levering social innovation-mobilizing technology, ingenuity and entrepreneurship to solve pressing social problems • Exporting Ontario’s expertise in higher education management, education and research • Optimizing the role of and linkages with non-profit organizations and social enterprises in these initiatives

5. Context III: OTF’s Response • Funding Priority: “Enhanced employment and economic potential” • Future Fund: “To support initiatives that create significant and sustainable opportunities for communities, networks and individuals to participate fully in Ontario’s labour market… to break down barriers to employment and self-employment” • Implication: To rapidly convert innovation into large-scale implementation.

6. Context IV: OTF Evaluation • Main Purpose: “To measure the success of a project or program by the end of the grant” • Learning: What works, what doesn’t and why? • Meeting Needs: Are the needs of the target population being met? If not, how can they be better met? • Replication: What are the key components of the intervention worth replicating? • Accountability: To what extent is the project or program efficient and effective?

7. Context: OTF Approach to Measuring Results • Results should be specific, concrete and measurable, of short-term or long-term duration, and should be based on qualitative or quantitative data, or both. • Results should be measured at the following levels: • Individuals • Organizations • Communities

8. Key Issues in Evaluation • Who will use the evaluation, and how will they use it? • How feasible is it to actually achieve the proposed results (are they realistic or overstated/hyped)? • What are the different expectations among stakeholders as to what constitutes a credible evidence base for assessing effectiveness? • What are the costs – in terms of money and time – of conducting the evaluation? • To what extent are there asymmetries in stakeholder participation in the design and implementation of the evaluation? • What are the real prospects and mechanisms for downstream replication and scaling-up?

9. Results-Based Management • Results-Based Management • Results Chain • Impact Value Chain • Results Grid • Case Study – Alterna Micro-Loan Fund Evaluation

10. Results-Based Management • “A life-cycle approach to management that integrates strategy, people, resources, processes and measurement to improve decision-making, transparency, and accountability. The approach focuses on achieving outcomes, implementing performance measurement, learning and changing, and reporting performance” (CIDA)

11. Results Chain • “A depiction of the causal or logical relationships between inputs, activities, outputs and outcomes of a given policy, program, or initiative” (CIDA, OECD)

12. Immediate Outcomes (Short-Term) Intermediate Outcomes (Medium-Term) Ultimate Outcomes (Long-Term) Inputs Activities Outputs Results Chain

13. Impact Value Chain Impact = Measure of Change Inputs Activities Outputs Outcomes Goal Alignment What you put in What you do Results that are measured Collection of all results intended and/or unintended —MINUS— What would have happened anyway, or next best alternative (results “but for” the intervention) = IMPACT How well outcomes align with intended goals • Funds • Time • Expertise • Networks • Influence • Grantmaking • Programs • Information • Sharing • Investing Activity results (e.g. number of people served) (Source: Goedeke and Pomares, 2009)

14. Level Immediate Outcomes Intermediate Outcomes Ultimate Outcomes (Impacts) Macro-Level (Policy) Meso-Level (Institution) Micro-Level (Local Organizations, Households, Individuals) Results Grid (Source: Jackson, 2010)

15. Mission Statement: To provide a compelling financial alternative to Canadians through a unique banking philosophy based on cooperative values and focused on professional financial counsel Program Components Inputs Activities Outputs Intermediate Outcomes Final Outcomes Impacts Small Business Development loans Alterna’s Capital (Credit Union members’ shares) 1 Full-Time Community Micro-Loan Manager + 1 Full-Time CSR Manager Provision of small loans (maximum $15,000 at prime rate + 6%) upon approval of loan requests submitted by interested applicants Loans disbursed to qualified applicants in a timely fashion (e.g. within two weeks from the applicants’ initial request) Start-up or expansion of small businesses (including purchase of productive assets, workspace rental, e-commerce development)  Micro-Level (Borrowers) Meso-Level (Corporate) Macro-Level (Policy) Free financial literacy programs and professional development opportunities Volunteer work by both Alterna Staff and recognized local business professionals/ Instructors - Business financial literacy programs offered to both members and prospective applicants - Member Display Booth and Information Boards at Alterna branches - Networking Café conducted as planned - Presentations made by Alterna CSR Officers at local business development centres and training colleges - Members’ enhanced understanding and adoption of effective business practices (e.g. business plan development, accounting, marketing) - Development of partnerships among borrowers in different fields Case Study: Logical Framework for the Alterna Savings Community Micro-Loan Program (Source: Tarsilla, 2010)

16. Evaluation Layers Final Outcomes Impacts Increase in Borrowers’ productivity and personal assets Increased borrowers’ payment of federal and provincial income taxes Micro-Level (Borrowers) Encouragement of borrowers’ self-sufficiency Reduced number of borrowers relying on government assistance Increase in borrowers’ annual income Enhanced well-being of borrowers and their households (including housing, nutrition, health) Meso-Level (Corporate) Increased free media-coverage of corporate CSR as well as of their services/products at large; boost corporate membership Increased corporate reputation within the community Development of new products and services specifically catering to the most marginalized and vulnerable population groups (e.g. new Canadians, individuals with no or low credit rating) Enhanced corporate brand differentiation Increased customer loyalty Consolidation of Alterna’s membership base (e.g. through referral) and enhance borrowers’ opening of additional accounts other than business related (e.g. personal lines of credit, mortgages). Macro-Level (Policy) Contributions to the health and success of the local economy; save government a large amount of resources otherwise allocated to assistance programs; contribute to federal and provincial revenues due to the borrowers paying higher income taxes over time. Promotion of self-employment and entrepreneurship among vulnerable population groups in Toronto Fostering the creation of new jobs (e.g. micro-loan borrowers hiring employees to run their business). Contribution to the reduction in the local unemployment rate Case Study: Results Grid for the Alterna Micro-Loan Program: Outcome (Source: Tarsilla, 2010)

17. Participatory Strategies • Participatory monitoring and evaluation • Why is PME important? • Core Principles • Front-end costs • Multi-Stakeholder engagement strategies • Stakeholder network • The PME cycle • Framework for Organizational Self-Assessment

18. Participatory Monitoring and Evaluation: Definition – PME • Participatory monitoring and evaluation is an approach to performance review in which stakeholders in an intervention (local citizens, policy makers, funding agencies, and non-governmental organizations) work together to decide how to assess progress, conduct data collection and analysis, and take action on their findings (Source: Jackson, 2005)

19. Why is PME Important? • PME is geared towards not only measuring the effectiveness of a project, but also towards building ownership and empowering beneficiaries; building accountability and transparency; and taking corrective actions to improve performance and outcomes (Source: World Bank, 2002)

20. Core Principles • Primary stakeholders are active participants – not just sources of information • Building capacity of local people to analyze, reflect and take action • Catalyzes commitment to taking corrective measures (Source: World Bank, 2002)

21. Front-End Costs • Time-consuming and expensive at the front end • Authentic stakeholder engagement takes time • Additional professional involvement as facilitators/intermediaries also adds costs (Source: Jackson, 2000)

22. Multi-Stakeholder Engagement Strategies • Evaluation Coordination: Evaluation committee, working group; evaluation team • Methods: • Qualitative: Workshops, community meetings, participant observation, online collaboration, interviews, questionnaires, video/film/theatre, other visualization tools • Quantitative: Large-scale surveys, cost-benefit analysis, statistical analysis • Other: Mapping, GPS, corporate/legal research, scientific monitoring (e.g. water quality) • Role of Evaluators: Catalyst and facilitators; ensure voice, choice; do no harm

23. Stakeholder Network Taxpayer*/ Donor* GovernmentAgency/ Foundation Implementing Agency Group/ Enterprise Individual* Non-Profits Spouse Non-Profits Consultants Consultants Children *The most results-oriented stakeholders (Source: Jackson, 1998) Other Family Members

24. 1 Deciding Who Participates 10 Taking Action on the Findings 2 Forming a Multi-Stakeholder Vehicle to Lead the Process 9 Sharing the Findings 3 Establishing Goals for the PME Exercise 8 Preparing the Report 4 Identifying Key Performance Issues 7 Analyzing the Findings 5 Developing Results Indicators 6 Gathering Relevant Data The PME Cycle

25. Framework for Organizational Self-Assessment (Sources: International Development Research Centre/Universalia)

26. Social Return on Investment • Social Return on Investment: Definition • Case Study: Turnaround Couriers • Expanded Value-Added Statement • Case Study: PARO Women’s Centre

27. Social Return on Investment: Definition • Discounted, monetized valuation of the social value that has been created, relative to the value of the investment • Key leaders in SROI include the Roberts Enterprise Development Fund, Jed Emerson, New Economics Foundation, Social Capital Partners, and the Social Economy Centre at OISE/UToronto • Receiving greater attention, but SROI can be resource-intensive and difficult to replicate

28. SROI Snapshot: Turnaround Couriers - Theory of Change Success Metrics Goals Methods • Youth are able to get out of shelter system and into independent housing • Youth meet or exceed job expectations • TurnAround helps youth secure next job and establish a career path • Youth are able to get off and stay off government financial assistance • Recruit youth from youth shelters and youth serving agencies across Toronto • Provide a real job, not a job training experience • Establish a supportive management environment • Assist youth with planning and making next steps regarding housing and employment • Hire couriers and office administrative staff from disadvantaged youth population • Provide transitional work experience to enable youth to develop employability skills, a resume and a support network • Enable youth to access the mainstream job market • Enable youth to stabilize life situation, begin a career path and leave the shelter system (Source: Social Capital Partners, 2009)

29. SROI Snapshot: Turn Around Couriers (Extract) • Overview of Target Population (sample) • 38% recruited directly from shelters • 23% female • Average age: 21 • 100% unemployed at time of hire • 54% receiving social assistance at hire • 54% been involved with justice system • 54% did not complete high school • Sustainable Livelihood Outcomes (sample) • 89 youth in total have been hired over 5 years • 100% target population recruited from shelters able to get out of shelter system and secure independent housing within 6 months of employment at TAC • 85% who relied on income support through social assistance at time of hire able to get off and stay off • Employment Outcomes (sample) • Increased target/non-target staff ratio to 83% • 69% continue to work at TAC (9) • 15% moved onto mainstream employment in window cleaning industry (2) • 8%went on to post secondary education (1) (Source: Social Capital Partners, 2009)

30. Expanded Value-Added Statement • An integrated social accounting method that employs carefully constructed assumptions to quantify, in dollar terms, the social value that a non-profit or co-operative produces (e.g. volunteer hours contributions, unpaid contributions to other stakeholders, etc.) • Usually applied to a single fiscal year, but can be used to assess multiple indicators over 10-20 years • The EVAS integrates the calculations of different types of social value into the overall financial statements of the organization being assessed.

31. Case Study: PARO Women’s Centre • Provides training, mentoring and small loans to women in northern Ontario, and manages a store that sells artisnal products • EVAS, calculated for 2005-2006, found that PARO created at least $132K from volunteer contributions, the value of mentorship and consulting by the Executive Director • For every dollar of purchases on goods and services, PARO created more than $2 worth of value added, including about $.70 worth of social value (Source: Babcock, 2007)

32. Next Steps • Reflect on key issues relating to the evaluation of your Future Fund grant • Learn more about evaluation strategies and methods • Work with OTF and other stakeholders to design an evaluation that is results-oriented, participatory and measures social return • Build a community of practice on FF evaluation • Document your experience for others, and learn from them as you proceed forward.

33. Resources • Blended Value (blendedvalue.org) • Canadian Evaluation Society (evaluationcanada.ca) • Canadian International Development Agency (acdi-cida.gc.ca) • Carleton Centre for Community Innovation (carleton.ca/3ci) • International Development Research Centre (idrc.ca) • International Program for Development Evaluation Training (ipdet.org) • Institute of Development Studies, University of Sussex (ids.ac.uk) • New Economics Foundation (neweconomics.org) • Organisation for Economic Co-operation and Development (oecd.org) • Roberts Enterprise Development Foundation (redf.org) • Social Capital Partners (socialcapitalpartners.ca) • Social Economy Centre, OISE/UT (sec.oise.utoronto.ca) • Social Finance (socialfinance.ca) • Treasury Board of Canada Secretariat (tbs-sct.gc.ca) • Universalia (universalia.com) • World Bank (worldbank.org)