Monitoring and Evaluation in CBA - The specificity of M&E in the field of adaptation

1. Monitoring and Evaluation in CBA- The specificity of M&E in the field of adaptation UNDP M&E framework for adaptation, VRA and IAS C. O. Nyandiga and A.F Wittmann Presented at the UNDP CBA- UNV Workshop, August 2-10, Dakar-Senegal

2. What is Monitoring and Evaluation? • Monitoring = A continuous data collection and analysis process implemented to assess a project (or program or policy) and compare it with the expected performance • Evaluation = A systematic and objective measurement of the results achieved by a project, a program or a policy, in order to assess its relevance, its coherence, the efficiency of its implementation, its effectiveness and its impact, as well as its sustainability (source : World Bank)

3. Why do we do M&E ? • Monitoring and Evaluation = a systematic and on-going process, that is part of project inception, planning and implementation • M&E is necessary to ensure that the OBJECTIVES of the project are met M&E • Provides project management and stakeholders with on-going indications of progress or lack of progress in achievements of objectives • Helps identifying problems and successes of projects, during implementation • Provides a basis for corrective actions • Helps reinforcement of initial positive results • Helps in determining if a project is still relevant

4. UNDP M&E Adaptation framework ADAPTATION : What do we want to measure ? We want to make sure our objectives are achieved : improving the adaptive capacity / reducing the vulnerability of the communities and the ecosystems on which they rely. What can we measure ? • Coverage : extent to which the project reaches vulnerable stakeholders (individuals, households, businesses, government agencies, policymakers…) • Impact : extent to which the project reduces the vulnerability, through policymaking, capacity building…) • Sustainability : ability of stakeholders to continue the adaptation process beyond project lifetime • Replicability : extent to which projects generate and disseminate results and lessons of value for replication

5. UNDP M&E Adaptation Measurements at Project Level for all Thematic Areas Coverage • i. Number of households, businesses engaged in vulnerability reduction or adaptive capacity development activities, as a proportion of households in the community or region targeted by the project. • ii. Number of policies introduced or adjusted to incorporate climate change risks. • iii. Number of investment decisions revised or made to incorporate climate change risks. • iv. Number of stakeholders (individuals, households, communities, etc.) served by new or expanded climate information management systems (e.g. early warning systems, forecasting,etc.). Impact • i. Percent change in stakeholders’ behaviours utilizing adjusted practices or resources for managing climate change risks, assessed via QBS. • ii. Percent improvement in stakeholders’ capacities to manage climate change (e.g. communicate climate change risks, disseminate information, or make decisions based on high quality information), as relevant, assessed via QBS. • iii. Percent reduction in perceived vulnerability: • a. Percent improvement in stakeholder perceptions of vulnerability to a recurrence of primary climate change-related threat(s), assessed via QBS.

6. UNDP M+E Adaptation Measurements for all TA -ctd Sustainability • i. Number of beneficiaries of project receiving training in implementation of specific adaptation measures or decision-support tools. • ii. Local (or spatially appropriate) availability of skills and resources necessary to continue adaptation after conclusion of project, assessed via QBS. • iii. Support for project activities among participating communities as assessed by QBS. • iv. Number of outside programmes, policies or projects incorporating project results into their processes. Replicability • i. Number of “lessons learned” from the project

7. CBA M&E Adaptation framework • CBA falls within the UNDP Framework for M&E of Adaptation to Climate Change • The specificity of CBAdaptation = it is a challenge to monitor and evaluate it, because : • “Adaptation is not generally an outcome, but rather consists of a diverse suite of ongoing processes that enable the achievement of development objectives under climate change” • “Adaptation cuts across many development objectives” • The results of adaptation can be measured in the long-term, whereas projects have a short duration • It is difficult to decouple climate risks from other drivers • (UNDP Framework for Monitoring and Evaluation of ACC)

8. M&E in the CBA programme • M&E aims to ensure and show that the CBA objectives are met : • Building adaptive capacity • Reducing community vulnerability • Increasing the resilience of global environmental benefits to climate change (including variability) in the GEF focal areas • M&E applies to the 3 CBA outcomes • Outcome 1 – Local level (projects) • Outcome 2 – National level • Outcome 3 – Global level

9. CBA M&E system • 4 sets of indicators are monitored • The UNDP Adaptation Indicators • Adaptive capacities • Sustainable community management of natural resources • The SGP’s Impact Assessment System (IAS) • GEB • Livelihood / Empowerment • The Vulnerability Reduction Assessment • Volunteerism and Inclusive Participation indicator

10. CBA M&E system • 4 sets of indicators are monitored • At the project level • Aggregated at the national level • Then again aggregated at the global level

11. UNDP adaptation indicators • Adaptive capacity fostered among natural resource dependent communities through awareness building, enhanced access to climate change and scenario information, and improved access to alternative resources • 2.1 Number of stakeholders at community level (e.g. businesses, community representatives, CBOs, NGOs) engaged by project and provided with training in climate change risk management and scenario planning. • 2.2 Population covered by awareness building programmes to increase understanding of risks associated with climate change among general public and key stakeholder groups. • 2.3 Increase in awareness of climate change related risks to natural resources (QBS) • 2.4 Percentage change in natural resource dependent population with access to alternative or supplementary livelihood options (QBS) • Sustainable community management of natural resources in the face of climate change promoted • 4.1 Percentage of population in relevant areas engaged in sustainable community management activities • 4.2 Number of measures deployed as part of sustainable resource management activities. • 4.3 Percentage of area of concern in which sustainable resource management activities are implemented. • 4.4 Number of stakeholders (e.g. families/households) benefiting from sustainable resource management activities (e.g. in terms of increased income or food security). • 4.5 Success of sustainable resource management interventions in securing livelihoods and protecting resources (QBS).

12. SGP’s Impact Assessment System • Why ? Measure the Global Environmental Benefits and the Livelihood / Empowermentbenefits generated by the project • What ? • GEB / • Biodiversity-(# of species, innovations/new technologies, local/national policies) • Land Degradation- (ha. of land restored, land sustainably managed, tons of soil erosion prevented, # of innovations/new technologies, local/national policies) • Livelihood / Empowerment / • Poverty Reduction –(# of households or individuals who benefited from the project, income generation achieved through the project…) • Capacity Building –(# of NGOs, community groups whose capacities were increased, #of women participating, support ensured from local / governmental institutions…)

13. SGP’s Impact Assessment System ctd. • Who, How ? And When ? At project conception, chose one GEB indicator and one LIVELIHOOD / EMPOWERMENT indicator During project development, measure the baseline values of the indicators and prepare your monitoring plan (in project proposal) – During project implementation, measure the evolutions of the indicators (project reports) – RESPONSIBILITY OF THE GRANTEE At the end of the project, measure the final indicator (final participatory evaluation / final report) – RESPONSIBILITY OF THE GRANTEE • Monitoring Plan – • Whatyouwillmeasure (indicator) • Whatis the target value of yourindicator • Howyouwillmeasureit • Whenyouwillmeasureit • Whowillmeasureit Don’tforget to report the indicators in the SGP database

14. Examples of the type of possible indicators for GEB Arid & semi-Arid zone ecosystems • Local population estimates of arid and semi-arid zone species • Changes in extent of undegraded arid and semi-arid habitat. Coastal, Marine and Freshwaters: • Local population estimates of target aquatic species • Changes in extent of healthy coral reef • Changes in extent and quality of mangroves (patch size, maximum tree size). Forest: • Local population estimates of forest species • Changes in natural forest extent • Number of concessionaries in the formulation of timber extraction Mountain: • Local population estimates of montane species • Changes in extent of undegraded montane habitat • Rate of deforestation on slopes >20% in project areas Agrobiodiversity: • Changes in number of local land races & domestic livestock breeds. • Changes in extent of agricultural area using low-input high diversity production methods • Incorporation of local land races and indigenous breeds in national breeding programmes • Certification standards for agricultural products

15. Ex-post measurements WHAT • Systematically capture and document off-site, indirect and longer-term impacts- “ex-post evaluation”. Generates GEB Measurements HOW • Using proxy and development indicators while project is ongoing • Using structured QBS during and after project closure • External independent consultants WHEN • Minimum 5 years after project completion-Done by UNDP EO

16. Vulnerability Reduction Assessment (VRA) • What ? The objective of a CBA project is to REDUCE the vulnerability of the community. VRA is a tool that contributes to measure achievement of this objective. VRA is a form of Participatory Impact Assessment, which focuses on the community’s own perception of its vulnerability and adaptation capacities. • Why ? • Measure the community’s perceptions of current and future climate change risks and adaptive capacities (now and in the long-run) • Measure the obstacles to adaptation, and the assets • Evaluate the sustainability of the project • Capture qualitative information / knowledge from the community, essential for project development/management (to make sure the project responds to community priorities) and for knowledge-management • Capture quantitative (thus comparable) information (=scores) that will allow to verify the achievement of CBA objectives (at project / national / global levels)

17. Vulnerability Reduction Assessment (VRA) • How ? When ? VRA is composed of 4 indicators, that are transformed into 4 questions (tailored to the local context). These questions form the core of a “VRA participatory workshop”, that is organized at least 3 times in the course of a project, in the community (at the beginning, at mid-course of project implementation, at the end).

18. Vulnerability Reduction Assessment (VRA) • The H-Form = the tool that is used for VRA facilitation and data collection

19. Organizing a VRA session Where ? Choose an appropriate location in the village, according to what you already know about the community. The location has to be accepted by / accessible to everyone When ? Accommodate the community schedule, including women’s schedule Who ? Facilitating team : 1 or 2 lead facilitators (with leverage and experience ; who know the local language and culture) 1 person to take notes on the H-form 1 person to take extensive notes on the side 1 person to take pictures and coordinate the logistics MOBILIZE the grantee as well as community-members. BE GENDER-SENSITIVE QUICK TIPS -Make sure all the members of the community are invited and informed about the meeting -Create a friendly atmosphere, where everyone is at ease to speak -Promote open discussion (the participants are the “teachers”, because we want to learn from them and capture their perceptions) -Use a language that is appropriate -Prepare your H-forms in advance (as well as all the material you might need) -Inform the authorities in advance

20. Sequencing a VRA session (example from Morocco) • 1-Introduction • Present the facilitating team • Explain the context of the workshop (CBA project) and the objectives of the workshop • Explain the “rules” : promote open participation (there is no wrong / right answer; everyone is allowed to speak…) • 2-Launch a general discussion about the local environment and climate • Use simple questions and tackle climate aspects one by one • what is the “usual” climate here ? • when does it rain ? when is it hot ? For how long… • Are there any storms, droughts ? When ? • you can draft a season calendar for example • 3-How does the community experience climate change? • Have you noticed any changes in the past years ? • Has it rained more ? less ? When did it rain last year ? for how long ? • Is it hotter than in the past ? is the dry season longer ? • Are there more storms etc. QUICK TIPS • -Adapt to relevant climatic matters • -Be very specific : referring to “climate change” is too vague in most contexts • -Ask simple questions one by one, so people have time to discuss them

21. Sequencing a VRA session (example from Morocco) 4-What are the impacts of these changes on your livelihood = VRA QUESTION 1 / Fill out H-form 5-What will be the impacts if these changes (more storms / more droughts) double in the future ? = VRA QUESTION 2 / Fill out H-form The impacts are noted in 2 columns (positive impacts / negative impacts) After the discussion, assessment : how serious are these impacts, in the view of the community What are the possible solutions to limit these impacts ? 6-What are the obstacles / assets for adaptation ? = VRA QUESTION 3 / Fill out H-form What prevents you from implementing the solutions that you mention ? After the discussion, assessment : are the obstacles stronger than the assets What are the possible solutions to overcome the obstacles and build on the assets ? 7-Do you think that with the project, you will be able to adapt further in the future ? = VRA QUESTION 4 / Fill out H-form (Assess the sustainability of the project, and how people will contribute) After the discussion, assessment : are the participants optimistic about the outcomes of the projects ? are they willing to actively participate ? 8-Conclusion Thank people for their participation Present the next steps of the project

22. Vulnerability Reduction Assessment (VRA) • What are the results of a VRA session • QUANTITATIVE DATA = SCORING Generate a score for each question (noted on 5; 1 shows that the vulnerability is high / 5 shows that the vulnerability is low Calculate the average from each question Multiply the total average by 2 to obtain a score noted on 10 • QUALITATIVE DATA = all the information captured during the session, and classified in the different parts of the H-form • Negative impacts • Positive impacts • Ideas, Solutions • How to use the results of a VRA session • The initial VRA results • Are used for project development an incorporated in the project proposal (provides data for baseline assessment / local knowledge / and ideas for project outcomes/outputs) • Provide the baseline vulnerability measurement, reference for project monitoring • The 2nd and 3rd VRA results are to be compared with baseline, in order to monitor the project. QUICK TIPS • Always make sure you thoroughly document the VRA sessions • Take detailed notes on your own H-form, • Note your personal side-comments –how many people ? Where ? When ? Who facilitated ? What went right / wrong ? What can be improved ? … • Take pictures Don’tforget to report the indicators in the SGP database

23. Vulnerability Reduction Assessment (VRA) • Lessons learned from implementation • VRA is a powerful way to capture a lot of information from the community • It gives an opportunity for mobilization and discussion, and contributes to fostering community ownership of the project • Language and communication are critical • Gender appropriateness is essential (in the choice of the location, scheduling, language, facilitators) : in Morocco, for example, we do specific women’s workshops • VRA is a flexible tool : its organization can be tailored to the specific context (for ex. In Morocco, we don’t always use the big H-form, and we added some questions/discussions to prepare for the VRA questions) – PROVIDED ALL THE INFO IS COLLECTED FROM THE DISCUSSION • VRA can be incorporated in a larger workshop (information about CC, training, rural assessment…) so that the communities are prepared and so that the discussions are to the point. • The main challenge is the QUANTITATIVE DATA (for discussion) How can they be better understood (scale of value is not necessarily translated in numbers) ? How can they be better captured (voting/scoring might block participants; people don’t want to stand out)? How can they be more reliable for monitoring (composition of the group changes from one VRA to the next ; participants might over/underestimate the scores in order to ensure the funding… ? There is a potential for improvement, in order to be able to prove the impacts of our projects.

24. Volunteerism / Inclusive participation BASELINE indicators (project proposal) • What are the mechanisms for volunteerism that already exist in the community before the CBA project (for example, traditional mechanisms for mutual assistance, associations, etc.)? • Number of volunteers in the community already engaged in climate change adaptation activities before the CBA project • What are the opportunities or obstacles that could facilitate or impede people from engaging in voluntary activities? The CBA project aims at reinforcing Volunteerism : these indicators should be monitored over the course of the project Be sure to capture quantitative / qualitative data on community volunteers and participation during the project : in the activity reports / follow-up on the Volunteerism chart Inclusive participation : monitor the vulnerable group’s participation (esp. women, youth, people with a handicap, elders, property-less farmers…). In the activity reports / follow-up on the Volunteerism chart

25. M&E timeline

26. Conclusion and discussion • More and more, we are asked to prove our results. • M&E is ESSENTIAL to show that our projects achieve their objectives. • This is why we need to assess the challenges of our system, and try to improve it.

27. Pleaseshare … • How do you implement our M&E ? • How do you explain it to the proponents / communities, and support them in implementing ? • What are the challenges you are facing ? • What do you need to overcome these challenges ?

28. QUESTIONS • Quantitative aspect of VRA ? • Consolidate our 4 set of indicators into 1, without losing any of the information ? • Incorporate project outcome indicators ?