1 / 34

INDICATOR CHARACTERISTICS

INDICATOR CHARACTERISTICS. L-MEP. Liberia - Monitoring & Evaluation Program. How closely do the indicators used measure the results?. INDICATOR CHARACTERISTICS: OBJECTIVES. Use 2 key questions to create indicators Use 7 indicator criteria to select/improve indicators. L-MEP.

lucien
Télécharger la présentation

INDICATOR CHARACTERISTICS

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. INDICATOR CHARACTERISTICS L-MEP Liberia - Monitoring & Evaluation Program How closely do the indicators used measure the results?

  2. INDICATOR CHARACTERISTICS: OBJECTIVES Use 2 key questions to create indicators Use 7 indicator criteria to select/improve indicators L-MEP Liberia - Monitoring & Evaluation Program 2

  3. What is a Performance Indicator ?(ADS Def. p 67) L-MEP Liberia - Monitoring & Evaluation Program • A particular characteristic or dimension used to measure intended changes defined by a Result Framework.… • A phenomenon we observe that tells us there is a problem and whether it is being solved. • The measurement that shows how and by how much a project is achieving its intended results. 3 3

  4. An example of an Indicator L-MEP Liberia - Monitoring & Evaluation Program Body temperature : an indicator that may show that illness is present. In order to use this indicator to diagnose illness, it is necessary to know, at a minimum, how far above the normal level the temperature is. 4 4

  5. Role of Indicators in a Results Framework L-MEP Liberia - Monitoring & Evaluation Program • Indicators apply at all levels of the project hierarchy of results. • Indicators are used to measure change over time. The result statement will show the direction of change desired (increase or decrease) NOT the indicator. • Example: Percentage of children ages 12 to 24 months in the target areawho are fully vaccinated. (NOT Increased percentage) • This indicator tells us thephenomenonto observe, not which way the change might move. 5 5

  6. Context Indicators Reality surrounding a development problem is usually greater than USAID’s manageable interest. Example: Damages caused by forest fire reduced. Tracking rainfall as a context indicatorinforms the analysis of USAID efforts to control forest fires. L-MEP Liberia - Monitoring & Evaluation Program 6 6 6

  7. If a Result is the amount of change achieved or a problem solved: What is a Problem? A discrepancy in somebody’s head between a Perception (based on current data) and an Ideal (based on ideal data) floating on an Emotion supported by Values L-MEP Liberia - Monitoring & Evaluation Program P(cd) / I(id) E V Data Reality 7 7

  8. To formulate Indicators for a Results Framework, ask these key questions How do we know it is a problem? = Data What data generated the problem through which of the 5 senses? What phenomenon are we observing? = Indicator What will we observe to see whether the problem is solved? L-MEP Liberia - Monitoring & Evaluation Program 8 8

  9. Table Exercise V L-MEP Liberia - Monitoring & Evaluation Program Each table receives an envelope with cards. On each card is a problem statement or an indicator. Instructions: Match the indicators to the appropriate problem statement and post the matches on a flipchart page or on a table. 10 min. 9 9

  10. L-MEP Liberia - Monitoring & Evaluation Program Problem Statements…Matched with Indicators • The loss of tropical glaciers in the Andes impacts the hydrologic cycle and raises grave concerns over future water supplies throughout the region (Barnett et al., 2005). • Number of years of glacial water supply remaining • Tropical glaciers loss rate • The chronic lack of planning capacity, linkages, and communication among and between stakeholder groups is resulting in an increase in conflict between lowland and highland populations in Peru and other Andean countries • (Oré 2005). • Number of water resources management-related conflicts 10 10

  11. L-MEP Liberia - Monitoring & Evaluation Program • Coastal regions are home to nearly 70 per cent of the country’s population and that produce 60 per cent of its agricultural GDP • percentage of country population in coastal regions • percentage of agricultural GDP of coastal regions • Water from the Santa watershed also powers three hydroelectric stations with a total capacity of 271 MW, or 5.5 per cent of the total national hydroelectric output for Peru, that directly and indirectly serves 1,697,221 people throughout the region • Ratio of current capacity to original capacity of three hydroelectric stations 11 11

  12. L-MEP Liberia - Monitoring & Evaluation Program Water resources management in the watershed basins of Santa/Piura Chira improved Indicators: Tropical glaciers loss rate Number of years of glacial water supply remaining Number of water resources management-related conflicts Percentage of country population in coastal regions Percentage of agricultural GDP of coastal regions Ratio of current capacity to original capacity of three hydroelectric stations 12 12

  13. L-MEP Liberia - Monitoring & Evaluation Program MCS Case: Indicators with Associated IRs Investment in improved water resources management increased $ value of qualified project investments Conflicts over water avoided # of water related conflicts Political will to change water management decision making generated # of working groups formed Politically balanced decisions made # of consensus decisions reached Good representation at the conference secured % of targeted stakeholders who attend Decision-making processes function effectively # of active highland/lowland decision making processes 13 13

  14. Task VI b: Create Indicators for the Results Framework Using any Problem Description/Analysis, create one or more indicators for each Result. Write each indicator on an A4 paper and place it under the corresponding result in your Results Framework. Ask these questions: How do we know it is a problem? Ask this question until you get to the data that generated the problem in your heads. What phenomenon are we observing that generated the data? Ask this question to define the indicator(s) Add any informative Context Indicators.45 min L-MEP Liberia - Monitoring & Evaluation Program 15 15

  15. Indicator Job Description and Criteria Objective Practical Useful for Management Direct Attributable to USAID efforts Timely Adequate (ADS 203.3.4.2– p 16-17) L-MEP Liberia - Monitoring & Evaluation Program Job Description: To tell the most truth about whether a result is being achieved at the least cost when you need to hear it. USAID’s characteristics of goodperformance indicators: 16 16

  16. Objective? Unambiguous and operationally precise about What is being measured What data are being collected Uni-dimensional -- measures only one thing Consistent over time L-MEP Liberia - Monitoring & Evaluation Program • Result: • - Performance of export firms improved • Imprecise Indicator: • - Number of successful export firms • Precise Indicator: • - % of export firms experiencing annual increase in revenues of at least 5% 17

  17. PRACTICAL? Data are available when required for decision making Data can be obtained at reasonable cost L-MEP Liberia - Monitoring & Evaluation Program 18

  18. USEFUL? Is the indicator useful for management? Which indicators are most meaningful at a given point in time? L-MEP Liberia - Monitoring & Evaluation Program 19

  19. DIRECT? The indicator closely tracks the result it is intended to measure. L-MEP Liberia - Monitoring & Evaluation Program • Result: • Citizens’ knowledge of their rights expanded • Direct Indicator: • % survey respondents able to identify 3 or more key civil rights 20

  20. L-MEP Liberia - Monitoring & Evaluation Program PROXY INDICATORS • Indirect measures of the intended result. • Use if data for direct indicator is unavailable or not feasible to collect. • Result: • - Citizens’ knowledge of their rights expanded • Direct Indicator: • - % survey respondents able to identify 3 or more key civil rights • Proxy Indicator: • - Number of civil rights cases brought to court by targeted community organizations

  21. ATTRIBUTABLE? Indicators selected for the PMP should measure changes that are clearly and reasonably attributable, at least in part, to USAID effort. Attribution exists when the links between USAID outputs and the results being measured are clear and significant. A simple way to assess attribution: L-MEP Liberia - Monitoring & Evaluation Program If there had been no USAID activity, would the measured change have been different? 22

  22. TIMELY? Indicators should be available when they are needed for decision making. Are the data available frequently enough? Are the data current? L-MEP Liberia - Monitoring & Evaluation Program 23

  23. ADEQUATE? Taken as a group, the indicator and its companion indicators should be the minimum necessary to capture progress towards the desired result. How many indicators? As many as are necessary and cost effective for management purposes Too many indicators results in information overload Too few indicators could be misleading Use the “necessary and sufficient” rule to select the minimum number per result L-MEP Liberia - Monitoring & Evaluation Program RULEOF THUMB 2-3 PER RESULT 24

  24. Table quiz: Selecting indicators (“x” rejection criteria) (20 min.) L-MEP Liberia - Monitoring & Evaluation Program 25 25

  25. Indicators L-MEP Liberia - Monitoring & Evaluation Program Quantitative: Number, amount, ratio, percentage, proportion, average scores, rating, weighted or non-weighted index, etc. Qualitative: Description of the status of an intended achievement, documented observations, representative case descriptions, perceptions, opinions of value, photos, etc. 26 26

  26. INDICATORS: QUANTITATIVE, QUALITATIVEOR BOTH? QUALITATIVE Expert opinion on comprehensiveness of a law QUANTITATIVE Dollar value of plantains exported L-MEP Liberia - Monitoring & Evaluation Program BOTH • Country score on “Corruption Perceptions Index” 27

  27. Outcome Output L-MEP Liberia - Monitoring & Evaluation Program Example of Indicators at Different Levels Impact • Under-five mortality rate • Incidence of diphtheria, pertussis and tetanus in young children • No. of children vaccinated against diphtheria (D) • No. of children vaccinated against pertussis (P) • No. of children vaccinated against tetanus (T) • No. of vaccines provided to clinics Input/Activity 28 28

  28. Disaggregation and Gender ADS 203.3.4.3 (p 17-18): Performance management systems and evaluations at the AO and project or activity levels must include gender-sensitive indicators and sex-disaggregated data when the technical analysis supporting the AO, project or activity to be undertaken demonstrate that: The activities or their anticipated results involve or affect women and men differently; and If so, this difference would be an important factor in managing for sustainable program impact. L-MEP Liberia - Monitoring & Evaluation Program 29 29

  29. Water resources management in the watershed basins of Santa/Piura Chira improved • Tropical glaciers loss rate • Number of years of glacial water supply remaining • Number of water resources management-related conflicts • % of country population in coastal regions • % of agricultural GDP of coastal regions • Ratio of current capacity to original capacity of three hydroelectric stations ? Water resources management improved # of WRM investments validated both scientifically and politically Politically balanced decisions made # of consensus decisions reached Investment in improved water resources management increased $ value of qualified projects Conflicts over water avoided # of water related conflicts Effective decision making processes function # of active highland/lowland decision-making processes Political will to change water management decision making generated # of working groups formed Stakeholder capacity increased # of shared planning tools used in decision making Good representation at the conference secured % of targeted stakeholders who attend 30 30

  30. L-MEP Liberia - Monitoring & Evaluation Program Assessing Performance Indicators Against Criteria Result: Water resources management in the watershed basins of Santa/Piura Chira improved 31 31

  31. L-MEP Liberia - Monitoring & Evaluation Program Assessing Performance Indicators Against Criteria Result: Water resources management in the watershed basins of Santa/Piura Chira improved 32 32

  32. L-MEP Liberia - Monitoring & Evaluation Program Context Indicators 1. # of water resources management-relatedconflicts 2. Rate of glacial melt Critical Assumptions 1. National policy supports watershed basin regional planning MCS Case Indicators • Result: Water resources management in the watershed basins of Santa/Piura Chira improved • . Number of Hectares (HA) classified as “restored” • . Number of HA classified as “conserved” 3. Number of water resource management investments validated both scientifically and politically IR1: Models for restoration/conservation validated 1. Number of models replicated 2. Number of replications IR2: Water management decision making restructured 1. Cases of highland/lowland decision making processes functioning IR3: Investment in improved water resources managem. increased 1. $ value of qualified projects 33 33

  33. TASK VII: Select Good Performance Indicators Select one Result with multiple Indicators Use the Indicator Criteria table to put your indicators on the left with the 7 standards across the top. Do your indicators meet all the criteria? Fill in the boxes with “Yes” or “No”. Anything need to be changed? 30 min. L-MEP Liberia - Monitoring & Evaluation Program 34 34

  34. Session 3. Summary Use 2 key questions to create indicators Use 7 indicator criteria to select/improve indicators Reality doesn’t have problems, we do. L-MEP Liberia - Monitoring & Evaluation Program 35

More Related