Workshop to Strengthen the Design, Monitoring, and Evaluation of Family Planning Projects

1. MODULE 1 Program Design, Monitoring and EvaluationVirginia Lamprecht, Senior Technical Advisor, USAID Presented at CCIH Annual Conference, May 28, 2005 Workshop to Strengthen the Design, Monitoring, and Evaluation of Family Planning Projects

2. What Is a Results Framework? • A presentation of a whole view of a program including— • The big goal • Our ideas about the things that have to be in place to achieve success.

3. Why Use Results Frameworks? • Because they help us focus on the END result(s) and the strategies that we can use to achieve them • Because they force us to identify the logic and links behind our programs and to identify necessary and sufficient elements for success • Because a Results Framework is a simple tool to present what we are trying to accomplish in a program or project

4. Elements of Results Frameworks • Goal Statement—the change in health conditions that we hope to achieve • Strategic (or Key) Objective (SO)—the main result that will help us achieve our goal and for which we can measure change • (Intermediate) Results (IRs)—the things that need to be in place to ensure achievement of the SO • Strategies & Activities—what a project does to achieve its intermediate results that contribute to the objective

5. Some Typical Health Goals and Related Strategic Objectives • Goal: Stabilize population/Reduce fertility (we don’t measure) • SO: Increased FP use or reduced unmet need for FP • Goal: Reduce child mortality • SO: Increased use of preventive health behaviors (immunization) • Goal: Improved adolescent health • SO: Increased use of risk reduction behaviors among adolescents

6. General Characteristics of an Intermediate Result • Statement of a desired outcome or a situation that changed as a result of project intervention—not an activity or process • Contraceptive services accessible to target population • Community members are knowledgeable about and approve of FP. • This outcome contributes to our ability to get to our SO (e.g., use of FP). • The result is measurable.

7. Strategies and Activities • These are things the project does in order to achieve the desired outcomes or changes in the situation. For example: • Train and supply CBDs • Strengthen FP logistic systems • Advocate for contraceptive supply

8. Common Difficulties in Formulating Results Frameworks and Program Design • Mixing up results, strategies, and activities • Starting Project Design with a list of activities that may not logically lead to the desired objective • Choosing indicators that truly measure the results (we will get to this later on)

9. GOAL: Improved Reproductive Health Status SO: Improved Use of FP/RH Services and Practices IR 3 IR 1 Improved knowledge of, attitude toward, and interest in FP (and other health and/or care seeking behaviors) IR 2 Increased quality of FP/RH service delivery in facilities and community IR 3 Increased access to/ availability of FP/RH services/ supplies IR 4 Improved social and policy environment for FP/RH services and behaviors IR 1 IR 2 IR 4

10. A Complete Program Design Process • Involves the project team and key counterparts • Requires background research and data collection • Uses data collected to identify gaps • Creates a Results Framework • Helps the team determine strategies that address gaps • Develops an M&E plan linked to the RF/Project Design

11. Steps in the Design Process

12. Step 1—Situation Analysis (SA) Four Components of SA: • Secondary Data Compilation and Review • Policy Environment Scan • Participatory Qualitative Assessment • Health Service Delivery Assessment

13. What Is Situational Analysis? • Situational analysis describes— • Background setting • Current environment • Behaviors • Service delivery quality, use, patterns. • Situational analysis provides the basis for planning interventions, which— • Address gaps • Build on existing strengths within the community and the health system.

14. Tool for Synthesizing Situation Analysis Data (1 per IR)

15. Steps in the Design Process

16. Steps 2 and 3: Results Framework and Strategy Selection • Step 2: Create a Results Framework with desired results from the preceding step • Step 3: Articulate major strategies that will be used to achieve the results and/or address gaps/barriers as described in Step 1

17. Assumptions • With any program design, there are key assumptions that we make that are critical to success—it is worthwhile to note these! • General assumptions such as political stability, collaboration from MOH, and ability to hire competent staff • Specific assumptions such as availability of contraceptives; collaboration between partners for different elements, such as training and supervision; and favorable policy environment • Other examples

18. Project Summary Results Framework Improved Health Status Strategic Objective Improved (Sustained) Use of Key Health Services and Practices/Behaviors Intermediate Results Increased knowledge of and interest in… Increased quality of… Increased availability/ access to… Improved social / policy environment… Strategies: Strategies: Strategies: Strategies:

19. Small Group Work What are some strategies you would use to address— • Demand issues • Husbands are concerned that their wives will become infertile if they use the pill. • Quality problems • Facility staff do not respect confidentiality—may share a woman’s FP use with her husband. • Access problems • FP services in the clinic 8 kilometers away. • Policy problem • IUD cannot be inserted without negative lab results of pregnancy test.

20. An Example RF for a FP Project GOAL: REDUCED FERTILITY IRs SO: Increased FP use and improved FP/RH practices Increased knowledge of, improved attitudes toward, and acceptance of key services and behavior Increased quality of FP counseling and services for Increased availability/access to FP/RH Improved social and policy environment for FP Strategies: • Increase availability of educational materials at clinic and community level • Community mobilization (using PRA and PDI) including men • Implement mass media strategy • Mobilize opinion leaders at national and local level • Design/ implement supportive supervision System • Train service providers (in-service and preservice in FP counseling and management of side effects • Remodel clinic to allow for privacy • Design and implement quality improvement program • Strengthen logistics management • Mobilize private sector providers • Mobilize CHWs/CBDs • Encourage socially marketed pills Advocate for community based distribution of pills Promote addition of Depo injections to EPI outreach strategy Pilot social marketing of pills

21. Once We Have Selected Results and Likely Strategies… Step 4: It’s time to select indicators that measure the results we want to achieve and fit the strategies we will use.

22. Steps in the Design Process

23. Monitoring and Evaluation Planning Matrix

24. What Is an Indicator? • A variable • That measures • One aspect of a program/project • For a specific population. A well defined evaluation indicator typically has both a numerator and denominator but not always.

25. Let’s Take a Look at Indicators • We choose indicators that will show us whether we achieved what we set out to do. • We choose indicators that correspond to the stated results and specific strategies. • We use existing resources to identify and monitor already tested indicators with good definitions.

26. What Is a Good Indicator? • Valid: Measures the effect it is supposed to measure • Reliable: Will give the same result if measured in the same way • Precise: Is operationally defined so people are clear about what they are measuring • Timely: Can be measured at an interval that is appropriate to the level of change expected • Comparable: Can be compared across different target groups or project approaches

27. Criteria for Indicator Selection • Consistent with project design—measure the desired result • Useful—contributes to project design, management, and evaluation • Available • Affordable

28. Steps in the Design Process

29. Step 5: M&E Plan Development • Fill in desired results (SOs and IRs). • Fill in selected indicators for each result. • For each indicator, determine source of information, frequency of data collection, and responsible agent.

30. M&E Plan Matrix

31. M&E Plan Creation • Matrix should be accompanied by text that describes how data collected will be used by the team regularly to monitor progress, make course corrections, etc. • Targets (if desired) will be set after baseline data are collected.

32. Steps in the Design Process

33. Step 6: Baseline Quantitative Data Collected • Can be done through KPC survey or other methods (e.g., health service stats, complete community registration) • Measures knowledge, use of services/current behaviors, or proxy of these (e.g., CYP) • Provides baseline values for indicators selected

34. Steps in the Design Process

35. MODULE 2 Situation Analysis: Secondary Data Review and Policy Scan Step 1 of the Design Process Components A and B of Situation Analysis

36. Steps in Design Process

37. What Are Secondary Data? • Secondary data are information resources that someone else has collected or developed. • They can provide current or recent information about health knowledge, practices and status of the population, types and distribution of health facilities, policies and protocols in place, quality of care, and community perceptions of health.

38. Advantages of Secondary Data • Readily available (usually free) • Provide an overview of the health situation, but they may not be exactly the same as the conditions in the specific project area • May identify social or policy issues that could contribute to the success or failure of a project • Offer a starting point for developing the background for a situation analysis.

39. Benefits and Limitations Benefits • Inform health project design with data pertinent to knowledge, demand, access, quality, and policy • Raise questions for further research. Limitations • May not be specific to project area, interventions, or target groups • May be outdated • Often do not reflect community perspective • Incomplete.

40. Tool for Synthesizing Situation Analysis Data (1 per IR)

41. MODULE 3 Situation Analysis: Participatory Qualitative Assessment Step 1 of the Design Process Component C of Situation Analysis

42. Steps in Design Process

43. Information from Qualitative Research • Answers: Why? How? When? What? Who? • Explains • Broad patterns of behavior • Motivations for behaviors • The how and why of knowledge and behavior • Provides in-depth understanding • Complements quantitative methods

44. What is Qualitative Research? • Any research that captures information and produces findings not reached by means of quantitative procedures. The art of learning and watching The art of asking why It is about… Answering questions raised by quantitative research

45. What Is Not Obtained? • Qualitative research does not answer— • How often? How many? How much? • Does not produce— • Frequencies, rates, averages, numbers of knowledge and behavior • Statistically significant information.

46. Limitations of Qualitative Research • Highly susceptible to subjective bias • Questions asked by interviewers • Interpretation of responses • Often applied and analyzed inappropriately • Flexible

47. Qualitative Methods, How Information Is Obtained • Open-ended questions • Techniques that encourage participation by all • Flexible methodologies that allow facilitator to change techniques in order to improve dialog or observation • Respecting local knowledge and capabilities • Use of a flexible guide to help the process

48. Steps to Analyzing Qualitative Studies • The challenge in qualitative analysis is to organize and categorize data effectively so they can be used, without compromising the data in the process. • Reading/Immersion • Identification of themes • Categorizing data by theme • Reducing • Interpreting

49. Pitfalls in Qualitative Analysis • Quantifying focus group information • Overemphasizing marginal or minority data

50. Participatory Qualitative Data Collection Tools • Observation • In-depth interviews • Focus group discussions • Visual techniques such as community maps, transect walks, body mapping, social or network mapping • Group ranking