Monitoring and Evaluation: Information Sources and Systems - PowerPoint PPT Presentation

monitoring and evaluation information sources and systems n.
Download
Skip this Video
Loading SlideShow in 5 Seconds..
Monitoring and Evaluation: Information Sources and Systems PowerPoint Presentation
Download Presentation
Monitoring and Evaluation: Information Sources and Systems

play fullscreen
1 / 81
Monitoring and Evaluation: Information Sources and Systems
98 Views
Download Presentation
renate
Download Presentation

Monitoring and Evaluation: Information Sources and Systems

- - - - - - - - - - - - - - - - - - - - - - - - - - - E N D - - - - - - - - - - - - - - - - - - - - - - - - - - -
Presentation Transcript

  1. Monitoring and Evaluation: Information Sources and Systems

  2. Session Objectives At the end of this session, participants will be able to: • Name the main information sources for PHN M&E • Describe the main strengths and weaknesses of different data sources • Discuss the main data-quality issues that need to be considered • Explain why complementary data sources are often required to monitor and evaluate health systems • Identify potential data sources that might apply in a specific program context.

  3. Overview • Types of information • Strengths and weaknesses of selected data sources • Data quality • Linking data sources • Exercise

  4. The Finagle’s Laws of Information The information you have is not the information you want The information you want is not the information you need And the information you need is usually not available

  5. Definitions (1) • Data: the raw facts that are collected and form the basis for what we know. • Information: the product of transforming the data by adding order, context, and purpose • Knowledge: the product of adding meaning to information by making connections and comparisons and by exploring causes and consequences

  6. Definitions (2) • Health system “all resources, organizations and actors that are involved in the regulation, financing, and provision of actions whose primary intent is to protect, promote or improve health.” (WHO, 2000) • Program A set of procedures to conduct activities. The objective is normally the solution to a problem Neither a health system or program is a static phenomena. They experience a continuous process of changes due to pressure from both outside the system and from within the system.

  7. Definitions (3) • Health Information System (HIS): A health-information system (HIS), similar to a health management information system (HMIS) “…a system that provides specific information support to the decision-making process at each level of an organization” (Hurtubise, 1984) • Data Systems “a way of talking about the whole set of M&E indicators in a performance monitoring-and-evaluation plan, and all of the data and other information that need to be gathered and understood in an orderly fashion that makes sense and help in program management and implementation”

  8. Types of Information • Surveillance • Epidemiological • Behavioral • Routine service reporting • Special program reporting systems • Administrative systems • Vital registration systems • Facility surveys • Household surveys • Censuses • Research and special studies

  9. Frequency of Data Collection • ROUTINE or continuous data collection • NON-ROUTINE or periodic data collection

  10. Routine Non-Routine Classify the previous information types by frequency of data collection

  11. Frequency of Data Collection • ROUTINE or continuous data collection • Health facility-based (patient information and service statistics) • Community-based (service-statistics) • Program-based (administrative) • Vital registration • Sentinel reporting/demographic surveillance • NON-ROUTINE or periodic data collection • Household or facility-based surveys • Population census • Rapid-assessment procedures (RAP) • Special studies/research

  12. Geographic System Levels • National • Sub-national (e.g. district) • Program area

  13. The Health Information System: Data for Planning, Monitoring and Evaluation in the PHN Sector Routine Non-Routine TYPE USE Aggregated Service Statistics Aggregated Mgmt Data Aggregated Surveillance Data Financial Data Vital Registration Systems Policy-Making Strategic Planning Program Tracking Disease Surveillance Technical & Logistical Support National Level Population-based surveys e.g. DHS Facility-based surveys e.g. Situation Analysis, SPA Rapid Assessment Methods Aggregated Service Statistics Aggregated Mgmt Data Sentinel Sites Observation Checklist Self-Evaluation (e.g. COPE) Planning (Access) Management (Quality/Efficiency) Supervision (Performance) Disease Surveillance District Level Special Studies e.g. EPI cluster surveys, KAP studies, etc. Facility/ Client Client Records Financial Records Supply Records Facility logbooks/data records Aggregated Community Data Client Mgmt and Follow-Up Health Unit Management Work Planning/Priority Setting Census Birth and Death Records School Records CBD logbooks Drug Revolving Fund records Client Mgmt and Follow-up Supplies Management Community Awareness Community

  14. Data-Collection Levels • Policy or program • Service environment • Client • Population • Spatial/geographic

  15. Data Sources at the Policy/Program Level • Official documents (legislative, administrative) • National budgets or other accounts data • Policy inquiries • Reputational rankings • Program effort scores

  16. Trends in Family-Planning Effort Score: 1972-1999

  17. Data Sources: Service Environment Level • Administrative records • Service statistics • Management information • Financial data • Service-delivery point • Routine service statistics • Audits/inventories • Facility surveys • Agent, staff or provider • Performance, competency • Training records

  18. EXAMPLE: SERVICE ENVIRONMENT LEVEL DATA

  19. HEALTH MANAGEMENT INFORMATION SYSTEMS (HMIS) • Note: An important way of monitoring routine data over time is through a Health Management Information System. An HMIS is a system for ongoing (routine) collection and reporting of data about client-service delivery. In many countries, this system operates at the national level. Ideally, these routine data are collected from a comprehensive set of service delivery points, and should cover topics such as: – Costs – Stockouts – Births – Mortality – Morbidity – Numbers of clients seen, referred (inpatient; outpatient) – Numbers of clients by types of service

  20. Data Sources: Client • Client-exit interviews • Case surveillance • Epidemiology of disease • Provider-client observation • Management of the sick child • Vendor-client interaction • Contact or visit registers • Customer record

  21. Data Sources: Population • Census • Vital registration system • Sample household surveys • Special population surveys • Demographic (elderly, youth) • Risk groups (CSWs, MSMs, IDUs) • Occupational (farmer, skilled labor) • Area-based (catastrophe-affected) • Biomarkers

  22. Spatial/Geographic Data Sources • Satellite imagery • Aerial photography • Digital line graphs • Digital elevation models • Cadastral maps • Global Positioning System data • PLACE (site-based surveys)

  23. Aerial Photography Digital Elevation Models Integrated GIS Database Satellite Imagery Cadastral Data Digital Line Graphs GPS Data Chris Betz 1757 Millbrook Ln 28226 Y 2 Christian Carl 1761 Millbrook Ln 28226 Y 1 Chris McAfee 1765 Millbrook Ln 28226 Y 2 Dale Legere 1776 Millbrook Ln 28226 N 6 Donna Black 1780 Millbrook Ln 28226 Y 2 Demographic Data Differential GPS

  24. High Transmission of HIV Guguletu, Cape Town South Africa Carolina Population Center University of North Carolina at Chapel Hill Neighborhood Statistics Showing High Transmission Sites within 500 Meters 1 - 13 14 - 25 Air Photo Showing Potential High Transmission Sites in Guguletu 26 - 37 Individual Structures Can be Identified 38 - 49 50 - 61 No Data

  25. Strengths and Weaknesses of Selected Data Sources

  26. Focus M&E Data Sources • Facility-based routine information systems • Facility surveys • Population-based surveys • Program records/administrative data

  27. Facility-Based RHIS: Types of Information Generated • Service statistics • Outcomes of health interventions if individual patient records kept • Not coverage (but can be estimated in some cases with other data) • Not incidence (except nosocomial infections) • Not prevalence

  28. What is Wrong with Existing RHIS? • Irrelevance of information gathered • Poor data quality • Duplication and waste among parallel health information systems • Lack of timely reporting and feedback • Poor use of information • Centralization of information management without feedback to lower levels

  29. Strengths of Routine Health Information Systems • Continuously collected – suitable for frequent reporting • Existing system – no new data collection; builds local capacity; sustainability • Typically available at lowest administrative levels (e.g. district, facility) • Integral part of health system – direct link to health system actions

  30. Common Problems With Facility-Based RHIS • Variation in quality and completeness of reporting • Timeliness of reporting • Difficulty of providing coverage estimates • Indicators may not be exactly what you want in a particular context • May only cover government facilities • Double-counting

  31. Facility Surveys: Types of Information • Readiness to provide services (inventory) • Infrastructure, staffing, hours of operation, fees • Health worker knowledge • Provider interviews • Quality of Care • Client-provider observation • Client satisfaction • Exit interviews

  32. Strengths of Facility Surveys • Can cover both public and private health facilities • More detailed information than is typically available in routine systems • Can be tailored to specific program needs • Timing can coincide with program implementation • Can combine with population survey for outcome monitoring and impact evaluation • Quality control may be easier than in routine systems

  33. Limitations of Facility Surveys • Survey sampling design and analysis may be complex • Expensive, time-consuming • Stand-alone – sustainability concerns; less connected to ongoing program decision-making • Information rapidly outdated, unless repeated – not available regularly • Coverage/sample size constraints • National vs. sub-national • By type of facility • Small client sample sizes for some services (e.g. FP, STIs)

  34. Facility Survey Initiatives and Tools • Service Provision Assessment (SPA/HSPA) – DHS • Service Availability Mapping (SAM) – WHO • Quick Investigation of Quality (QIQ) – M/Eval (FP only) • Situation Analysis (SA) – Population Council (FP only) • JICA facility surveys and mapping

  35. Population-based surveys: Types of information collected • Knowledge and attitudes • Practices • Coverage

  36. Strengths of population-based surveys • Representative of the general population – no selection bias • Wide range of outcome-level indicators can be collected • Program coverage • Well-tested instruments; quality control

  37. Limitations of population-based surveys • Coverage; national versus sub-national – not suitable for district-level estimates • Frequency; typically only conducted every 3-5 years. • Cannot detect small changes or changes over short periods of time without large sample sizes (expensive) • Not suitable for some types of information (e.g. retrospective attitudes – recall bias)

  38. Household survey programs (national) • Demographic and Health Surveys (DHS) • CDC Reproductive Health Surveys (RHS) • UNICEF Multiple Indicator Cluster Surveys (MICS) • PVO Knowledge Practices and Coverage Survey (KPC) (not national) • CDC Young Adult Reproductive Health Surveys (YARHS)

  39. Class Activity (1) How to improve facility-based routine information systems in developing countries?

  40. Class Activity (2) What are the determinants of health information systems performance?

  41. Data Quality

  42. Overview • Types of information • Strengths and weaknesses of selected data sources • Data quality • Linking data sources • Exercise

  43. Data Quality Issues • Coverage • Completeness (census, sample) • Accuracy – measurement error; biases; double-counting • Frequency of collection • Reporting flow • Reporting schedule • Public accessibility • Supervision

  44. Hierholzer (Am. J. Med 1991; 91; 21-26) has called data the Researcher’s (M&E expert) sand. A lens maker takes sand, refines it, melts it, and through a long process of grinding and smoothing, fashions a lens with which to see the world more clearly. Similarly, a M&E expert takes data, refines it and smoothes it until a clearer picture of nature is revealed. If the sand is dirty or impure, the lens will be cloudy and distorted. If data is unreliable or invalid, the M&E expert’s understanding of nature will be clouded and distorted.

  45. By paying close attention to the data collection process from the conception of the data collection document through the editing of the data after it is collected, the M&E expert help keep his “sand” pure so that, in the end, nature may be viewed with much clarity and possible • No amount of sophisticated analysis can salvage either a poorly designed or a badly carried out study

  46. Linking data

  47. Linking data • Data can be linked from different sources, across different levels, or over time • Linking data appropriately requires planning, preferably prior to data collection • Understanding linked data can provide depth and continuity to enrich otherwise discrete points of information

  48. Linking Data Why link? • Survey data sets (e.g., household and facility information) can be linked to compare services available and health outcomes across geographical units • Geographical and survey data can be linked to examine the effects of physical attributes on service utilization • Time series and panel data can help build causal explanations of program or project effects Why not link? • May not be necessary for a given program in a given context • Improper methodology can confuse issues more than explain them • Analyzing linked data more appropriate for • evaluation than monitoring

  49. Linking Data Examples • Population and facility data can be linked to ascertain health outcomes correlated with service availability, training, or quality of care (e.g. % of live births in catchment area attended by a trained personnel or % of women exclusively breastfeeding until 6 months among women going to facilities where provider training took place.) • Facility and client data can be linked to learn about program expenditures per new family planning acceptor • Facility and staff data can be combined to provide information about the proportion of clients per provider or the proportion of doctors per facility

  50. Population-based data highlight limitations in HIS