Module 5: Monitoring the Policy Process

Tools for Civil Society to Understand and Use Development Data: Improving MDG Policymaking and Monitoring Module 5: Monitoring the Policy Process

What you will be able to do by the end of this module • Know the key institutional issues with respect to monitoring • Understand the main types of monitoring indicator • Explain best practice monitoring principles

The Challenge of Monitoring • In many countries the monitoring system is still in its infancy • The institutional arrangements are often not yet functioning and suffer from capacity problems

Capture the multidimensionality of poverty in the monitoring system • The multidimensionality of poverty is increasingly recognised • Broaden scope from a narrow focus on reducing the percentage below the poverty line • The MDGs have helped to ensure commitment to a broader range of goals • Participatory research can broaden the poverty reduction agenda

Some key principles in monitoring systems • Country owned and led • Enhance accountability (parliament etc) • Medium-term perspective is important • Enhance coherence across government by reducing fragmentation of project monitoring • Focus on indicators and targets that have been agreed with a variety of stakeholders

Using the Results from Monitoring • Findings should be disseminated to a variety of groups: policymakers, civil society, media, academics etc but… • Experience suggests that there is a limited demand for monitoring data in government • Monitoring agencies will therefore need to develop goodmarketing and communications skills • Tailoring feedback to the specific interests of targetedstakeholders is clearly important

Institutional Arrangements for Monitoring • Institutional arrangements need to be carefully thought out to ensure flow of information • No blueprint – the key difference between systems is the power given to various bodies (i.e. leadership or co-ordination?) • Non-governmental stakeholders to be included • Links between agencies need to be embedded

Much needs to be done in monitoring public expenditure • Tracking the share of the budget that goes to priority sectors, and identifying targets, is poorly developed in many countries • The link between expenditure tracking and other data is often missing in countries without a developed monitoring system • Public expenditure review processes have been closely linkedto the PRS in some countries

More Monitoring Challenges • Difficult to build up demand for information in civil society • Agreement of national level targets and indicators may conflict with donor requirements for measuring progress • National monitoring institutions and systems sometimes lack capacity and donors set up parallel systems

Centralised or Decentralised Responsibility for Monitoring? • Not a question of centralising all information gathering and analysis in one institution • Monitoring system will always need to involve a network of institutions • The key question is whether it is strategic to have a well-placed unit that assumes responsibility for making things happen or • Having a secretariat to a committee with responsibility being dispersed

Alternative Institutional Models for Monitoring • One approach is to place monitoring close to the Ministry of Finance so that links with budget maintained • But in most countries monitoring is located away from the Ministry of Finance, often in the statistical bureau, planning agency or president’s office

Definition and Classification of Indicators Indicators are instruments that show the status and tendency of a given phenomenon, and they are used to show progress toward MDG targets. There are two broad groups of indicators: • Intermediate • -Input, output • Final • -Outcome, impact

Intermediate Indicators • An indicator that contributes to an outcome or impact is called an intermediate indicator • There are two types of intermediate indicators: input and output indicators • Input indicators measure the financial (e.g. money spent on new classrooms) and physical resources, as well as the processes and human resources needed • Output indicators measure the goods and services produced by the input (e.g. number of new classrooms)

Do not neglect intermediate indicators – the missing middle • Inputs, and especially outputs, are sometimes missing from monitoring systems • Final indicators change slowly but… • Input and output indicators generally change more quickly in response to government action • They enable policies and programmes to be tracked more effectively • However, there also needs to be greater linkage all along the chain from inputs to impacts

Final Indicators • An indicator that measures the effect of a policy or programme on well-being is called a finalindicator • There are two types: outcome and impact indicators • Outcome indicators measure access to, or satisfaction with, public services (e.g. primary school enrolment rate) • Impact indicators measure the dimensions of well-being that are the goals of public policy (e.g. literacy rate)

The framework needs to establish clear monitoring chains Intermediate Final Input Output Outcome Impact Financial, physical resources Goods and services produced by inputs (classrooms built, textbooks provided) Access to, use of, and satisfaction with services (enrolment, dropout rates) Effect on dimension of well-being (literacy)

Quantitative and qualitative indicators • Quantitative indicators use only numbers to describe a phenomenon - counts (the number of children immunized), - continuous measures (the proportion of the population below the national poverty line) • Qualitative indicators are used to describe intangible characteristics that are difficult to measure in numeric terms

Quantitative Indicator • May be computed from other indicators • New indicator measures a distinct phenomenon Example: compute the ratio of girls to boys in primary, secondary and tertiary education, 1. the number of children enrolled must be disaggregated by sex, 2. ratio of two resulting values computed. The new indicator measures the degree to which girls and boys have equal access to primary education – something quite different than the simple count of girls and boys enrolled.

Qualitative Indicator • Express a range of non-numeric values such as “Good”, “Satisfactory” and “Poor”, which represent the respondent’s estimations of degree • Often are more complex and may be derived from several types of information and measure several dimensions of a phenomenon Example ‘Satisfaction with clinic services’ = combined result of success of treatment + distance traveled to reach the clinic + kind and quality of facilities provided.

Cross-classification of Indicators QUANTITATIVE QUALITATIVE INPUTExpenditure on primary education Adequacy of curriculum OUTPUTNumber of primary school teachers Quality of teaching atmosphere in the classroom OUTCOMEEnrollment and dropout ratesSatisfaction with teaching methods IMPACTLiteracy levelChange in perception of empowerment and poverty status

Why are MDG indicators exclusively quantitative? • Purpose of the MDG indicators is to measure progress toward set targets, a function that implies repeated assessments of where a country stands at given points in time • Indicators are useful tools only if we can attach value to them • Even when well defined, the information indicators provide may not be objective, which reduces their credibility • MDG indicators are exclusively quantitative precisely to minimize the ambiguity

Summary • It is important to capture the multidimensionality of poverty in the monitoring system • MDGs are the easy tool for civil society to monitor and discuss progress in social development • Monitoring is in its infancy in many countries and national capacity is weak • Monitoring public expenditure needs to be developed in most countries • Two groups of indicator: intermediate and final • Do not neglect intermediate indicators

Practical 5 • 1. Describe the monitoring system of your country’s medium-term plan or strategy (PRSP if there is one) in terms of: • Institutional Arrangements • Scope • Comprehensiveness • Involvement of Civil Society • 2. What are its strengths? (Identify three) • 3. What are its weaknesses? (Identify three) • 4. What progress has been made in monitoring public expenditure/budget execution? • 5. Are the outputs from monitoring and evaluation disseminated and used? How are they used?

Module 5: Monitoring the Policy Process