ACP S&T Grant Contract N° FED/2009/217058

1. Objective verifiable Indicators (OVI) and Smart Indicators of CoBaSys results Dr Sarah Bracking WP6 Leader University of Manchester ACP S&T Grant Contract N° FED/2009/217058

2. Objectives • To report on the philosophical approach to monitoring and evaluation • Introduce a Multi-level accountability framework • Describe role of OVIs in this M&E system • Generating Objective verifiable Indicators (OVI) and Smart Indicators • Plan of action

3. Monitoring and Evaluation • Thin approaches have origins in quality control, which was normally carried out by comparing ‘results’ to externally generated standards, such has those used in manufacturing or licensing • From 1980s monitoring and evaluation became a thicker process, where process as well as results measured. Incorporated into new public management models and performance measurement. ‘Objective verifiable indicators’ emerged as its core tool • From late 1990s M&E became yet more holistic as applied to greater areas of public life, many of which were qualitative • Impact assessment and ‘change’ monitoring emerged where inputs, process, outputs were related to measures of impact for various interventions

4. Multi-level accountability framework approach to M&E • But form a philosophical perspective we can ask an a priori question of why we are doing M&E. The answer lies in establishing accountability to multiple stakeholders • A multi-level accountability framework incorporates • Internal accountability by work programme [WP and partner specific OVIs; project level OVIs] • Accountability to ‘project affected persons’ [process and ethical issue] • Accountability to target population in terms of overall project goals

5. Project partners, WP level accountability • and partner relational accountability • Communities in which we work, • Quality of work done, ethics, representational, behavioural • Project level accountability • To objectives of the project as a whole • To interests of target group implicated • To scientific criteria of a CoBaSys • To funders Levels of accountability

6. Three levels or scales of assessment • Internal and project level M&E is the substance of the first deliverable for WP6, and within this we will have a framework of indicators. We need to choose these – more below • To establish accountability to our second level, the slightly wider community of project affected persons, WP6 will produce a project brief 2. • This will be a field instrument for use on monitoring visits • To establish the third or outer level of accountability (project brief 3) an evaluation of how the project will meet its obligations to the wider public or target population will be done

7. Accountability Cycle: multi-stakeholder UoM draft Project brief 1, Circulated to all WPs for suggestions WP Leaders suggest OVIs for own projects, in line with objectives WP6, with UNIMORE and UoH suggest OVIs for project as a whole underwrite liability WP specific data provided by WP leaders WP6 personnel make field visits WP specific field reports In project brief 2 and 3 additional accountability levels provided for PAPs and programme target group as a whole Project brief 3 provides M&E summary for ‘project as a whole’ against 3 accountability levels

8. For level 1: Principles of indicator setting for internal accountability • Should assist in performance management • Can be qualitative, quantitative or composite • Can be input, output, process or impact indicators • Differ by proximity, can be direct or indirect. Most in practice are proxy indicators • Where appropriate can be adopted from external ‘authorities’, as international standards [such a qc in manufacturing or qualifications] • Or related to a standard for a sequence of like events which should perform to a template [such as our focus groups] • Or context specific

9. SMART and SPICED indicators • A number of approaches to selecting project indicators • SMART criteria are oriented towards enhancing the speed and ease of data collection • Specific to the intended changes • Measurable and unambiguous • Attainable or achievable by the activity • Relevant to the activity • Time bound • SPICED criteria give particular emphasis to participatory approaches to development. • Subjective in that informants are uniquely placed to offer insight based on their experience • Participatory, involving these informants • Interpreted or explained to provide an understanding of the local context in which they occur • Cross-checked against other indicators, informants and methods • Empowering to affected groups • Diverse in nature and measured from a variety of informant groups

10. Indicators cont: • The system preferred for indicator selection is: • CREAM criteria focus on managing for development results: • Clear: Precise and unambiguous • Relevant: appropriate to the subject at hand • Economic: available at a reasonable cost • Adequate: provide a sufficient basis to assess performance • Monitorable: amenable to independent validation • The most significant change technique • When using the most significant change technique beneficiaries and/or staff are asked to report the most significant changes (positive, negative, planned, unplanned) over the last period and explain why they have identified these as significant. This approach allows different stakeholders to identify those changes that are important to them, and these can then be compared with the original indicators established for the Activity. The most significant change technique should be used to complement one of the other techniques. • See NZAID Tools Activity Cycle Tools, online

11. OVI’s for ‘whole project’

12. Map of research projects: meeting OVIs for ‘project as a whole’? Policy advocacy Primary research Institutional capacity analysis, UEM RWP8, UDSM, field research, Iringa RWP9, UoB, field research, Old Naledi Focus groups, UNIMA, TARSC, REACH, use of field protocol, Comparative research platform, Ideally: ZW, MAL, MZ, TZ, BOT, NAM UNIMA, policy analysis RWP12 RWP10, UNAM field research, Caprivi RWP11: Institutional capacity analysis, UoH, UNIMORE, expats RWP13, IDS, field research, policy work Partners to do focus groups in country platforms alongside triangulation from own research sites?