Designing Influential Evaluations Session 7 Commissioning evaluations

1. Designing Influential EvaluationsSession 7Commissioning evaluations Uganda Evaluation Week - Pre-Conference Workshop 19th and 20th May 2014

2. Framework for commissioning evaluations Planning and scoping for evaluation, this session explores a framework for working towards a TOR: • Evaluabilityassessment • Working to timeframes • Looking at budgets • Deciding who the evaluators will be • Tendering processes

3. Evaluability assessment • Is a systematic process • describing the structure of a programme • analyzing its plausibility and feasibility for achieving objectives • is there a theory of change; could one be retro-fitted? • assessing suitability for in-depth evaluation • appropriateness to programmemanagers, policymakers and programme implementers

4. Evaluation timeframes Adequate time for commissioning and doing evaluation Consultations with all parties to ensure TOR is realistic Consider length of programme & time needed to conduct formative as well as summative evaluation Be realistic and open to change

5. Evaluation budget Value for money a key concern Underfunding as wasteful as over-funding Balance between cost and quality Quality ultimately more important But also relevance for purpose Make sure all aspects adequately funded including consultation with stakeholders, reporting and dissemination Ensure evaluation design appropriate to budget as well as aims of programme

6. Open Closed Open and closed tenders Advantages • a range of expertise • keeps cost competitive • stimulates providers • brings in the new • reduces complacency • But... • inappropriate suppliers • possible selling rather than evaluation skills • time and cost consuming Advantages • fewer suppliers • time to fine tune brief with selected contractors • commissioners build on previous knowledge • less time & money on selection of evaluation team • But… • lack of dialogue w/potential others • does not encourage new ideas/evaluator

7. Against For Talking money: revealing the budget • tenderers put in a price just below the budget • collusion of tenderers could lead to inflation of the budget requirement • commissioners must have clear evaluation design in mind to realistically assess proposals • no evidence that any of these happen • better to make informed decisions • time-wasting to ‘guess’ • budget-ceilings facilitate quality proposals • a range of budgets allows for creative proposals

8. Principles of good commissioning Clarityof purpose – in the brief and expectations; aims, objectives and outputs Accessibility - of language, range of skills, the tender process and reporting Participation - collaboration between those engaged in research enterprise Flexibility - in design (scale of proposal matched to funding), in methodology, in process (to adjust to field/policy changes) and scope of study Integrity - of relationships, trust, honesty empathy, valuing different perspectives. Transparency in open tendering, declared budgets or ball park figures, management and reporting procedures

9. The terms of reference • Background • Linkages and synergies • Approach paper • Purpose, objectives, scope • Existing sources of information • Methodology • Level of detail and guidance? • Skills and qualifications • Logistics and procedures • Outputs • Communication plan • Study report • Reporting & contracting arrangements • Budget • Annexes • Steering groups/ advisors; ethics; standards; arrangements for QA

10. Task • Draft the following elements of a terms of reference for the final evaluation of a project • Purpose, objectives, scope • Approach & methodology to be used • Skills and qualifications of the team • Outputs • Communication plan

11. Programme 14 Information and National Guidance Objectives: • To regulate, coordinate and disseminate government policies, programmesand projects to the public for economic and social accountability. • To develop and nurture a National Value system based on a shared National Vision, mindset change, beliefs and perceptions as well as popularising government policies and programmes to create nationwide awareness for socio-economic transformation. Outputs: • Meetings with various stakeholders in the media industry • National and International events publicised; • Public officials sensitized on ATIA; • Leaders and members of the public sensate • In and out of school youths mobilised and sensitized; • The Barazas initiative supported • Carry out cadreship training courses; • Identify land for the regional training institutes. • Field research, Pre-sensitization surveys and evaluations on national activities carried out.

12. Project 1078 Karamoja Integrated Development Programme (KIDP) Objectives: • To consolidate peace and security for the people of Karamoja and strengthen coordination and harmonization of interventions both at national and regional levels. Outputs: • Housing facilities for people in Karamoja improved. • Food and nutrition security for the poor and vulnerable households improved. • Crop and livestock production and productivity increased. The quality of education in Karamoja improved. • Clean and safe water for human consumption and for production facilities for multipurpose use provided.

13. TASO Support to OVC This project is trying out new ways of providing support to OVC Objectives: • Children continue in school through to complete secondary as a result of receiving appropriate support • Children who drop out or do not progress at school qualify through apprentice training • Individual OVC and their families become self-sustaining after eduction Activities: • Selection of the most needy orphans and vulnerable children to participate in the programme • Training given to parents, guardians and teachers in working with and caring for OVC • School equipment and medical supplies provided for the children • Counselling provided for OVC at school

14. NAADS 25-year programme, with an initial phase of 7 years; first 2 years for testing of programme approaches and concepts leading to a refinement of the design. Objectives: • To promote market oriented/commercial farming (farming as a business) • To empower subsistence farmers to access private extension services, technologies and market information • To create options for financing and the delivery of appropriate advisory and technical services for different types of farmers • To shift from public to private delivery of advisory services in the first five years of implementation • To develop private sector capacity and professional capability to supply agricultural advisory services Activities: • Deepening decentralization • Market oriented farming • Managing natural resource productivity • Harmonization of public extension oriented programmes • Development of Public Private Partnerships(PPPs) for agribusiness and market linkage Target groups: • The principle beneficiaries are the poor farmers and in particular women, youth and people with disabilities.

15. SHG/MFI Programme Objectives: • Job creation and income generation for women formed in self-help groups • Empowerment of women • Improvements in family welfare Activities: • Formation of self help groups • Groups savings and loan schemes • Access to credit • Business training • Value chain analysis, market linkages, vocational training

16. End