Designing Influential Evaluations Session 3 Evaluation Design

1. Designing Influential EvaluationsSession 3Evaluation Design Uganda Evaluation Week - Pre-Conference Workshop 19th and 20th May 2014

2. Training Objectives By the end of the session, participants will: • Understand the use of programme logic models and theory of change • Understand how to develop evaluation questions Lewis Caroll, author of ‘Alice in Wonderland’

3. Logic models Logframe and Theory of Change

4. Simple results chain Management control diminishes Internal perspective External perspective Results Physical & financial resources Processes which turn Inputs into Outputs Operational changes (new skills or abilities, the availability of new products and services). Behavioural or institutional/ changes in response to the outputs Progress towards thematic priorities External factors become more important

5. How progress & success will be measured Objectives Indicators Verification Assumptions Goal(s) What How What the project seeks to achieve Purpose will be will it be Outputs measured? measured? Factors that may affect progress & success What the project will do & deliver Activities Inputs Costs Pre-conditions The resources required How much the project will cost A logframe is an ‘abstract’ of a project

6. The missing middle Goal Purpose Outputs Activities

7. Goal Purpose Response to project services Behaviour or system change! Outputs Activities Inputs Costs Response to outputs – the ‘miracle’! Objectives Indicators Verification Assumptions Critical conditions for success

8. Discussion • Turn to your neighbour or work in a small group • Discuss for 5 minutes: • What are the common weaknesses in development and use of logframes? • Summarise 2-3 points • Brief plenary discussion on common themes

9. Analysis of DFID logframesDrew & Albone, 2008; Agulhas, 2007 Problems: • Only 18% of performance indicators measured outcomes • 64% measured ‘inputs’ and ‘processes’ • Only 8% disaggregated data by gender • Indicators not specific and time bound • Measurement and the evidence base not well planned • Assumptions/risks not monitorable • Clear targets support better performance • Only 24% of projects with a partially clear Purpose scored 1 or 2 at OPR • 73% of projects with unclear Purpose scored <3

11. What is ‘Theory of change’? A theory of change is a set of statements about how change happens, and/or an attempt to describe underlying logic and assumptions

12. Key elements of a theory of change But no agreed format or convention

13. Theory of change for a primary education project Assume children can be released from other tasks Other contextual factors such as roads & transport Assume no barriers to children enrolling Assume training effective Parents respond to improvements and permit boys & girls to enrol Teachers apply new skills Enrolment improves Increase in teaching quality Attendance improves Teachers return to schools Improved facilities supplemented by materials & curriculum Boys & girls respond to improved quality Risk of teachers moving schools Assume building is of satisfactory quality Assume Min of Ed funds books & materials Retention improves Risk of children leaving for employment or family duties Build/ rehabilitate schools Teachers attend training Assume teachers can travel to training Mobilise PTA for building Select teachers Boys & girls literacy rates increase Assume selection on merit Decision to fund School needs assessment End Buildings Attendance Teachers Curriculum development Funds to Min of Ed. Start

14. Discussion exercise • Working in small groups, consider how a Theory of Change approach might affect the way you plan your evaluation? • Prepare your ideas for presentation and discussion in plenary.

15. Theory of change & evaluation criteria Evaluation questions

16. Formulating questions – from simple logframe for education 1 2 3 4 5 • What was the distribution of funding? How many staff were appointed? • How many schools received building projects? How many teachers were trained? How was the curriculum developed? • How much did access change? What aspects of quality were improved? • Did a change in net enrolment take place? • Did literacy rates improve? Funding & staffing School building, teacher training & curriculum development Improved access to & quality of schooling Increased net enrolment Increased literacy

17. Theory of change for a primary education project Assume children can be released from other tasks Other contextual factors such as roads & transport Assume no barriers to children enrolling Assume training effective Parents respond to improvements and permit boys & girls to enrol Teachers apply new skills Enrolment improves Increase in teaching quality Attendance improves Teachers return to schools Improved facilities supplemented by materials & curriculum Boys & girls respond to improved quality Risk of teachers moving schools Assume building is of satisfactory quality Assume Min of Ed funds books & materials Retention improves Risk of children leaving for employment or family duties Build/ rehabilitate schools Teachers attend training Assume teachers can travel to training Mobilise PTA for building Select teachers Boys & girls literacy rates increase Assume selection on merit Decision to fund End School needs assessment Buildings Attendance Teachers Curriculum development Funds to Min of Ed. Start

18. If … then … - the intervention logic Ifthe school needs assessment is effective Thenthe schools with the greatest need and potential will be chosen Ifteachers are selected on merit Thenthose with potential will be put forward for training Ifteachers are able to travel Thenselected teachers will attend training Ifteachers gain new skills Anddo not leave for jobs in other locations Andfacilities are improved by the building programme And improved facilities are complemented by teaching materials and the new curriculum Thenteachers will apply new skills If teachers will apply new skills Thenquality of teaching will improve Ifparents respond to improvements Andchildren can be released from other tasks And there are no socio-cultural barriers to boys and girls enrolling Thenenrolment of boys and girls will increase Ifboys and girls respond to improved quality Thenattendance will increase

19. Hypotheses about education improvement

20. Formulating questions 1 2 3 4 5 • How were schools selected for the project; what was known about their enrolment and literacy status in advance? • Who was selected for teacher training; how effective was it; how many teachers were retained afterwards? • How much did access change and for who; what aspects of quality were improved and against what standard? • Did a change in net enrolment take place? Was it the result of the outputs delivered under this project or did other factors such as improved roads and transport, or demographic changes affect the outcome? • Did literacy rates improve? For both boys and girls? How widespread or consistent have these changes been. Do they reflect specific locations or a general trend? Funding & staffing School building, teacher training & curriculum development Improved access to & quality of schooling Increased net enrolment Increased literacy

21. OECD-DAC Evaluation criteria

22. Relate to evaluation criteria Economy Efficiency Effectiveness Impact Funding & staffing School building, teacher training & curriculum development Improved access to & quality of schooling Increased net enrolment Increased literacy

23. Developing questions

24. Discussion exercise • Working in small groups, consider at what stage evaluation questions should be developed and whose responsibility that should be? • Prepare your ideas for presentation and discussion in plenary.

25. Summary • Analyse the intervention logic using a logic model such as the logframe or (better) theory of change • Develop evaluation questions from the details of the intervention logic • These provide the basis for considering an approach and methods • An evaluation framework can help link questions to approach and methods

26. End