Evaluation – Principles and methods

1. Evaluation – Principles and methods Matt Barnard Head of Evaluation NSPCC

2. Purpose of evaluation • Definition “Examine how a policy or intervention was designed and carried out and with what results.” (Magenta Book) • Asks objective questions • What were the impacts? • How was it delivered? • What were barriers and facilitators? • Did it deliver value for money? • Aims to provide • ‘Scientific’ basis for policy making

3. Evaluation Design Process Evaluation Design Evaluation implementation Evaluation findings Intervention Design Logic model Intervention implementation planning Intervention implementation

4. Logic models • Characteristics • Mechanisms not processes • Key steps not every step • Explanatory not descriptive • Reflects theoretical assumptions • Benefits • Sense check • Identifies realistic outcomes • Facilitates evaluation design

5. Strength of design Strength of design matrix

6. Strength – Evaluation design Randomized controlled trial Quasi-experimental design Before and after measures

7. Types of design • RCT • Individual randomization • Cluster randomization/ roll out • BAU/waiting list/alternative services • Quasi-experimental designs • Matched area/ groups • Matched individual • Interrupted time series • Regression discontinuity

8. Factors influencing methodology • Intervention stage of development • Early exploration • Defined and established but not proven • Transferability • Potential Costs and benefits • Resources • Timescales

9. Key principles • Clarity about key question • Avoid ‘default’ questions • Methods matched to question • Ensure methods match desired questions • Claims match evidence • Avoid over-claiming • Have a coherent story to tell