Evidence for Agile Policy Makers The Contribution of Transformative Realism

1. Evidence for Agile Policy MakersThe Contribution of Transformative Realism • Graham Room • 14 February 2014 • Victoria University Wellington

2. Evidence for Policy • Assess impacts of different interventions by systematic review of evidence • ‘What Works Centres’: • National Institute for Health and Clinical Excellence. • Education Endowment Foundation • Centre for Local Economic Growth

3. X Y

6. The Hierarchy of Evidence and Rigour • 1a Evidence from a systematic review of randomized controlled trials • 1b Evidence from at least one randomized controlled trial • II a Evidence from at least one study without randomization • II b Evidence from at least one other type of quasi-experimental study • III Evidence from non-experimental descriptive studies (e.g., case control studies, correlation studies) • IV Expert consensus

7. Pawson’s Realist Critique • Pawson, R. (2006), Evidence-Based Policy: A Realist Perspective • ‘Impact’ (medical) versus ‘engagement’ (social policy) • Interventions unfold in institutionally complex environments • Rarely is it the ‘same’ intervention that is delivered across all contexts

8. Pawson’s contingencies of intervention

9. Darwin’s Tree of Life

10. ‘Transformative realism’

11. Three Paradigms of Evidence-Based Policy-Making • EBPM: Evidence of impact of a given intervention: universal and unconditional • Pawson: Impact of a given intervention under a range of contingencies • Transformative realism: Dynamic synergies among interventions

12. The Choice of Paradigms

13. Types of Knowledge • Paradigm 1: Well-evidenced generalisations about impact (universal truths?) • Paradigm 2: Well-based theory as to out-turn under specific local contingencies • Paradigm 3: Need real-time watching and learning-by-doing

14. Real-Time Watching and Learning-by-Doing • How to tune social developments in real time? • Where to probe and where to avoid? • What signs of tipping points to avoid or approach? • What data in real time? Big Data? • Ian Stewart (1997), Does God Play Dice?) • Tuning a complex system (engineering and medicine) • Can this approach be applied to social systems?

16. Real-Time Watching and Learning-by-Doing • How to ‘read’ the ‘sea conditions’? • Networks of ‘navigators’ – tracking how policy ecosystem is developing and deciding what ‘tuning’ is needed • Elizabeth Eppel, Amanda Wolf, David Taylor: Complex Policy Implementation: The Role of Experimentation and Learning • ‘Ordinary’ situations - rule of thumb • Anomalous (extraordinary) situations – examine carefully and devise a specific strategy • What thresholds to take as triggers for concern?

17. Concluding Reflections • Danger of de-politicisingpolicy • What works for whom? • Corporate and other interests try to shape our policy futures: is there any shared national interest? • Good research on impact is necessary but not sufficient for good policy

18. References • Behavioural Insights Team (2012), Test, Adapt, Learn, UK Cabinet Office • Eppel, E., Turner, D., & Wolf A. (2011). ‘Complex policy implementation: The role of experimentation and learning’. In B. Ryan & D. Gill (Eds.), Future State: Directions for public management in New Zealand (pp. 182-212). • Pawson, R. (2006), Evidence-Based Policy: A Realist Perspective • Room, G. (2011), Complexity, Institutions and Public Policy: Agile Decision-Making in a Turbulent World • Room, G. (2013), 'Evidence for Agile Policy Makers :The Contribution of Transformative Realism', Evidence and Policy, 9(2): 225-244. • Stewart, I (1997), Does God Play Dice?