Problematizing Evidence-based Policy-making

1. Problematizing Evidence-based Policy-making Mark Evans Director, Institute for Governance and Policy Analysis

2. What is evidence-based policy making? • The latest rediscovery of evidence-based policy-making may be viewed as part of a longer historical search for usable and relevant knowledge generated through rational scientific methods to help address and resolve social problems. • This quest dates back to the enlightenment but finds its modern expression in the work of the Webbs at the turn of the 19th and 20th centuries and in the rational model of decision-making developed by Herbert Simon (1945) in the seminal Administrative Behavior (New York: Macmillan). ...rational decision-making involves the selection of the alternative which will maximise the decision maker’s values, the selection being made following a comprehensive analysis of alternatives and their consequences (Simon 1945: 1).

3. The Blair government in the UK’s (2001) Better Policy making mapped out an evidence based approach to policy driven by Health Sciences. Led to the establishment of the ESRC UK Centre for Evidence Based Policy-making and Practice at Queen Mary College, University of London and even an academic journal (Evidence and Policy). Blair’s approach clearly influenced Rudd; ‘Evidence-based policy making is at the heart of being a reformist government’ (Rudd, 2007). The aspiration for evidence-based policy-making in Australia is ‘to produce the knowledge required for fine-tuning programs and constructing guidelines and ‘tool-kits’ for dealing with known problems’ (Head, 2008). Hence, the currently famous phrase that defines much of the movement – ‘what works?’ ANZSOG has recently followed suit with the announcement of the launch of a new journal, Evidence Base, under the editorship of George Argyrous at the University of New South Wales. The “What Works” Movement

4. Institute’s research in this area • Seeks to problematize the quest for evidence-based policy making by bringing the politics back in through the study of: • Critically Examining the notion of Policy Success • Critically assessing attempts to enhance strategic policy capability (Edwards, Evans and Scott)

5. Policy Success? • Given importance of this issue, under-studied. • Almost exclusively focuses on the programmatic dimension – this is the dimension behind the mantra of evidence-based policy making – need to assess whether policy does (if assessment of existing policy) or is likely to (if proposed policy) achieve its aims. • However, there are other dimensions of policy success – and significant problems of interpretation

6. Dimensions of policy success: an heuristic • PROCESS – Did it pass with limited opposition/amendment? • PROGRAMATIC – Did it achieve its aims? • POLITICAL – Did it contribute to the Government’s electoral popularity/re-election? • Obviously may be major contradictions between outcomes/success on each dimension • Unintended consequences if all dimensions not considered

7. Complexity factors • Ontology and epistemology, as always! • Success for whom? • The temporal dimension – different time-scales for different actors – different time-scales related to different dimensions of success

8. Assessing policy success: using the Heuristic 1 • J. Rutter, S. Sims and E. Marshall (2012), The 'S' Factors: Lessons from IFG's Policy Success reunions- http://www.instituteforgovernment.org.uk/publications/s-factors • Identified 6 UK Policy Successes over previous 30 years • Policy Success Reunions – Brought together all key actors to discuss why they were successful

9. Assessing policy success: using the heuristic 2 7 key lessons 1. Understand the past and learn from failure 2. Open up the policy process 3. Be rigorous in analysis and use of evidence 4. Take time and build in scope for iteration and adaptation 5. Recognise the importance of individual leadership and strong personal relationships 6. Create new institutions to overcome policy inertia 7. Build a wider constituency of support

10. On strategic policy capability • Comparative project on strategic policy capability in Australia and New Zealand • Seeks to evaluate the impact of various learning interventions on policy capability

11. Project aims • To assemble an information base to allow cross-jurisdictional comparisons of policy advising frameworks and capability requirements for strategic policy development and advisory work. • To identify areas where there are specific deficits and to design and evaluate different initiatives to address them drawing on international as well as local initiatives within some ANZSOG jurisdictions. • To promote wider discussion, learning and reflection by practitioners and academics on the diverse approaches used by individual agencies and departments, Public Service Commissions/State Service Authorities.

12. Learning interventions

13. Because what you told me is absolutely correct but completely useless Where am I? Yes, how did you know? You must be a researcher Because you don’t know where you are, you don’t know where you’re going, and now you’re blaming me You’re 30 metres above the ground in a balloon You must be a policy maker Yes. How did you know?

14. Key questions • What are the drivers of strategic policy capability internationally and domestically? • Is there a problem? • If there is – what is the nature of the problem? • Is it systems-based? • Is it competency based? • Is it a problem of politics? Or a virtue of liberal democracy? • Can learning interventions make a difference?

15. Evidence-based (rational) versus policy-based (real world) policy-making • Notwithstanding that evidence will always be contested in a survey of Commonwealth and State SES (ANZSOG, 2012; N=120) we posed four sets of questions using survey and workshop methods: • Questions crystallized around whether evidence was a sufficient criterion for winning the war of ideas (Survey). • Questions centering on the barriers to evidence-based policy-making (Workshop). • Questions relating to perceived qualities of policy leadership (Survey) • Questions relating to perceptions of the ingredients of better policy-making (Workshop)

16. Is evidence a sufficient criterion for winning the war of ideas? • 94% recognised the importance of evidence-based policy-making • 84% identified an ongoing tension between short-term imperative and evidence-based policy-making • 62% noted “Ministerial indifference over the facts” • All respondents (N=48) who had been in the service for 10 years or more states that the use of evidence in policy-making was in ‘dramatic decline’

17. % time spent on developing new policy, programmes or interventions through a “rational process of learning”

18. % time spent on retrofitting evidence to decisions that have already been taken

19. Is evidence a sufficient criterion for winning the war of ideas? • Breakdown by gender. Marginal difference. • Breakdown by age. More significant difference. •  Breakdown by jurisdiction. Predictable difference due to greater delivery function at the state level.

20. What are the major barriers to getting evidence into policy-making?

21. What are the most important qualities of politically aware policy leadership? • Broad role (rather than maturity) experience • Policy-makers who define politics purely as the means for pursuing self-interest score poorly; those that focus on empowerment score highly • Cross sectoral/overseas experience • Strong advocacy skills • Ability to read people and situations • Ability to build teams around weaknesses • Ability to focus on the work & not get dragged into the game-laying

22. What are your principles ofpolicy leadership for the Twenty-First Century?