Scoring Goals or Changing the Game: What Impacts Should We Measure?

1. Scoring Goals or Changing the Game:What Impacts Should We Measure? Jonathan Lomas Canadian Health Services Research Foundation Presentation to ESRC Symposium: ‘New Approaches to Assessing the Non-Academic Impact of Social Science’ London, May 12-13, 2005

2. My question is, are we having an impact?

3. Is Research Ready for Action? Medline search (1966-02] to identify articles stating: a. “need more research” or “need less research” Need more 161/162 Need less 1/162 b. “more questions than answers” or “more answers than questions” More questions 163/166 More answers 3/166 David, AS. BMJ 2002; 323:1462-3

4. Assessing the Impact of What? The Research Produced? • A single research study published in a journal • A summary of some research studies written in plain language and posted on a web site • A systematic review with targeted dissemination of key messages to potential users • A body of research knowledge developed and discussed face-to-face with potential users

5. Assessing the Impact of What? (cont) The Research Production Process? • All the activities of a research commissioning or granting agency (including training) • All the activities of a research production facility (e.g. institute, department, university) • The activities of a potential research user organization and its staff • The entire ‘research regime’ in a country

6. Where I Work CHSRF’s mission: “To support evidence-based decision-making in the healthcare system” Our ultimate desired impact is cultural change in the research and healthcare systems Changing the game, not just scoring goals

7. The Discipline of Objectives & Logic Models Yogi Berra on objectives: • “If you don’t know where you’re going, you might not get there” Lewis Carroll on logic models: • “If you don’t know where you’re going, any road will take you there”

8. Decision Makers PolicyMakers Managers Service Professionals ‘Client’ & Public Organized Interests e.g. drug companies, professional associations Decision Maker Diversity

9. PROBLEMS SOLUTIONS Researchers PolicyMakers Managers Decision Makers Service Professionals Patient & Public Organized Interests e.g. drug companies, professional associations System-based University-based Management Consultants Stakeholder-based Why Social Science isn’t IBM or General Electric

10. Linkage and Exchange Other Influences Personal Experience, Anecdote, Wants, Interests, Myths, Assumptions, etc. ISSUES & PRIORITIES Decision Makers IDEAS Knowledge Purveyors Receptor Capacity Priority-setting structures Research Funders Critical Evaluation Funding and Training Vehicles PROBLEMS SOLUTIONS RESEARCH EVIDENCE PRIORITY TOPICS Researchers Synthesis and Influence Evidence-Based Decision-Making

11. CHSRF’s Objectives • To increase health system decision-makers’ appreciation of the value of research • To increase the production of research relevant to the needs of health system decision- makers • To increase the availability and acquisition of needed research by health system decision-makers • To increase the appraisal and application of needed research by health system decision makers

12. Increased Appreciation of Value of Research • Programs and activities: • overall ‘linkage & exchange’ approach • consultations with users for priorities • case study presentations of value • Measurable outcomes: • # of decision-makers participating in research • expenditures on research commissioning by system • % of employees in system with research training

13. Increased Production of Relevant Research • Programs and activities: • priority theme-based program-funding • commissioned syntheses on current issues • applied training programs • encourage university incentives for applied research • Measurable outcomes: • amount of research in priority theme areas • self-reported awareness/use of research syntheses • # of graduates with applied research skills

14. Increased Availability/Acquisition of Research • Programs and activities: • plain-language research summaries (1:3:25,Mythbusters) • face-to-face exchanges on timely topics • creation/support of knowledge networks • creation/support for knowledge brokering • Measurable outcomes: • self-reported awareness of disseminated research • self-reported follow-up contact with researchers • self-reported use of web-based and other resources for research evidence acquisition (audit computer bookmarks)

15. Increased Appraisal/Application of Research • Programs and activities: • training users in research appraisal and application • organisational best practices in research use • Measurable outcomes: • self-reported application of research • changes in organisational structures and processes to better accommodate research • Increased sense of ‘decision-certainty’ where synthesised research is available

16. The Attribution Challenge What, if any, is/was the role of research versus all the other influences on behaviour?

17. Experimental Quasi-Experimental Counterfactual Experimental Quasi-Experimental Qualitative Theories of Change Public Consultation Distributional Data Surveys Admin Data Comparative Qualitative Implementation Evidence Ethics Evidence Organizational Evidence Cost-Benefit Cost-Effectiveness Cost-Utility Econometrics Program or Intervention Effectiveness Surveys Qualitative Economic/ Financial Evidence Attitudinal Evidence Forecast Evidence Multivariate Regression Types of Research Evidence Adapted from Philip Davies, 2005

18. Experimental Quasi-Experimental Counterfactual Experimental Quasi-Experimental Qualitative Theories of Change Public Consultation Distributional Data Surveys Admin Data Comparative Qualitative Implementation Evidence Ethics Evidence Organizational Evidence Cost-Benefit Cost-Effectiveness Cost-Utility Econometrics Program or Intervention Effectiveness ResearchEvidence Surveys Qualitative Economic/ Financial Evidence Attitudinal Evidence Forecast Evidence Multivariate Regression Types of Research Evidence Adapted from Philip Davies, 2005

19. Professional Experience & Expertise Pragmatics & Contingencies Political Judgement ResearchEvidence Lobbyists & Pressure Groups Resources Habits & Tradition Values Combining Research and Colloquial Evidence for Information Adapted from Philip Davies, 2005

20. A Final Zany Idea The ‘impact file’ - a deductive approach to assessing impact Add photo of impact file

21. What’s In the Impact File? • T F H M & Y D G W - testimonials • Altered career trajectory (researcher or decision-maker) • Changes in similar organization’s programs & processes - ‘lateral impact’ • Better communication of research and its implications - dissemination • Awareness of research by decision-makers - acquisition • Changed decisions based on research – application • Changes in researchers’ or decision-makers’ structures and processes - ‘cultural change’

22. Maybe Settling for Second Best is OK! Using an impact file is: • Motley, ad-hoc and not comprehensive or systematic • Biased to individual not organizational responses • Qualitative and potentially non-generalizable But, as Churchill said of democracy: “Many forms of government have been tried … No one pretends that democracy is perfect … Indeed, it has been said that democracy is the worst form of government, except all those other forms that have been tried from time to time.” House of Commons, 1947

23. THANK YOU! www.chsrf.ca or www.fcrss.ca