May 2014

1. Evaluating Genomic Science: A Conversation with Dr. Samantha Evans Director of Evaluation, Genome Canada May 2014

2. Session Outline Genomic science, performance measurement and evaluation: Genome Canada – overview of mission and current programs Program Evaluation – what is it, why is it important and how it is changing Genome Canada’s approach to evaluation Outcomes and benefits – how is this defined by GC 2

3. Genome Canada’s mission We are driven by a mission to: connect ideas and people across public and private sectors to find new uses for genomics. investing in large-scale science and technology (S&T) to fuel innovation, and translate discoveries into applications to maximize impacts across all sectors. 3

4. Fulfilling our strategic plan 2012–2017

5. Canada has an unprecedented opportunity to play a world leadership role in the new bioeconomy • Given the excellence of our knowledge base in the life science sectors, our vibrant genomics enterprise, and our nation's footprint in natural resources, agriculture and other bio-based resources, Canada can claim a disproportionately-large share of the $1 trillion emerging global bioeconomy. How we know if we are successful in achieving our mission?

6. We Need to show we act as a catalyst building a thriving genomics enterprise in Canada that delivers tangible impacts

7. evaluation Evaluation ‘is the systematic collection and analysis of evidence on the outcomes of programs to make judgments about their relevance, performance and alternative ways to deliver them or to achieve the same results’ . TBS, Policy on Evaluation, April 2009

8. The old Approach Performance monitoring and evaluation in S&T environments has traditionally focused on the quality and quantity of the science produced, the outputs. For example, metrics you may be familiar with: Number of publications Number of citations Number of highly qualified personnel trained This happens because: It’s relatively easy to collect, It’s what funders seem to ask for, It’s what everyone does. BUT IT DOESN’T ANSWER THE SO WHAT? QUESTION 8

9. Things are changing Policy makers increasingly need to know their program investments provide them with the results they need. Why is this important for your research? Funders need to ensure programs & projects delivering on the promises. If we can’t demonstrate that what we fund is relevant; well administered; and has impact… 9

10. How is it shifting? There is impetus for S&T organizations to demonstrate: Quality of the science (checked by peer review/ due diligence, ongoing monitoring etcetc) BUT ALSO Success along the innovation continuum; Translational benefits (i.e. commercialization, tech transfer, & value-added impacts to society beyond science); and Longitudinal outcomes.

11. the pressure to demonstrate outcomes and Impacts is being felt by all Funders • Industry Canada’s S&T portfolio funds the following partner organizations: • Canadian Institute for Advanced Research; • Canada Foundation for Innovation; • CANARIE; • Genome Canada; • Perimeter Institute for Theoretical Physics; • Institute for Quantum Computing; and; • Mitacs. • GC is working with these organisations and IC to: • Develop common performance indicators; • Develop common definitional understandings of • Develop a community of practice for evaluation • WE ALL STRUGGLE WITH HOW BEST TO DEMONSTRATE OUTCOMES AND IMPACT. THERE IS NO QUICK OR EASY FIX OR CLEAR DIRECTION FORWARD

12. Theory of Change Where S&T funders have traditionally focused Where we need to now focus

13. Theory of Change It’s a road map for an initiative/program/intervention: Defines all the building blocks required to bring about a given long-term goal Usually depicted as a graphic representation of the change process. (logic model) Articulates the assumptions that stakeholders use to explain the change process Assumptions explain both the connections between early, intermediate and long term outcomes and the expectations about how and why proposed interventions will bring them about Provides a commonly understood vision of the long-term goals, how they will be reached, and what will be used to measure progress along the way.

14. A Systems Approach to outcome and Impact evaluation We are trying to establish an evaluation function based on systems approach: Understanding the role and relationship of components with the system as a whole Holistic, a change in part of a system affects the whole system The whole is more than the sum of the parts. A common starting-point and/or the use of different roads may lead to different results. Circular, a causal connection is not linear

15. A Systems Approach In practice that means: Using frameworks to identify the complexity of component and their relationships; Paying careful attention to how boundaries are drawn (does not mean including everything); Use of a balanced approach to collecting performance data, before, during and after program design; Considers processes as well as outcomes and impacts; Uses evaluation for both learning and accountability; Quantitative and qualitative needed to properly support decision-making 15

16. How GC has been setting up a systematic Evaluation Framework Genome Canada Strategy Mission and Objectives Logic Model Performance Measurement Framework Evaluation Work Plan Performance Measurement Strategy Development Data collection and analysis Application Inform Communication Societal Perspective Inform Public, Genomic Enterprise, & Media on Genome Canada’s performance Accountability Stakeholder Perspective Accountability to the Board Accountability to Government Accountability to Canadian public Management Organizational Perspective Risk mitigation Informing decision making Assessment of performance Monitoring social and economic outcomes

17. What GC is working towards As a funder GC has an opportunity to lead the way to be innovative in the way it evaluates the research it funds We look at evaluation requirements at the different stages of a Program lifecycle Program development Program life Program closure

18. What GC is working on • Program Development • Mapping expected program outcomes & impacts while developing RFA’s • Developing better definitions for social and economic benefits upfront • Evaluating the science through peer review • Considering separate RFA’s for evaluation research concurrent with funded programs. • Program Life • Collecting performance metrics with the Centres- ongoing through project life cycle • Mining interim reviews & Research Oversight Committee work for outcomes information • Annual reporting of metrics to Industry Canada • Exploring ways of working with Comms and Centres in collecting great outcomes ‘stories’ of research • Program Closure • Using final reports to undertake program or thematic reviews • Feeding data/information on outcomes into the development of new competitions • Exploring initiatives to capture outcome data post funding

19. What are the challenges to this new approach to evaluation? • We want to find ways to measure: • Success along the innovation continuum; • Translational benefits (i.e. commercialization, tech transfer, & value-added impacts to society beyond science). • Longitudinal outcomes • This will need: • new ideas, • new methods, • new ways of working together. • There are plenty of challenges.

20. SUCCESS ALONG THE INNOVATION CONTINIUM

21. We’ve designed A Portfolio of Programs along the innovation continuum

22. Multiple StakeholdersMultiple SectorsMultiple Needs Evaluation

23. Multiple Foci over time Complexity of measurement Time 23

24. Attribution Versus Contribution Attribution is relatively clear with measuring outputs, more tricky with both intermediary and long term outcomes and impacts; Observed change in outcomes may be the result of many other factors other than GC funding- other programs-economic factors-social trends etcetc; Need to shift thinking………less about precision of attribution and more about increasing understanding and knowledge of our contribution

25. Then: The Truth Behind the Hype Evaluation is about measuring reality

26. Methodological Challenges • Substantive evaluation of the scientific/innovation process relatively new to researchers • Accountability has been mainly administrative rather than substantial • Introducing metrics/data collection and evaluation causes changes in practise – consequences not always known • Lack of evaluation budgets often limit risk-taking in data collection and experimenting with cutting edge methodologies. • Capturing significant impacts and outcomes outside academia and unintended outcomes. 26

27. Benefits: social and Economic Challenge is we can’t give a quick and easy definition of S &E benefits because every program and every project is different Even though it affects: How objectives are set for RFA How applications are prepared How progress is monitored through the life of the project How reports are done at the end of the project and How outcomes and impacts are demonstrated that often come to fruition post funding Our approach has been to keep it broad to allow researchers flexibility to interpret and apply to their research. Is this still sufficient???

28. ABC: ‘Benefits for Canada’ in the RFA To be realized within five (5) years from the end of the project potential benefits could include: job creation and economic growth in Canada, development of a product or service, (a clear commercialization process, which includes IP management and ownership, technology transfer and benefit sharing, must be described) an impact on society, quality of life, better health, and a cleaner environment, knowledge generation or translation, potential licenses and/or new start-ups or the creation of new policies.

29. ABC: Direction on Outcomes in the Final Report We tried to provide direction with some questions. Do they help??? Explain how the project has (or will in the future) transfer, disseminate, use, and/or apply the potential deliverables from the research to realize the anticipated benefits. For example, key new products and/or services developed, commercialization processes put in place (including IP management and ownership, technology transfer and benefit sharing). How far along the “innovation pipeline” would you describe the project? What role or function has been played by the co-funders or end-users involved in the project and what impact has this had?

30. ABC: Direction on Outcomes in the Final Report Describe the integration between the GE3LS research and the overall goals of the project. What opportunities and challenges has the project faced in this area? What were the principal means of knowledge dissemination or translation into policy and practice and how successful were these? If the project had a chance to “do it over again”, what other or different GE3LS research would the project team undertake? Briefly describe further research or development work which could be undertaken as a direct result of this project and the possible outcome(s). Describe future collaborations that the project team may establish. Are there any anticipated next steps for the research?

31. ABC- what we asked for In the Benefits to Canada section of final report • Potential benefits could include: job creation and economic growth in Canada; development of a product or service; an impact on society, quality of life, better health, and a cleaner environment; knowledge generation or translation; or potential licenses and/or new start-ups or the creation of new policies that have advanced the following key areas: • securing Canada’s future bio-based economy • feedstock optimization • sustainable processing • value-added bioproducts • crops for a healthy population • agriculture and food production sustainability

32. LSARP 2012 Personalised Health In our recent Personalised Heath Competition we stated: lead to social and/or economic benefits for Canada in terms of contributing to a more evidence-based approach to health and a more cost-effective health-care system. These benefits could include, for example, adoption of a new technology, a change in clinical practice guidelines, an application of an existing drug to a new indication, or a reduction in the number of adverse drug reactions. In addition, there could be other positive impacts on society,the economy (e.g., development of products with commercial potential), quality of life, or the environment. Still a lot of emphasis on the economic benefits in the applications though!!!!!!

33. LSARP 2012 Personalised HealthCriteria for assessing SEB Deliverables To what extent have the applicants identified appropriate deliverables in terms of their potential to have eventual clinical utility and/or practical applicability? What is the probability that the deliverables will be achieved by the end of the funding period? Expected Benefits Significance of anticipated benefits contributing to a more evidence-based approach to health and improved cost-effectiveness of the health-care system? Is the expected timeframe of the anticipated benefits reasonable?

34. LSARP 2012 Personalised HealthCriteria for assessing SEB Strategy for Realizing Benefits How persuasive is the strategy set out for realizing benefits from their research? How strong is the plan for knowledge translation and development of benefits, i.e., how well does the plan explain the next steps of how the deliverables from the research will be transferred, disseminated, used, and/or applied to realize the socio-economic benefits? How convincing is the rationale (including an economic component) explaining how the outcomes of the project will potentially contribute to a more evidence-based approach to health and a more cost-effective health-care system?

35. LSARP 2012 Personalised HealthCriteria for assessing SEB Expertise for Realizing Benefits How appropriate is the expertise of the team that will further develop and implement the strategy for realizing benefits? To what extent are likely end-users involved in the project and the strategy to realize benefits? If the strategy includes commercialization, to what extent has appropriate technology transfer expertise been included? How strong is the expertise assembled to ensure appropriate attention to ethical, environmental, economic, legal and social factors in the process of realizing benefits?

36. Working together We are all trying to figure this out together We can be adaptive It has to work for all stakeholders, the projects, the funders etcetc There will never be a clear model/ process or system to follow We are open to idea, thoughts, feedback as we work our way through this. Start anticipating the potential benefits of your next application early in the process- outcome mapping