Long term impacts of the EU's Framework Programmes for research and technology development

1. Long term impacts of the EU's Framework Programmes for research and technology development American Evaluation Association, 5 November 2011 Neville Reeve PhD, European Commission DG Research and Innovation

2. What is the FP? • Framework Programme (FP) is a multi-annual research funding programme implemented at EU level • 7th Framework Programme (2006-2013) - €50 billion – 27 Member States – more than 100 participating countries – tens of thousands of transnational co-funded projects

3. 1952: ECSC Treaty; first projects started 1955 1957: EURATOM Treaty; Joint Research Centre set up 1984: First Framework Programme (1984-1987) 1990: Third Framework Programme (1990-1994) 1994: Fourth Framework Programme (1994-1998) 1998: Fifth Framework Programme (1998-2002) 2002: Sixth Framework Programme (2002-2006) 2006: Seventh Framework Programme (2007-2013) Successive FPs

4. FP Budgets

5. Cooperation – Collaborative research (Multi-partner projects) Ideas – Frontier Research (Individual Grants) People – Marie Curie Actions (Mobility) Capacities – Research Capacity (Infrastructures) + JRC non-nuclear research Euratom direct actions–JRC nuclear research Euratom indirect actions – nuclear fusion and fission research FP7 (2007-2013) - The Structure

6. First four years of FP7 245 concluded calls more than 77,000 proposals received 59,000 proposals evaluated (312,000 applicant organisations and individuals) 12,000 proposals retained for negotiations (69,000 participants) 169 countries involved requested EU funding of € 20,4 billion FP7 in figures

7. Changing priorities

8. Changing FP Objectives • Early phase –expansion from ICT and energy • Second phase – widening objectives • Third phase – European value added and social objectives • Fourth phase – European Research Area • Fifth phase – Frontier research, innovation

9. Programmes Instruments New thematic initiatives SETPLAN JTI Artemis Eniac Clean Sky IMI FCH Art. 169 AAL Bonus EMRP ERANET JTI PPP ICT-FET (Flag Ships) ERANET+ (external view of the current landscape)  Wind Energy 6B€ Solar Energy 16B€ Bioenergy 9 B€ Carbon Capture & Storage 13B€ Electricity Grid 2 B€ Sustainable Nuclear Energy 7 B€ Smart Cities Ageing (More Years Better Lives) Climate Knowledge (Clik-EU) Seas and Oceans Antimicrobial resistance Urban Europe Water challenges • Alzheimer • Agriculture, Food Security & Climate change • Health and Diet • Cultural Heritage National and Regional Funds   Joint Programming  Eureka!    ELSA  EERP * NER300 *  EUROSTARS ENV ENE TRS eHealth eIdentity ICT for TT Energy efficiency FP7 HEALTH  CIP NMP SPA SEC CSH  KBBE  ICT PPP Energy Efficient Buildings Future of Factories Green cars Future Internet  EC Funds    ERC SMEs and SME Associations  eHealth Smart grid TT, mobility & logistics Content Large Scale Demos & trials  INFRASTRUCTURES  PEOPLE Deployment Fundamental Applied Development Innovation

10. Annual Monitoring FP6 Ex-post evaluation - 2008 FP7 Progress report - 2009 FP7 Interim evaluation - 2010 Ex ante Impact Assessment FP8 - 2011 Ex post evaluation FP7 - 2015 Evaluation and Monitoring Roadmap

11. Independence Coverage Evidence Coordination Impact Communication Principles to describe activities

12. Decentralised approach All evaluations carried out by independent external experts contractors selected through rigorous public procurement process - expertise, quality of tender, price experts selected according to expertise, with balancing issues such as geography, background, gender… Use of non-European experts Wide consultation on Terms of Reference for FP evaluations Programme Committees Independence

13. (Nearly) all thematic areas covered by one or more evaluations during each FP Specific evaluations for Instruments (JTIs, PPPs, IPs, NoEs) Increasing variety and complexity of MS impact evaluations FP7 projects to advance techniques Coverage

14. Combination of different evidence types – quantitative and qualitative Indicators base in the annual monitoring reports Major advance provided by CORDA has supported new types of analysis including networking and bibliometrics 64 individual evaluation studies (DG RTD) in last 3 years Evidence

15. FP7 Interim – Approach and Sources EVIDENCE BASE • Statistical data • Reviews & evaluations • ERC • RSFF • Marie Curie • etc. • Independent studies EXPERT GROUP • Deliberation • Successive drafts • Own expertise CONSULTATIONS • ‘Stakeholders’ • Self-assessments • Other sources • NCP survey • Papers submitted HEARINGS • Commission • Agencies • Researchers • Stakeholders SUPPORTING EXPERTS • Specific topics

16. Evaluation study structuring – who coordinates in FP6

17. Evaluation study structuring – hubs and coordination in FP6

18. Commission network across the ‘research family’ European RTD evaluation network around 35 members twice yearly meetings Evaluation manual Control Standards Coordination

19. Clear link between FP6 ex post, FP7 Progress report, FP7 Interim evaluation and Horizon 2020 Similar evidence from earlier evaluations Council conclusions Recent debate on ‘wider participation’ Impact of evaluations

20. FP level reports sent to Institutions with follow-on debates and opinions Dedicated website Conferences Member States (EUFORDIA, ‘Half Time Highway’) Commission (‘Building the future knowledge base’) Participation in international academic meetings Communication

21. What have we learnt? • Funds high-quality R&D: appraisal is tough; competition is fierce; participants perform better than non-participants in bibliometric terms; they include the scientific elite • Growth of the FP has paralleled growth in high-quality international co-publication • Attracts the more excellent researchers in their fields - engages the more research-intensive companies in theirs • It is – by design – a pre-competitive, collaborative programme, primarily producing ‘intermediate knowledge outputs’ as well as technical and market network relationships that are re-used in other R&D and business processes • Participants who enter projects with a deliberate product or process innovation objective are more likely to obtain short-term results than others

22. ……and this • FP is a place to exploit existing strength – generally too competitive to allow capacity building – that has to be done with national resources • Most participants have only a fleeting relationship with the FP. However, • Strong core of established players and networks whose composition slowly shifts over time - most participants have only a short relationship with the FP - new participants appear to learn the value of networked R&D and increasingly to participate in ‘open innovation’ • But…..we know very little of the details of how networks work, how networking relates to strategy or how network shape relates to success

23. …and more • Despite trend to larger instruments in recent Framework Programmes, bigger networks do not seem to be more productive than smaller ones – in fact, the evidence there is suggests the opposite • FP often associated with pre-normalisation R&D and the development of technical standards • Most participants believe that FP participation increases their competitivity but the indirect nature of the FP’s effects (through ‘intermediate knowledge outputs’) makes them very hard to track • Parts of the FP that focus on smaller firms and more direct results have been evaluated using a cash benefit-cost approach, which suggests high benefit-cost ratios. Smaller firms benefit less than larger ones (and are generally less satisfied with FP participation)

24. But the long term view is missing…. • Evaluation system fits the policy cycle but misses longer term effects • Complexity of the picture limits taking a longer term view • But we now have more than 30 years of experience…. • What's it done for jobs and growth?

25. Thank you for your attention Evaluation webpage:http://ec.europa.ehttp://ec.europa.eu/research/evaluations/index_en.cfmu/research/evaluations/index_en.cfm

26. Strategy prior to HORIZON 2020 launch; coverage of all areas and components; detailed timetables Governance interaction with Member States and stakeholders; expert advice Coherence Cross-cutting studies on transversal issues (scientific quality, job creation, impacts, …); common templates and methodologies; common key indicators Evidence focus on throughput, output and impact; integration of reporting information - project reviews/ project outcomes; Transparency Evaluation strategy; website Building Blocks for the Future