Nuove frontiere e prospettive di finanziamento per la ricerca europea nel periodo 2007-2013

1. Nuove frontiere e prospettive di finanziamento per la ricerca europea nel periodo 2007-2013 Valutazione delle proposte e probabilità di successo FP6 IST Giancarlo Abbate G. Abbate Dip. di Sc. Fisiche Università di Napoli Federico II

2. Sommario La valutazione delle proposte Area Tematica del 6 PQ: Tecnologie per una società dell’informazione • Le procedure di valutazione • I criteri ed i sottocriteri di valutazione per i differenti “strumenti” di finanziamento • Il processo di selezione delle proposte • Gli esperti indipendenti. Lettura individuale • Consensus meeting • Lettura incrociata. Panel meeting. Classifica. Audizione • Commenti sul processo di selezione • Paragone con le procedure di valutazione di alcuni altri programmi. G. Abbate Dip. di Sc. Fisiche Università di Napoli Federico II

3. Le procedure di valutazione • Procedura a due stadi: esempio FET-Open STREP ftp://ftp.cordis.lu/pub/ist/docs/fet/int-o-1.pdf G. Abbate Dip. di Sc. Fisiche Università di Napoli Federico II

4. Le procedure di valutazione • Procedura a due stadi: esempio FET-Open STREP G. Abbate Dip. di Sc. Fisiche Università di Napoli Federico II

5. Le procedure di valutazione • Procedura a due stadi: esempio FET-Open STREP G. Abbate Dip. di Sc. Fisiche Università di Napoli Federico II

6. Le procedure di valutazione • Procedura a due stadi: esempio FET-Open STREP G. Abbate Dip. di Sc. Fisiche Università di Napoli Federico II

7. Procedura a singolo stadio: Esempio FP6-IST-2002-2.3.2.2 Optical, opto-electronic, & photonic functional components http://www.europa.eu.int/comm/dgs/information_society/evaluation/pdf/5_y_a/ist_5ya_final_140105.pdf The scientific and technical evaluation of proposals is carried out by external experts from the public and private sector. Full proposals are evaluated, ranked and selected. The IST theme under FP5 and FP6 generally has a high oversubscription of the budget – e.g. the first call for the IST Priority Theme in FP6 attracted 1400 proposalsrequesting€ 6.2 B, whereas the available budget was only € 1.07 B. G. Abbate Dip. di Sc. Fisiche Università di Napoli Federico II

8. I criteri ed i sottocriteri di valutazione per i differenti “strumenti” di finanziamento Common evaluation criteria for evaluating proposals A number of evaluation criteria are common to all the programmes of the Sixth Framework Programme and are set out in the European Parliament and the Council Regulations on the Rules for Participation (Article 10). These are: a) Scientific and technological excellence and the degree of innovation; b) Ability to carry out the indirect action successfully and to ensure its efficient management, assessed in terms of resources and competences and including the organisational modalities foreseen by the participants; c) Relevance to the objectives of the specific programme; d) European added value, critical mass of resources mobilised and contribution to Community policies; e) Quality of the plan for using and disseminating the knowledge, potential for promotinginnovation, and clear plans for the management of intellectual property. G. Abbate Dip. di Sc. Fisiche Università di Napoli Federico II

9. I criteri ed i sottocriteri di valutazione per i differenti “strumenti” di finanziamento Furthermore, in applying paragraph (d) above, the following criteria are also to be taken into account: a) For networks of excellence, the scope and degree of the effort to achieve integration and the network’s capacity to promote excellence beyond its membership, as well as the prospects of the durable integration of their research capabilities and resources after the end of the period covered by the Community’s financial contribution; b) For integrated projects, the scale of the ambitionof the objectives and the capacity of the resources to make a significant contributionto reinforcing competitiveness or solving societal problems; c) For integrated initiatives relating to infrastructure, the prospects of the initiative’s continuing long term after the end of the period covered by the Community’s financial contribution. G. Abbate Dip. di Sc. Fisiche Università di Napoli Federico II

10. I criteri ed i sottocriteri di valutazione per i differenti “strumenti” di finanziamento • Horizontal issues • Are there gender issues associated with the subject of the proposal? If so, have they been adequately taken into account? • Have the applicants identified the potential ethical and/or safety aspects of the proposed research regarding its objectives, the methodology and the possible implications of the results? If so, have they been adequately taken into account in the preparation of the proposal? • To what extent does the proposal demonstrate a readiness to engage with actors beyond the research community and the public as a whole, to help spread awareness and knowledge and to explore the wider societal implications of the proposed work? • Have the synergies with education at all levels been clearly set out? • If third country participation is envisaged in the proposal, is it well justified and the participation well integrated in the activities? G. Abbate Dip. di Sc. Fisiche Università di Napoli Federico II

11. I criteri ed i sottocriteri di valutazione per i differenti “strumenti” di finanziamento • 1. Relevance (threshold score 3 out of 5) • The extent to which the proposed project addresses the objectives of the work programme. • 2. Potential impact (threshold score 3 out of 5) The extent to which: • the proposed project is suitably ambitious in terms of its strategic impact on reinforcing competitiveness (including that of SMEs) or on solving societal problems. • the innovation-related activities and exploitation and/or dissemination plans are adequate to ensure optimal use of the project results. • the proposal demonstrates a clear added value in carrying out the work at European level and takes account of research activities at national level and under European initiatives (e.g. Eureka). • 3. S&T excellence (threshold score 4 out of 5) The extent to which: • the project has clearly defined objectives. • the objectives represent clear progress beyond the current state-of-the-art. • •the proposed S&T approach is likely to enable the project to achieve its objectives in research and innovation. G. Abbate Dip. di Sc. Fisiche Università di Napoli Federico II

12. I criteri ed i sottocriteri di valutazione per i differenti “strumenti” di finanziamento • 4. Quality of the consortium (threshold score 3 out of 5) The extent to which: • the participants collectively constitute a consortium of high quality. • the participants are well-suited and committed to the tasks assigned to them. • there is good complementarity between participants. • the profiles of the participants, including those to be included later, have been clearly described. • the real involvement of SMEs has been adequately addressed. • 5. Quality of the management (threshold score 3 out of 5) The extent to which: • the organisational structure is well matched to the complexity of the project and to the degree of integration required. • the project management is demonstrably of high quality. • there is a satisfactory plan for the management of knowledge, of intellectual property and of other innovation-related activities. • 6. Mobilisation of resources (threshold score 3 out of 5) The extent to which: • •the project mobilises the minimum critical mass of resources (personnel, equipment, finance…) necessary for success. • the resources are convincingly integrated to form a coherent project. • the overall financial plan for the project is adequate. • Overall threshold score 24 out of 30. G. Abbate Dip. di Sc. Fisiche Università di Napoli Federico II

13. I criteri ed i sottocriteri di valutazione per i differenti “strumenti” di finanziamento • 1. Relevance (threshold score 3 out of 5) The extent to which the proposed project addresses the objectives of the work programme. • 2. Potential impact (threshold score 3 out of 5) The extent to which: • Europe has a strategic need to strengthen S&T excellence on the topic by means of a restructuring of the existing research capacities and the way research is carried out. • the goals of the network are, in that connection, suitably ambitious particularly, in terms of achieving European leadership and acting as a world force on this topic. • the proposal demonstrates a clear added value in carrying out the work at European level and takes account of research activities at national level and under European initiatives (e.g. Eureka). • there is an effective plan for spreading excellence, exploiting results and disseminating knowledge, including to SMEs and to those outside the network. • the proposed approach is likely to have a durable structuring impact on European research. • 3. Excellence of the participants (threshold score 3 out of 5) The extent to which: • the participants are currently conducting excellent research relevant to the topic of the network or are capable of important contributions to the joint programme of activities. • the participants are well suited to the tasks assigned to them. • they collectively have the necessary critical mass of expertise and resources to carry out the joint programme of activities successfully. G. Abbate Dip. di Sc. Fisiche Università di Napoli Federico II

14. I criteri ed i sottocriteri di valutazione per i differenti “strumenti” di finanziamento • 4. Degree of integration and the joint programme of activities (threshold score 4 out of 5) The extent to which: • the expected degree of integration justifies supporting the proposal as a network of excellence. • the joint programme of activities is sufficiently well designed to achieve the expected degree of integration. • the participating organisations have made a convincing commitment towards a deep and durable integration continuing beyond the period of Community support. • 5. Organisation and management (threshold score 3 out of 5) The extent to which: • the organisational structure of the network provides a secure framework for any necessary structural decisions to be taken • the management of the network is demonstrably of high quality. • there is a well-considered plan for promoting gender equality in the network. • Overall threshold score 20 out of 25. G. Abbate Dip. di Sc. Fisiche Università di Napoli Federico II

15. I criteri ed i sottocriteri di valutazione per i differenti “strumenti” di finanziamento • 1. Relevance (threshold score 3 out of 5) The extent to which the proposed project addresses the objectives of the work programme. • 2. S&T excellence (threshold score 4 out of 5) The extent to which: • the project has clearly defined and well focused objectives. • the objectives represent clear progress beyond the current state-of-the-art. • the proposed S&T approach is likely to enable the project to achieve its objectives in research and innovation • 3. Potential impact (threshold score 3 out of 5) The extent to which: • the proposed project is likely to have an impact on reinforcing competitiveness or on solving societal problems. • the proposal demonstrates a clear added value in carrying out the work at European level and takes account of research activities at national level and under European initiatives (e.g. Eureka). • exploitation and/or dissemination plans are adequate to ensure optimal use of the project results. G. Abbate Dip. di Sc. Fisiche Università di Napoli Federico II

16. I criteri ed i sottocriteri di valutazione per i differenti “strumenti” di finanziamento • 4. Quality of the consortium (threshold score 3 out of 5) The extent to which: • the participants collectively constitute a consortium of high quality. • the participants are well-suited and committed to the tasks assigned to them. • there is good complementarity between participants. • the opportunity of involving SMEs has been adequately addressed. • 5. Quality of the management (threshold score 3 out of 5) The extent to which: • the project management is demonstrably of high quality. • there is a satisfactory plan for the management of knowledge, of intellectual property and of other innovation-related activities. • 6. Mobilisation of resources (threshold score 3 out of 5) The extent to which: • the project foresees the resources (personnel, equipment, financial…) necessary for success. • the resources are convincingly integrated to form a coherent project. • the overall financial plan for the project is adequate. • Overall threshold score 21 out of 30. G. Abbate Dip. di Sc. Fisiche Università di Napoli Federico II

17. Il processo di selezione delle proposte • Quality. Projects selected for funding must demonstrate a high scientific, technical and managerial quality in the context of the objectives of the RTD programme in question. • Transparency. In order to provide a clear framework for researchers preparing proposals for funding and for evaluators evaluating proposals the process of reaching those funding decisions must be clearly described and available to any interested party. In addition, adequate feedback must be provided to proposers on the outcome of the evaluation of their proposals. • Equality of treatment. A fundamental principle of EU RTD support is that all proposals should be treated alike, irrespective of where they originate or the identity of the proposers. G. Abbate Dip. di Sc. Fisiche Università di Napoli Federico II

18. Il processo di selezione delle proposte • Impartiality. All proposals are treated impartially on their merits. • Efficiency and speed. The procedures have been designed to be as rapid as possible, commensurate with maintaining the quality of the evaluation, appropriate use of public money and respecting the legal framework within which the specific programme is managed. • Ethical considerations. Any proposal which contravenes fundamental ethical principles may be excluded from being evaluated or selected at any time. G. Abbate Dip. di Sc. Fisiche Università di Napoli Federico II

19. Il processo di selezione delle proposte Appointment of independent experts • Calls for applications from individuals published in the Official Journal of the European Communities; or • Calls addressed to research institutions with a view to establishing lists of suitable candidates. • The lists of individuals from which panels of experts may be chosen, are drawn up by the Commission using the following selection criteria: • an appropriate range of competencies; • an appropriate balance between academic and industrial expertise and users; • a reasonable gender balance; • a reasonable distribution of geographical origins of independent experts; • regular rotation of independent experts. I valutatori sono retribuiti dalla CE ! G. Abbate Dip. di Sc. Fisiche Università di Napoli Federico II

20. Il processo di selezione delle proposte Impegni sottoscritti dai valutatori • Conflict of interest • Confidentiality • (Independent observers) Eventualmente ammessi al processo di selezione G. Abbate Dip. di Sc. Fisiche Università di Napoli Federico II

21. Il processo di selezione delle proposte • Step 1: Briefing of the independent experts • Step 2: Individual evaluation of proposals Each proposal is evaluated against the applicable criteria independently by several experts who fill in individual evaluation forms giving marks and providing comments. Proposal marking 0 - the proposal fails to address the issue under examination or can not be judged against the criterion due to missing or incomplete information 1 - poor 2 - fair 3 - good 4 - very good 5 – excellent Where appropriate, half marks may be given. The outcome of this step is the Individual Assessment Report G. Abbate Dip. di Sc. Fisiche Università di Napoli Federico II

22. Il processo di selezione delle proposte • Step 3: Consensus meeting For each proposal a consensus report is prepared. The report faithfully reflects the views of the independent experts referred to in Step 2. Nella (quasi) totalità dei casi il Consensus meeting termina con il consenso unanime dei valutatori. Il meeting è moderato da uno “scientific officer” della CE. The experts attempt to agree on a consensus mark for each of the blocks of criteria. They justify their marks with comments suitable for feedback to the proposal coordinator and agree on an overall consensus report, which is signed by them. The outcome of the consensus step is the consensus report G. Abbate Dip. di Sc. Fisiche Università di Napoli Federico II

23. Il processo di selezione delle proposte • Step 4: Panel evaluation A panel discussion may be convened, if necessary, to examine and compare the consensus reports and marks in a given area, to review the proposals with respect to each other and, in specific cases (e.g. equal scores) to make recommendations on a priority order and/or on possible clustering or combination of proposals. The panel discussion may include hearings with the proposers. A Commission official acts as moderator of the Panel • Cross readings • Comparison of consensus reports • grouping of proposals The outcome of the Panel meeting is An evaluation summary report for each proposal; A list of proposals passing thresholds, if any, along with a final mark for each proposal passing the thresholds and the panel recommendations for priority order. G. Abbate Dip. di Sc. Fisiche Università di Napoli Federico II

24. Il processo di selezione delle proposte • Hearings (Audizione) Per i nuovi strumenti del 6° PQ è prevista un’audizione individuale per ogni proposta che ha superato la soglia. Il panel di cui al punto precedente si riunisce (eventualmente in una composizione ridotta) ancora una volta (dopo un tempo opportuno, cioè circa un mese) e, al termine delle audizioni, propone la graduatoria di merito finale. • Final steps Reporting on the evaluation process Feedback to the proposers Negotiation Selection of proposals These final steps do not involve the independent experts G. Abbate Dip. di Sc. Fisiche Università di Napoli Federico II

25. Commenti sul processo di selezione • Il mio personale commento è che il processo di selezione raggiunge tutti gli scopi che si prefigge ed è il più efficiente ed obiettivo tra i vari processi a cui ho partecipato come soggetto attivo e/o passivo. • Alcuni feedback da valutatori IST: It was fascinating to see how such a complex problem, namely how to rank such a large number of disparate proposals, could be solved, and it was my pleasure to participate in the process. I'm glad that finally we made it! It was hard working, as you say, but I really enjoyed it: I enjoyed working with clever and skilled people as evaluators and very professionals people in the EU commission. I enjoyed the international involvement and the possibility of knowing interesting people in the field from all over Europe. I enjoyed the possibility of having an overview of top level research people is now working on. G. Abbate Dip. di Sc. Fisiche Università di Napoli Federico II

26. Commenti sul processo di selezione I learned a lot of very important things about science, present and future trends, and mainly how a good proposal should be prepared and how it should not It was great experience to take part in EC Project evaluations for me. I have learned how to work hard, precisely and effectively and without any external pressures I wish to say that I was very glad to participate in evaluations of IST proposals. Not only because of the scientific topic which is extremely interesting for me but also because of perfect organisation, competence of your team and friendly atmosphere during evaluation. Thank you very much I have been already in Growth panels in the past and had the opportunity to realize that the commission do everything is possible to be fair and objective when evaluating proposals and distributing public money. I do believe that IST staff is the paradigm in that manner G. Abbate Dip. di Sc. Fisiche Università di Napoli Federico II

27. Paragone con le procedure di valutazione di alcuni altri programmi • INTAS The International Association for the Promotion of Co-operation with Scientists from the New Independent States (NIS) of the Former Soviet Union • COST Founded in 1971, COST is an intergovernmental framework for European CO-operation in the field of Scientific and Technical Research, allowing the co-ordination of nationally funded research on a European level. COST Actions cover basic and pre-competitive research as well as activities of public utility Running Actions ~ 200 Total Budget (2005) € 13,500,000 • MIUR - PRIN G. Abbate Dip. di Sc. Fisiche Università di Napoli Federico II

28. Grazie! ... e buon appetito! G. Abbate Dip. di Sc. Fisiche Università di Napoli Federico II