Erasmus Mundus programme experience Karsten Suhre , external expert to EACEA Riga, 25.02.2011

1. Erasmus Mundusprogramme experienceKarsten Suhre, external expert to EACEARiga, 25.02.2011

2. Outline • To write a successful proposal, it is essential to do this from the point of view of the person who eventually evaluates it. • The following slides have been used by the agency to brief its external experts for the evaluation. • Here I shall present, at hand of concrete examples, how this briefing has been put into practice.

3. An Erasmus Mundus Action 1 Joint Programme (at masters or doctoral level) is... ... an integrated study (/research) programme, delivered (/managed) by a consortium of EU and, if relevant, non EU HEIs (/research organisation) that includes a mandatory mobility component and leads to the award of a fully recognised joint (/double, multiple) degree.

4. Action 1 – Joint Programmes Joint Programmes should have a double objective: • To offer high level training possibilities in a large variety of thematic fields to excellent postgraduate students / doctoral candidates from all over the word • Proposals have to demonstrate their excellence, expertise and innovation potential in the thematic field(s) concerned • To contribute to the shaping of a European Master and European Doctorate model • Proposals have to demonstrate strong programme integration and institutional commitment, addressing the EM programme requirements and recommendations but also those defined in the EU education and research policies (Bologna and Ljubljana Processes)

5. An EM Joint Programme must be • Excellent in all its dimensions • Outstanding academic / research quality • Professional and transparent admin. /fin. /H.R. management • Integrated in all its dimensions • Jointly designed • Joint application, selection and assessment criteria • Joint students registration and admission conditions • Programme content fully recognised by all partners • Sustainable beyond EU funding • Strong Institutional Commitment • Diversified sources of funding • Closely linked to its socio-economic environment • Added value for the academic / professional sector • Role of “associated members” (programme’s sustainability and evaluation; students /doctoral candidates employability)

6. Joint Programmes may vary for what concerns: • their duration, • thematic area (broad or narrow) • role (/number, /origin) of individual partners, • duration (/number) of mobility tracks, • professional or research orientation, • placement/internships possibilities, • type of final degree(s) awarded, • etc. There is no prescribed model, BUT the selected model must solid, integrated, innovative, and take full account of the learning (/research) objectives as well as the participants’ needs and constraints.

7. Erasmus Mundus Action 1Joint Doctoral ProgrammesSpecificities

8. WHAT IS AN EMJD (1) ? • A three- to four-year integrated postgraduate programme at doctorate level, • designed and delivered by a consortium of universities representing at least 3 different European countries, • including a mandatory training / research period in at least 2 of the European consortium countries, • leading to the award of fully recognised double, multiple or joint degree(s), • offering EM fellowships for European and non-European doctorate candidates

9. WHAT IS AN EMJD (2) ? • EMJDs are based on the EMMCs model and on the experience of Marie Curie fellowships • Marie Curie focuses on the individual research of its grantees in a given research area, • EMJDs put more emphasis on a long-term structured inter-university cooperation model and the design of an integrated doctoral programme • EMJDS have a double objective • To contribute to the shaping of doctoral studies in the EAHE • To support young researchers and offer them a framework of excellence to implement their PhD activities

10. AN EMJD MUST (1) : • Be fully developed at the time of the application and ready to run for 5 consecutive editions (as from acad. year 2011/12) • Be an integrated programme in all its aspects • Define its own Participation Costs (/ programme fees); these costs • must be transparent and independent from the candidate’s mobility track / HEIs visited; • can be different for « laboratory » and « non laboratory » based research; • should be co-funded by the consortium (/other sponsors) if higher than the EM fixed contribution.

11. AN EMJD MUST (2) : • Be based on a consortium agreement signed by the governing bodies of the partner HEIS • Recruit the doctoral candidates through employment contracts(except in adequately documented cases or when national regulation would prohibit this possibility) • Independently from the training/research language, provide for the use of min. 2 European languages spoken in the candidate’s host countries • Follow the principles described in the European Charter for Researchers and the Code of Conduct for the recruitment of Researchers

12. AN EMJD MUST (3) : • Sign detailed Doctorate Agreements with the candidates defining the EMJD rules and both parties rights and obligations • Where applicable, ensure that the relevant ethics committees have approved the individual research projects • Support third-country candidates in obtaining their visas and residence permits • Take the necessary measures to promote the joint programme and recruit top class doctoral candidates

13. EMJDs overall assessment • Will this proposal actively contribute to the training of excellent young researchers and to the innovation process in the thematic field(s) concerned ? • Will this proposal contribute to the shaping of a strong and sustainable « European Doctorate » model in line with the objectives of the Bologna Process and which follows the prescriptions included in the « Code of Conduct for the Recruitment of Researchers » and the « European Charter for Researchers »? • Does this proposal indeed demonstrate the capacity and the readiness to run if selected ?

14. EMJDs AWARD CRITERIA • Academic and Research Quality (25%) • How clearly described and justified are the: • objectives and needs analysis (scientific and socio-economic terms) • Added value and distinctiveness (comparing to other existing programmes) and • Outcomes of the proposal • How does the proposal address: • Originality and innovation • Inter-sectorial and inter-organisational collabor°

15. EMJDs AWARD CRITERIA • Partnership experience and composition (25%) • How welldoes the proposaldescribe and justify: • The composition and the expertise of the partnership and its relevance to the objectives of the proposal • Relevant international cooperation and networkingactivities of all partners • Diversity and complementarity of the partners in terms of activities and types of organisations • The inclusion and appropriateness of the professionalsector

16. EMJDs AWARD CRITERIA • European integration and functioning of the EMJD (20%) • How clearis the description of integration aspects of the proposal in terms of: • relevance and coherence of the mobilityperiods • Common mechanisms and standards for the selection, admission and review of candidates • the rationale and method of calculation of participation costs and how these are going to beusedwithin the consortium • Existence of joint supervision and monitoring mechanisms of the candidates to ensure the highestquality of outcomes • Clarity about the degree to beawarded and its recognition and possible measures for a possible joint degree

17. EMJDs AWARD CRITERIA • Provision for EMJD candidates and fellowship holders (15%) • Whatdoes the proposalforesee for: • The strategy to attact excellent candidates • The services thatwillbeoffered to the selected candidates • The linguistic support and policy in order to provide candidates the possibility for the use of at least twodifferentEuropeanlanguages • The overall management of the fellowshipscheme, the consortium mechanisms for the use of employmentcontracts and a clear and documented justification in case of the use of stipends • The furthercareer of the candidates and • The clarity of rights and obligations of the candidate and the consortium documented on a Doctoral Candidate Agreement

18. EMJDs AWARD CRITERIA • Programme management and quality assurance (15%) • How clear and sound are the proposed arrangements for cooperation among the partners and how strong is the institutional commitment of all of them • What is the planned sustainability strategy of the consortium to continue after Community funding and how convincing is it (in terms of targets and possible complementary funding possibilities explored and secured) • What will be the consortiums own internal evaluation methods and what are provisions for using external bodies for review and feed-back?

19. Erasmus Mundus Action 1Joint Masters CoursesSpecificities

20. WHAT IS AN EMMC ? • A 1 to 2 years integrated programme (masters level) • delivered by a consortium of universities located in at least 3 European countries • including a mandatory study period for students in at least 2 of the European countries of the consortium • leading to the award of a fully recognised double, multiple or joint masters degree • offering EM scholarships for students and scholars (EU and non EU).

21. AN EMMC MUST • Be fully developed at the time of the application - ready to run 5 consecutive editions • Take the measures to recruit top class students and invite high quality scholars • Support candidates in obtaining their visas & residence permits • Independently from teaching language, provide for the use of min. 2 European languages spoken in the student’s host countries. • Provide insurance coverage to students

22. OVERALL ASSESSMENT  What is the added value of the EMMC proposals from the innovation, academic, research, employability points of view, at EU and worldwide level?  Is the proposed joint course excellent, solid, reliable and coherent enough, from the content, organisational and structural points of view, to guarantee that this expected added value will be actually provided?

23. EMMC Award Criteria A1 - Academic quality - Course content (30 %) • added value of the course in terms of academic quality/innovation (EU and worldwide)? Contribution to excellence of EU HEI • academic quality/coherence of the course? Contribution of the partner HEIs in terms of resources and expertise (added value of third country HEI if any)? • interaction with non educational actors and impact on academic quality, long term sustainability, students employment prospectives?

24. EMMC Award Criteria A2 - Course integration (25%) How well is the course integrated in all its aspects? • based on a truly integrated course recognised in participating countries and leading to the award of a double, multiple, joint (even better) degree? • joint student selection and examination procedures defined by the consortium? • participation costs been jointly calculated taking into account the needs and resources of the course as a whole?

25. EMMC Award Criteria A3 - Course management, visibility and sustainability measures (20 %) • Has the consortium set up suitable management mechanisms/tools/structures that will result in an effective implementation of the course, and rely on the partners’ high level of commitment and effective cooperation? • How will this cooperation lead to sound management of financial resources, ensure efficient course promotion and attractiveness and lead to a strategy for long term sustainability?

26. EMMC Award Criteria A 4- Students’ services and facilities (15%) • Which services offered to students / invited scholars to ensure that the best ones apply and fully devote their time and attention to the course? i.e information provided to students prior to their enrolment , support for visa , hosting services, insurance scheme, language facilities, networking resources

27. EMMC Award Criteria A 5 - Quality assurance and evaluation (10 %) • How will the consortium organise the evaluation of the content but also of the management of the course and involve students and scholars? • Will they develop joint evaluation mechanisms also relying on the expertise of external experts?

28. Erasmus Mundus Action 1The Assessment ProcedureandThe Role of Independent Experts

29. Experts’ Role and Obligations (1) • Experts have been selected on the basis of their application to the « EACEA Call for Experts » • Their role is to • Provide comments and recommendations to decision making bodies AND to the applicants • Their obligation is to • Provide an independent, impartial and objective assessment • Guarantee the total absence of conflict of interest with the proposals assessed and/or discussed in the panels • Keep all the information gathered during the assessment phase confidential • Respect the assessment timetable and practical instructions • Provide coherent, meaningfull, usefull (and understandable!) comments to the applicants

30. Experts’ Role and Obligations (3) • Experts Assessment must • be focused ONLY on the award criteria as defined in the Call • Cover both the academic excellence ANDthequality of the joint programme’s integration • We have tried to map the proposal’s thematic field with the experts profile but this has not always been possible EVERYTHING IS IN THE CALL AND THE EXPERTS MANUAL SECTORAL PANELS WILL BE USED TO CLARIFY PENDING ISSUES (INCL. THEMATIC ONES)

31. Individual assessments • Experts have to: • Provide comments for each Award Criteria covering each individual issues / questions addressed • Provide an individual score for each of issue / question raised under each Award criteria • Provide a selection recommendation • Highly Recommended • Recommended • Not Recommended • Provide Comments to EACEA (if applicable)

32. 5 Excellent 4.5 4 Very Good 4.0 3.5 3 Good 3.0 2.5 2 Fair 2.0 1 1.5 Poor 1.0 0 Marking Scale Cannot be improved /High degree of agreement among evaluators Some very good points, Good overall with respect to the criteria, improvement possible Some good points and some weaknesses. Fair overall with respect to criteria Notable weaknesses in relation to the criterion Poorly presented, confusing information or poor technical content Fails to address criteria or missing information Marks must not go beyond half points Mark of “0,5” should not be given

33. Consensus discussions Purpose: to assure that both experts have a common understanding of the proposal: • If divergence of final scores is more than 20 points, (globally OR per criterion) experts 1&2 discuss to see reason for the differences. If necessary the lead expert can help. If the divergence is confirmed a third expert will assess the proposal

34. Consolidated Assessment • Expert 1, in close collaboration with Expert 2, will be responsible for • drafting the final text of the consolidated assessment • providing the final scores (by default, the assessment tool will suggest the mathematical average between the two scores but experts can revise it in accordance with their consensus discussion) • The consolidated Assessment will have to be endorsed (/signed) by both experts

35. Experts experienced in assessing EM Action 1 Joint Programmes Will coordinate the work of the independent experts within each of the 6 (2 x 3) thematic groups, from the content (coherence, completeness, relevance of the comments and scores) and procedural (/timing) points of view Will stay in close contact with their experts and EACEA during the assessment exercise Will assist the individual pairs of experts during the consensus discussions Will act as 3rd assessor in case of divergence Will chair the thematic panels and assist the Selection Board for the production of the final ranking list of proposals Role of Lead Experts

36. Questions?