May 11, 2015

1. May 11, 2015 The evaluation of research units at HCERES Alain Cappy, coordinator for the S&T domain

2. Outline • 1-HCERES présentation • Status and Organization • Founding Principles • Missions and Objectives • EU Recognition • Annual Work • IntegratedE • 2-Evaluation of research units • Principles • Report • Syntheses • 3- Conclusion

3. HCERES: Status and Organisation • Status: Independent Administrative Authority • Organization • President, chairs the board • Board of the Council: 30 members • 3 evaluation departments • Institutions • Research units • Programs and degrees • Observatory of Science and Technologies ( data base, bibliometric analysis, indicators…) • Support service

4. HCERES: Status and Organisation • Scientific organization • About 100 part-time scientific delegates • 4,500 experts commited every year, including 20% outside France • Administrative organization • About 100 full-time administrative and technical staff • Budget in 2015: about15 M€

5. Founding Principles • Independence: • Status of independent administrative authority. HCERES to carry out its missions without coming under pressure from: • government authorities, • Evaluated entities ( research units, Universities, research bodies), • or any other stakeholders • HCERES guarantees compliance with the rules of ethics, in particular those concerning conflicts of interests.

6. Founding Principles • Competence of experts and transparency of procedures • Quality of evaluation rests on: • the expert’s skills and correct matching his/her profile with the entity evaluated, • a methodology and procedures guaranteeing equality of treatment, • drawing up by HCERES of a quality management system. • Transparency of evaluation lies on publication (website) of : • criteria for and methods of evaluation, • the list of scientific delegates, experts and their positions, • Summary of evaluation reports.

7. HCERES: Missions and Objectives • Evaluating: • HE & R institutions (universities, schools and research bodies), taking account of all their missions and activities. • Research activity laboratories and research units of these institutions • Education programs and degrees in the French higher education system (BMD) • Assisting evaluated entities in the continuous improvement of their practices • Providing clarification for decision-makers: • The State (financing, recognition of qualifications) • Research bodies (certification of laboratories) • Others (Local Authorities, Companies, Families)

8. What HCERES Do and Do Not • HCEES evaluates and does NOT decide • Decisions about accreditation and funding belong to Universities, Institutions, State,… • HCERES does NOT evaluate persons • But representatives of person evaluation bodies are included at different levels : • HCERES Council • Visiting committees

9. European Recognition • HCERES is member of the European Association for Quality Assurance in Higher Education (ENQA). ENQA promotes European co-operation in the field of quality assurance in higher education . • HCERES is listed in EQAR, the European Quality Assurance Register for Higher Education.

10. Annual Work • Institutions: • 60-70 universities and schools • 3 research organisms • Research units: • 600 evaluation committees • Training courses: • 700 Bachelor degrees • 300 Master degrees • 50 doctoral schools

11. The Regional Breakdown of the Evaluation Campaigns A 5-Year Period

12. Integrated Evaluation For one institution ( University, Engineer School…) Evaluation of research units including on site visit Evaluation reports Evaluation of institution including on site visit. Evaluation of training programmes, ( Bachelors, Masters, Doctoral Schools) Evaluation reports

13. Document Management System • Web based evaluation process (GED) • File and report deposition • Evaluation committee creation • Flags to follow the state and progress of a specific unit evaluation

14. Document Management System (GED)

15. Evaluation of research units

16. Methodological Principles • The criteria defined and procedures implemented by the High Council take into account: • the diversity of the natures and missions of the research units • the diversity of disciplinary fields( Life Sciences, Social Sciences and Humanities, Physical and Engineering Sciences) • based on peer-assessment (national and international experts) • Evaluation goals: • Oriented towards improvement of unit performance • Combines quantitative data ( scientific production, impact) with qualitative information assessed by experts

17. Evaluation of Research Units: for Who ? • the recipients of the evaluation are: • Research units and researchers • research organisms, Universities, Grandes Ecoles • Students • Local Authorities (regions, etc.) • Ministry for Higher Education and Research and other ministries

18. Evaluation of Research Units • The annual task: evaluation of all State funded research units in specific Regions ( France is divided in 5 evaluation campaigns) • ~ 600 research units • > 3000 experts committed every years for the peer review • Covering ALL research bodies supported by public funds, including private universities, architectural and cultural research, and possibly clinical research…

19. Research Unit Evaluation Process Main steps The research unit writes a report following a model. This report is deposited on the HCERES document management system (GED). A HCERES scientific officer is designed to manage the evaluation An evaluation committee is appointed by HCERES The committee prepares a draft of the report before the visit A visit (1, 2 or 3 days ) of the research unit is organised. During the visit, the unit presents its main results and strategy. The committee has meeting with the unit personnel

20. Research Unit Evaluation Process Main steps (cont’d) After the visit the committee finalised the report The report is controlled by HCERES to be sure that it follows HCERES writing rules The report is transmitted to the unit and its supervising institutions. A summary of the report is published on the HCERES website.

21. The Evaluation Report • The report is the main output of the evaluation. • The report should be critical, not descriptive, • The report must contain evaluation details of the research unit at a whole but also for each research team, • The report has to specify strengths, weaknesses and recommendations.

22. Evaluation Report • The evaluation report: the six evaluation items • Scientific outputs and quality of research • Academic reputation and appeal • Interactions with the social, economic and cultural environment • Organization and life of the unit • Involvement in training • Strategy and five-year plan • These six items are used both for the research unit as a whole and for the research groups

23. Evaluation of research units • Synthesis by geographical site: • In addition to the research units evaluation reports, HCERES writes research syntheses : • Goal: To give a schematic view of the particularities of a geographical site based on: • The characterization of working forces in the three domains (Social sciences and Humanities (SH), Life sciences (LS),Physical and Engineering Sciences (PE)) • The analysis of evaluation reports in each domain • The synthesis contains two parts • - the workforce • - the synthesis of the evaluation reports • Specific syntheses can be produced by order

24. Breakdown of workforce: Grenoble

25. Breakdown of workforce: Grenoble

26. Open questions • Evaluation is a difficult task. Many questions are still open • How to evaluate interdisciplinary research ? • How to measure the impact of a research ? • How to efficiently use bibliometric data • HCERES working groups and seminars are organized to tackle these questions

27. Conclusion • HCERES, a ‘High Council for Research and Higher Education’ devoted in France to the evaluation of Institutions, research units and training courses • HCERES is an independent authority • HCERES performs an integrated evaluation in each site • HCERES is member of ENQA and EQAR • All the French research units and research groups are evaluated every 5 years