1. UNIVERSITIES EVALUATIONS AND RANKINGS Philippe VINCKE Rector of the Université Libre de Bruxelles

2. Evolution of higher Education • New Actors of higher education and research • Increasing mobility of students and researchers • Accountability of the universities, transparency • Evaluations, comparisons, rankings

3. Criticisms of the existing rankings (1) • Competencies of the authors of the rankings • Impossibility, for the reader, to reconstruct and verify the results (rankings are not« scientific ») • No information about the goals, the intended uses, the aimed public • Precise definition of « university » : are they all comparable ?

4. Criticisms of the existing rankings (2) • Choice of the criteria and of their relative importance • Research • Education • Costs • Services • Social aspects • National context, legislation • Financial ressources • Choice of the indicators • Data validation

5. Criticisms of the existing rankings (3) • Bibliometry • Quality of the data • Discrimination among the scientific fields • Different traditions (journals, books, proceedings, number of authors, time span of valid research) • Supremacy of the publications in English • Which indicators ? (IF, citation index, h index, …) • Experts • Do they exist ? • How to choose them ? • Which questions ? How to treat the answers ?

6. Numerical « manipulations » (1) • How is it possible to imagine that complex objects such as universities can be characterized by one number ? • The weighted mean can exclude good candidates Example : A 41 97 B 100 38 C 68 68 • Curious effects of normalization

8. RANKING : A, F, C, H, E,B,G,D

10. RANKING : A,D,G,B,E,H,C,F

11. Before : A, F, C, H, E, B, G, D • One modification of the score of A on one criterion. • No change in the scores of the other universities • After :A, D, G, B, E, H, C, F • Inverse ranking !!

12. Other comments • Rankings are contested but used • Rankings have an influence on reality • Excesses are possible (financial bonus, or incitements,…) • Standardization effect

13. Conclusions • The rankings relayed by the media are not scientifically valid at this stage • Evaluation of research and higher education is a necessity • But it must be realized by competent people in the context of a clear policy and with explicit goals • There does not exist a unique method applicable in all institutions

14. Main questions (1) • Wich « objects » ? • Universities (definition?) • Education programmes • Diploma’s • Research centers • Research programmes • …

15. Main questions (2) • What does one want to do ? • To compare • To select the «  best(s) » • To rank • To define « homogene » categories • To detect strong and weak points • To assign ressources • …

16. Main questions (3) • For whom ? • External autorities, government, … • Potential partners (universities, research centers, companies,…) • External teachers or researchers • Potential students • Funding agencies • Sponsors • Public opinion, media • Alumni • Internal authorities • Internal teachers or researchers • Internal students • …

17. Main questions (4) • For each « situation » (characterized by the answers to the 3 previousquestions): • Whichindicators ? • Quality of the data ? • Numericaltreatment of the data !

18. Different approaches for different concrete questions • Choose the « best » education programme for this student ? (« best » for him) • Allocate financial resources to research centers • Select the universities which could be « good » partners for this company • Identify the strong points of these universities for students interested in studies in that field • Necessity of an interactive decision-aid toolbox for each possible user and question.