On the Economics of University Ranking Lists: A Critical Approach

1. On the Economics of University Ranking Lists: A Critical Approach Ádám Török U of Veszprém, TU Budapest and Hungarian Academy of Sciences Conference on Universities and Innovations, U of Pécs October 21, 2005 - Török

2. The Research Question(s) • Is there any real competition between universities (for market shares, students, human and financial resources)? • If yes: how could the international dimension of this competition be assessed? • Are (increasingly widely quoted) international ranking lists of universities good measures of universities’ competitiveness? Conference on Universities and Innovations, U of Pécs October 21, 2005 - Török

3. University rankings: an introduction • Several rankings published regularly since the late 1980s (proliferation), three major sorts: • Universities • Economics departments • Business schools or MBA programs • Purposes of comparisons: • Ranking scientific excellence • Valuating investments in degrees with respect to future chances on labor markets • Estimating competitive positions of universities for fundraising/recruiting Conference on Universities and Innovations, U of Pécs October 21, 2005 - Török

4. Brief survey of literature I • Much published on comparisons and rankings of economics departments (see on literature Thursby, JEL, 2000), but less on entire universities • Econ dept comparisons narrow in scope: • focus on output in publications and PhDs, • results very similar for several comparisons, • innovation aspect missing • Regular comparisons of MBA programs (e.g. Business Week, Financial Times) with focus on return on personal investment, and rather in business press: • Parameters include: international character of students/faculty, post-degree earnings, strength of alumni network, business relations Conference on Universities and Innovations, U of Pécs October 21, 2005 - Török

5. Brief survey of literature II • Competitiveness aspect not explicitly mentioned, and also neglected in rankings • Very little theoretical basis for university comparisons in spite of 20 year-old tradition • First U comparison: US News and World Report,1983 • Regular ones since 1990: The Times, Fin Times, Guardian, Der Spiegel, Die Zeit (most of these only national) • Comprehensive int’l university comparisons have been published only in recent years • Those by Times Higher Education Supplement and Shanghai Jiao Tong University the most widely quoted Conference on Universities and Innovations, U of Pécs October 21, 2005 - Török

6. General comments on int’l U lists I • Wide international samples • We do not consider purely national lists such as: Times Top 100 Universities for UK, DAAD for Germany, Mihályi (2002) for Hungary • National definitions of „university” accepted from all over the world • Few parameters, quantitative/qualitative sometimes confused • Some universities (e.g. Caltech) can perform well with very small teaching load Conference on Universities and Innovations, U of Pécs October 21, 2005 - Török

7. General comments on int’l U lists II • Size does not matter? • In some countries (e.g. India, Singapore, Russia) centralized national „super-universities” exist with good chances of faring well on lists • Performance criteria of universities compared not always clear: • Research output? • Innovation output? • Mass (graduate) degree output? • Quality (PhD) degree output? Conference on Universities and Innovations, U of Pécs October 21, 2005 - Török

8. Our comments on university lists • More closely examined: Jiao Tong list (Top 500) and The Times Higher Education Supplement (Top 200) list (others also considered) • Our structure: • Presentation of list • Presentation of ranking criteria • Comments on list • Comments on ranking criteria • Features to be emphasized • Overall assessment of given list Conference on Universities and Innovations, U of Pécs October 21, 2005 - Török

9. The Shanghai Jiao Tong list I. Conference on Universities and Innovations, U of Pécs October 21, 2005 - Török

10. The Shanghai Jiao Tong list II. Conference on Universities and Innovations, U of Pécs October 21, 2005 - Török

11. Comments on the Jiao Tong list • Dominance of North American universities in the world, British universities in Europe greater than on other lists (17 US, 2 UK, 1 JPN in TOP 20, i. e. non-US: 15% in first 20) • In strong conformity with general perceptions of university strength • Private universities (mainly from US) more strongly represented • Validity of list questionable for positions below 100 (e.g. 100 U’s tied for position 202) Conference on Universities and Innovations, U of Pécs October 21, 2005 - Török

12. The Jiao Tong Criteria Conference on Universities and Innovations, U of Pécs October 21, 2005 - Török

13. Comments on the Jiao Tong criteria I. • A pure research center called „university” could also perform well (no criteria linked to teaching) • Inherent biases of publication counts (cf. Coupé, Simonovits) • Q: how are co-authored publications counted? • Past academic performance degressively, but even so too strongly considered • Are Nobel Prizes and Field medals of teachers good measures of scientific excellence (and the inclusion of former students…)? • U of Szeged: shown as one of best in region due to Szent-Györgyi’s Nobel Prize in 1937 Conference on Universities and Innovations, U of Pécs October 21, 2005 - Török

14. Comments on the Jiao Tong criteria II. • Usefulness of citation indexes for measurement of institutional R+D performance widely debated in literature • Size of institution often exogenous (depending on policy decisions) • „Integration” processes in order to create universities with more international weight • The presence of a medical school within an institution may greatly increase staff numbers and external funding Conference on Universities and Innovations, U of Pécs October 21, 2005 - Török

15. Specificities of the Jiao Tong list • Great emphasis on human factor and individual scientific excellence • Measurement is partly retrospective • List is size-factor adjusted: • Open debate in literature on whether absolute or size-weighted ranking lists in measuring R&D and innovation performance deserve priority • Very strong dominance of US institutions • Opinion of scientific/higher education community not considered • International dimension of „competitiveness” left in background (or vague) Conference on Universities and Innovations, U of Pécs October 21, 2005 - Török

16. Our assessment of the Jiao Tong list • Unsophisticated technology of comparison: • Simple to understand, some parameters/arguments (e. g. Nobel Prize based indicators) easily accessible to public • Only first 100 positions more or less realistic • List favours complex institutions • Financial/ownership background neglected • Size related matters not entirely clear • Educational output not considered, therefore: • JT list creates impression of only research-based competition between universities Conference on Universities and Innovations, U of Pécs October 21, 2005 - Török

17. The Times Higher Ed Supplement (THES) list I. 1 Harvard University 2 University of California, Berkeley 3 Massachusetts Institute of Technology 4 California Institute of Technology 5 Oxford University 6 Cambridge University 7 Stanford University 8 Yale University 9 Princeton University 10 ETH Zurich 11 London School of Economics 12 Tokyo University 13 University of Chicago 14 Imperial College London 15 University of Texas at Austin 16 Australian National University 17 Beijing University 18 National University of Singapore 19 Columbia University 20 University of California, San Francisco Conference on Universities and Innovations, U of Pécs October 21, 2005 - Török

18. The Times Higher Ed Supplement (THES) list II. 180 University of Southern California 181 Lancaster University 182 University of California, Davis 183 Arizona University 184 Aachen RWTH 185 Queen’s University Belfast 186 Bologna University 187 Norwegian University of Sci & Technol 188 Tulane University 189 Leicester University 190 Rutgers State University 191 Nijmegen University 192 Nanjing University 193 Southampton University 194 Aberdeen University 195= National Autonomous University of Mexico 195= Fudan University 197 Bremen University 198 City University of Hong Kong 199 Virginia Polytechnic Institute 200 Rensselaer Polytechnic Institute Conference on Universities and Innovations, U of Pécs October 21, 2005 - Török

19. Comments on the THES list • Relatively strong presence of non-American universities even in TOP 20 (5 UK, 1 PRC, 1 AUS, 1 JPN, 1 SIN, 1 CH), that is 50% • List strongly differentiates within field of competition: if leader’s (Harvard’s) score is 100%, then 50% belongs to Position 11 (LSE), 25% to Position 39 (Hong Kong University), and Positions 197 to 200 (U of Bremen, City U of Hong Kong, Virginia Polytechnic and Rensselaer Polytechnic) have about 10% • Regional lists (US, EUR, „Rest of the World” [basically Asia and Pacific]) also added/explained Conference on Universities and Innovations, U of Pécs October 21, 2005 - Török

20. Ranking criteria in the THES list Conference on Universities and Innovations, U of Pécs October 21, 2005 - Török

21. Comments on the THES criteria • Great weight (50%) on the peer review component • Important qualitative measure, but may be subjective and influenced by university marketing • University ranking lists could become „self-fulfilling” if this parameter is used • Citations (with all the shortcomings of this measure) also considered as important • At least one quantitative measure of teaching used • This indicator may potentially reflect rigid employment systems in state universities and/or poor student interest • International component has some weight • This indicator may be biased since work permits are not necessarily the easiest to obtain in the countries with the most quality-sensitive education systems Conference on Universities and Innovations, U of Pécs October 21, 2005 - Török

22. Specificities of the THES list • Peer review (questionnaire) method helps include universities from non-OECD countries, but worldwide university reputation counts much • See case of Lomonosov University Moscow: in spite of being hit by severe loss of teaching staff, it still enjoys a good reputation owing to former achievements • Reputation may also be strongly linked to scientific awards: contrast with Jiao Tong list less evident if this aspect is considered • Narrow specialization (e. g. polytechnics) does not seem to pay off in this list • If compared with MBA lists: surprisingly poor showing of Southern Europe here • Picture of absolute excellence is about the same here as on Jiao Tong list Conference on Universities and Innovations, U of Pécs October 21, 2005 - Török

23. Our assessment of the THES list • Gives more balanced picture of higher education in the world than Jiao Tong list • May be considered as a „middle approach” between Jiao Tong focussing on science and MBA lists focussing on business • Attempt at finding compromise between R&D output and teaching output orientation • Still open question: what is THE measure of competitiveness of a university? • Increasing market shares or completing a cultural mission? Conference on Universities and Innovations, U of Pécs October 21, 2005 - Török

24. Economic problems of the university ranking lists • Are firms or non-profit units of education/research compared? • Some U’s (and many more BizSchools) are firms, others non-profit organisations • Forms of both private and state ownership exist in higher education: • A private university is profit and (economic) competitiveness oriented • An endowment or trust based university is expected to fulfil the (sometimes vague) expectations of its founder – what does this mean in terms of competitiveness? • A state university’s performance might be measured only with respect to (sometimes unclear) political criteria and its contribution to the chances of re-election of its principals Conference on Universities and Innovations, U of Pécs October 21, 2005 - Török

25. University rankings: a complex problem of competitiveness analysis • Competitiveness based ranking lists: supply-side and demand-side approach combined • Universities: • Supply-side includes human capital and financing • Demand-side includes output, market shares and possible substitution effects (cf. SSNIP test) • Ranking lists surveyed: some of the most important components of usual competitiveness analysis missing • To be found: elements of human capital (e. g. awards and data on teaching staff) and output (e.g. citations) • Financing aspect and market shares completely missing Conference on Universities and Innovations, U of Pécs October 21, 2005 - Török

26. Our overall assessment: • Serious methodological and economic problems with these lists (technology quite primitive as yet) • Minuses/problems: • Tops of university ranking lists in conformity with general belief (intuition) and prestige – but is this not self-reinforcing? • Midfields differ greatly • However, the strength of US higher education system is well shown by these lists as well – cf. the EU’s Lisbon Agenda!!! • If the lists do not reflect competitiveness, do they simply show quality? • If they show quality, is it related to research or teaching, or both? • Is a good „research university” necessarily a good university? • A plus: • However, the comparative strength of the US higher education system is well shown by these lists as well – cf. the EU’s Lisbon Agenda!!! Conference on Universities and Innovations, U of Pécs October 21, 2005 - Török