1 / 23

Bibliometrics : Measuring the Quality of Research

Bibliometrics : Measuring the Quality of Research. Professor John Mingers Kent Business School , May 2013 j.mingers@kent.ac.uk. You have had the idea, done the research, written the paper, got it published – isn’t that the end?

sen
Télécharger la présentation

Bibliometrics : Measuring the Quality of Research

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Bibliometrics: Measuring the Quality of Research Professor John Mingers Kent Business School, May 2013 j.mingers@kent.ac.uk

  2. You have had the idea, done the research, written the paper, got it published – isn’t that the end? No! Now people want to evaluate the quality of the research. Is it world-class? Internationally good? Merely average? There are essentially two ways of evaluating the quality of a paper: Peer review – subjective judgements by experts Metrics – measuring the reactions of other academics quantitatively by their behaviour – primarily citations“Peer review by the whole world”! A. Measuring research quality

  3. Why is measuring quality important? Jobs and promotion Allocating funds, e.g., grants, Government funding League tables, e.g., REF It can occur at different levels A single output, e.g., a paper An academic A department A journal A university/multiversity

  4. Peer review Seen by many as the “gold standard”, hence the REF is still based on peer review, but It is extremely time consuming and expensive Experts can genuinely disagree (cf referees) There are unconscious and conscious biases It can be very conservative – the “Golden Triangle” – and can create an old boys club which can be difficult to enter

  5. Metrics (bibliometrics, scientometrics, infometrics) • Generally, the quantitative study of published or recorded information. In terms of evaluating research it takes the number of times a paper has been cited as a surrogate for its quality. • Citations really concern impact which is not quite the same. A paper could be cited because it is wrong! • Many papers are never cited – are they therefore worthless? • There are problems with recording or measuring citations, especially in the social sciences and humanities

  6. 7. Using the journal as a measure of quality • It is common, in some subjects, to use the quality of the journal as a surrogate for the quality of the paper • Not all papers in a journal are of similar quality • How do you measure the quality of the journal? Peer review or metrics!

  7. Citation databases – Thompson-ISI Web of Science/Knowledge (used to be SCI, SSCI) or Elsevier Scopus These record all the references for a given set of journals (WoS - 12,000) and then allow them to be searched by key words or cited authors They also now do citation analyses for both individuals and journals. They are rigorous and generally reliable but limited in coverage, especially in the social sciences (40% - 70%) and humanities (20% - 50%) They do not cover books at all, and only some conferences B. Where do citations come from?

  8. Google Scholar – this searches the web, like Google, looking for references to academic papers and books • Covers all kinds of outputs other than journals • Generally finds many more citations • Is equally good for all disciplines (about 90%) • But, it is not rigorous picking up references from teaching material, home pages etc not just research journals or books • The data is poor quality – spelling errors etc. – the same paper may occur many times and the authors’ names may be wrong • Best accessed through Publish or Perish (http://www.harzing.com/pop.htm)

  9. C. Citation behaviour I - Distributions Citations in 6 good quality journals over a 15 year (1990-2005) period for 600 papers

  10. Citation behaviour II - Dynamics Citations in 6 good quality journals over a 15 year (1990-2005) period for 600 papers

  11. You cannot predict the final number of citations from what happens in the first few years: • Shooting stars: papers that gain a lot of citations very quickly, often because they are empirical reviews of fashionable topics but then die away • Sleeping beauties: papers are are not cited initially but then get recognised and cited for a long time. Often because they are ahead of their time when first published

  12. Total number of citations for a person, department etc Person A: 1 paper, 50 citations; Person B: 10 papers, 5 cites each Ignores how many papers have produced the citations Citations per paper (CPP) – average citations per paper A is still 50, B is only 5 This is the standard measure in one form or another, but It now ignores the quantity of outputs completely, indeed it actually discourages you from producing more papers as they may bring down your score C: 1 paper 50 cites – CPP=50; D: 10 papers, 30 cites each – CPP=30 Citation rates vary significantly across the disciplines so cannot be compared without some form of normalisation to the disciplinary norms (cf Leiden methodology) They also vary with time – the older a paper the more citation it will have Can be skewed by some very high citations (cf h-index) Depends on robust data (cf h-index) D. Basic citation metrics

  13. Journal Impact Factors • A 2-year CPP. The number of citations in year n to papers published in years n-1 and n-2. e.g.,AcadMgt Review has IF 6.169 so this is the average citations in 2011 to papers published in 2009 and 2010. For management this is very high – many journals are less than 1.0. • But, in science we can get much higher figures: Ann Rev Immun– 52, Physiol Rev 36, Nature 31 • These are very short term. There is 5-year IF, but in social science citations may not reach their peak until up to 10 years

  14. Normalisation – the Leiden Methodology for a department • Calculate the CPP for all the citations for all the papers over a fixed time period (e.g., 5 years), let’s say it is 6.0 cites per paper • Look at all the journals from the field (e.g., Management), as defined in WoS, and calculate the CPP for all the papers in all these journals over the same time period, suppose it is 3.0 • Then the field normalised CPP is 6.0/3.0 = 2.0. So this department is getting cites at double the field average

  15. The h-index • A relatively new measure created by a physicist (Hirsch) in 2005 that is simple to understand and combines both impact and productivity:0, 0, 1, 2, 4, 8, 9, 9, 10, 15, 25, 50, 100There are 8 papers with at least 8 citationsHis paper has 2700 citations! • In order to get a high h-index you need both a lot of papers and high citationsA : 1 B: 5 D: 10 • Advantages: it is a measure of overall contribution, not just quality; easy to understand; robust to poor data; can be used for journals, departments etc • Disadvantages: it ignores the higher citations; it is not pure quality but productivity – so Einstein would have a very small h-index; it is very time dependent and only ever gets bigger and so disadvantages early researchers – can use the h-rate (h-index/years publishing): anything over 1 is good (in social science)

  16. In an empirical analysis, I found the following to be the major significant factors: Quality of journal. This is problematic – better journals have better papers in the main - but the effect was very strong Institutional rank of the lead author (Times Higher) but this could be just a marker for the quality of the author Type of paper – review, theoretical, empirical Size – pages and references In my experience: Be innovative and ahead of the game (sleeping beauties) Provide something valuable – datasets, reviews – state of the art, applications, literature Go for the best journal you can (5 yr IF) E. How can you improve your citations?

  17. References These are my papers but they will provide a way into the literature Mingers, J., and Q. Burrell. Modelling Citation Behavior in Management Science Journals. Information Processing and Management 42, no. 6 (2006): 1451-64. Mingers, J., and A.-W. Harzing. Ranking Journals in Business and Management: A Statistical Analysis of the Harzing Dataset. European Journal of Information Systems 16, no. 4 (2007): 303-16. Mingers, J. Exploring the Dynamics of Journal Citations: Modelling with S-Curves. Journal Operational Research Society 59, no. 8 (2008): 1013-25. Mingers, J. Measuring the Research Contribution of Management Academics Using the Hirsch-Index. Journal Operational Research Society 60, no. 8 (2009): 1143-53. Mingers, J., and L. Lipitakis. Counting the Citations: A Comparison of Web of Science and Google Scholar in the Field of Management. Scientometrics 85, no. 2 (2010): 613-25. Mingers, J., and F. Xu. The Drivers of Citations in Management Science Journals. European Journal of Operational Research 205, no. 2 (2010): 422-30. Mingers, J., F. Macri, and D. Petrovici. Using the H-Index to Measure the Quality of Journals in the Field of Business and Management. Information Processing & Management 48, no. 2 (2012): 234-41. Mingers, J., K. Watson, and M.P. Scaparra. Estimating Business and Management Journal Quality from the 2008 Research Assessment Exercise in the UK. Information Processing and Management 48, no 6(2012): 1078-1093

  18. Open Access Publishing • After the Finch Report the UK is moving rapidly towards open access publishing. What does this mean for us? • Forms of OA • Green – much like now, in repositories such as KAR. Does not cost anything but there may be delays for embargo period although increasingly this is ignored • Gold – the published version of the paper is immediately available to allIn most journals this will cost money • Article Processing Charge (APC) – up to £3000 per paper! There may well develop differential charges depending on the quality of the journal • Who will pay – the universities basically. And for the change-over period they will pay subscriptions as well. • RCUK have said that all papers coming out of their funds should be gold access • The REF 2020 has said that it will only accept open access papers but not clear if this includes green

More Related