Dag W. Aksnes

1. 7th euroCRIS strategic seminar, Brussels Recording ResearchUse in research management and evaluations Dag W. Aksnes

2. Background • NIFU STEP is a Norwegian research institute doing contract research, analyses and consultancy for various organisations such as ministries and research councils. • The main focus is on studies of education, research and innovation. One mission to provide knowledge for policy decisions within these areas and analyze the results and effects of policy through e.g., evaluations and studies on productivity. • This presentation will focus on how CRIS data have been used in Norway in relation to research management and evaluations.

3. CRIS-data Norway • Data on all types of scientific/scholarly publications in all fields of research in the Higher Education Sector are collected trough two documentation systems (Frida, ForskDOK). • In the database the publication output is reported by the institutions as ordinary bibliographic references. • A dynamic authority record of so far 19,000 controlled scientific and scholarly publication channels ensures that references to non-scientific publications are not entered into the system. • Publication data from professional bibliographic data sources (e.g. Web of Science) are imported to the documentation system in order to facilitate the registration of publications by the employees.

4. Relevance of bibliometics in science policy/evaluations • Number of publications used as an indirect measure of knowledge production • The extent to which the publications have been cited in the subsequent scientific literature used as an indicator scientific impact and international visibility. • Co-authorship used as an indicator of collaboration, e.g. the extent of international collaboration • The journal pattern used for obtaining information on the subfield distribution of the research activities and the extent of presence in leading journals • In turn, such indicators can yield much interesting information on various issues relevant for evaluation purposes.

5. Bibliometric analyses of research performance • Traditionally, bibliometric analyses of research performance have usually been based on the ISI-database (Web of Science, Thomson Reuters) • Multidisciplinary • Structured data well adapted for bibliometric analyses • Citation counts • Covers mainly publications in international journals • In Norway the CRIS-database has now been used as an additional source for bibliometric analyses • The CRIS database is well suited for this purpose: • The bibliographic references in the documentation system are standardized and analyzable by publication channel and type of publication. • Complete, verifiable and structured data • Covers all scientific/scholarly publications not only journal articles. • Non-scientific/non-scholarly publications can be excluded.

6. Using CRIS data for funding purposes and analyses • In 2004, Norway implemented a bibliometric model for performance based budgeting of research institutions. • The funding of these institutions is now partially based on the measurement of their scientific and scholarly publishing. • The CRIS databases are used for this purpose. • Other applications: analyses of the research output and evaluations

7. Bibliometric analyses of research performance • Particularly the humanities and the social sciences the ISI-base covers a small proportion of the research output • Due to poor coverage and a publication pattern where international journals are less important. • Example: publication types and ISI-coverage in major scientific and scholarly fields:

8. Figure 1. Publication types* in five major areas (N=24,154) *) Weighted: 1 book = 5 articles

9. Figure 2. Publication types* in social science disciplines (N=5,616) *) Weighted: 1 book = 5 articles

10. Research evaluations • In Norway, the Research Council of Norway regularly carries out evaluations of the national research activities in the various disciplines (peer evaluation by international panels). • NIFU STEP provides bibliometric analyses as background information for the evaluations. • Till quite recently all researchers encompassed by the evaluations have submitted their publication lists. • Bibliometric analyses have been carried out using this as a data source in combination with ISI-data • Time consuming • No structured data • Now the CRIS-database has instead been used as a data source • The researchers do not have to submit their publication lists anymore

11. ”Level 1 and level 2” • In addition to using CRIS as the main data source, information on the “level” of the journals and book publishers are used in the analyses • In the Norwegian system journals and publishers are divided into two levels. • The highest level (level 2) is giving extra weight and includes only the leading and most selective international journals and publishers (account for about 20 % of the world’s publications), • The national councils in each discipline participate annually in determining and revising the highest level under the guidance of the Norwegian Association of Higher Education Institutions.

12. Examples • The following slides show some examples of indicators and analyses provided. • Levels of analyses • Country • Institutions • Departments • Divisions/research groups • (Individuals – anonymous presentation)

13. Figure 3. Number and proportion of articles in leading Chemistry journals – “level 2”, Norway 1998-2007. *) Weighted: 1 book = 5 articles

14. Figure 4. Number of publications per year, 2003-2007, fractionalised counts. Department X. *) Weighted: 1 book = 5 articles

15. Figure 5. Department x. Number of publications per person 2003-2007, fractionalised counts.

16. Table 1. Number of publications 2003-2007, Department X

17. Table 2. Journal profile, 2003-2007 publications (whole counts). Department X

18. Table 3 The most frequently used journals, number of publications, 2003-2007, by groups/sections, Department X

19. Other examples • Analyses of the publication output at the University of Bergen • The University Library has initiated a project providing analyses of the publication output at the university based on CRIS data • Writing of reports being presented for the management at the university and departments • Examples of indicators provided

20. Figure 6. Number of publication points per researcher man year per department

21. Proportion of publication points in ”level 2” channels

22. Conclusions • Data quality of CRIS crucial issue for using the data for these purposes • Accurate and complete • Delimitation of scientific/scholarly publications • Use of authority registers, person and publication IDs (incl. ISI-ID) • Importing of data from professional bibliographic data sources, e.g. Web of science • In a format adapted for bibliometric analyses • The imported records have highest quality – although these data have imperfections as well • Mistakes more often occur for the non-imported records registered by the researchers, e.g. • Duplications • Editorials, corrections etc. registered as full articles • Mistakes in the title and journal title • Lacking information on co-authors