Diana Hicks & Jian Wang School of Public Policy Georgia Institute of Technology March 2009

1. Towards a Bibliometric Database for the Social Sciences and Humanities – A European Scoping Project Diana Hicks & Jian Wang School of Public Policy Georgia Institute of Technology March 2009 Consultative Stakeholder Workshop, Brighton, UK

2. Project tasks • Analysis of literature pertinent to SSH coverage in Scopus, Web of Science, Google Scholar • High level summary of known national evaluation systems with in depth comparison of US (NRC), UK (RAE) and Australia (Composite Index and ERA) • Analysis of the coverage of ERIH and Norwegian lists using Ulrich’s • Proposing solutions

3. Framework: the 4 literatures+

4. Abbreviations, definitions and distinctions • SSH • SS – social sciences • H - humanities • db = Database: • WoS = Web of Science = SCI & SSCI & AHCI = Thomson Reuters = ISI • Scopus = Elsevier’s Scopus • GS = Google Scholar • Ulrich’s – most comprehensive worldwide LIST of periodicals, not a database of articles in periodicals • All publications vs journal articles • Assessing published lists of journals covered in the db or articles actually found in the db. • Source and non-source/citation • Fields: SSH, STEM, all . . . • There is a small rate of error in every database.

5. Ulrich’s • Ulrich’s is the authoritative source of bibliographic and publisher information on more than 300,000 periodicals of all types from around the world • academic and scholarly journals • Open Access publications • peer-reviewed titles • popular magazines • Newspapers • newsletters, and more. • Ulrich’s has been used in bibliometric studies as the benchmark against which WoS and Scopus coverage is measured • Listing all the world’s periodicals, irrespective of language or country of publication is truly ambitious. In large measure Ulrich’s succeeds. Studies have found only very small numbers of journals that are not yet indexed in Ulrich’s. • We found 30-40 journals, all newer, that seemed to exist but were not yet indexed. • We bought 74k records covering active, regularly appearing periodicals in SSH fields.

6. Norwegian model reference list • The reference list of journals that are acceptable within the Norwegian evaluation system. The list covers all fields of science, social science and humanities. • Covers scholarly publications, a scientific or scholarly publication must: • present new insight • in a form that allows the research findings to be verified and/or used in new research activity • in a language and with a distribution that makes the publication accessible for a relevant audience • in a publication channel with peer review • Publications in local publication channels are not counted in the budgeting model. The level of a publication channel is defined by its mix of authors. • International - Authors from several countries; international language(s) • National More than 2/3 of the authors are from the same country • Local More than 2/3 of the authors are from the same institution • Source: G. Sivertsen, Bibliometrics for (or against?) the humanities, presentation 2008 • G. Sivertson kindly shared with us the SSH list containing 8,165 journals. 6,103 could be matched to Ulrich’s records and we analyze those.

7. European Reference Index for the Humanities (ERIH) • ERIH aims initially to identify, and gain more visibility for top-quality European Humanities research published in academic journals in, potentially, all European languages. It is a fully peer-reviewed, Europe-wide process, in which 15 expert panels sift and aggregate input received from funding agencies, subject associations and specialist research centres across the continent. (http://www.esf.org/research-areas/humanities/research-infrastructures-including-erih.html) • ERIH includes good, peer-reviewed research journals in 15 broad disciplines of the Humanities. (http://www.esf.org/research-areas/humanities/research-infrastructures-including-erih/frequently-asked-questions.html) • The 15 fields are: Anthropology (Evolutionary); Anthropology (Social); Archaeology; Art, Architectural and Design History; Classical Studies; Gender Studies; History and Philosophy of Science; History; Linguistics; Literature; Music and Musicology; Pedagogical and Educational Research; Philosophy; Psychology; Religious Studies and Theology • After cleaning, we believe there are 5197 journals in ERIH. 3,942 could be matched to Ulrich’s records and we analyze those.

8. What problems would there be constructing a database from the ERIH list? • Journal status • Errors • Non-scholarly material included • Scientific journals included Why did journals on lists not match into Ulrich’s dataset?

9. Norwegian list coverage English language European language, not English Norwegian list 863 16% Scopus 555 10% Scopus 4,331 35% Norwegian list 4,494 36% WoS, 258, 5% Ulrich's 5,554 100% WoS 2,366 19% Journals counted are: SSH fields published in a European country or U.S. active (not ceased) & published regularly identified as academic/scholarly by Ulrich’s Ulrich's 12,344 100%

10. ERIH coverage English language European language, not English Scopus 250 5% Scopus 1,534 26% ERIH 1,122 26% ERIH, 1,980 33% WoS 199, 6% WoS 1,166 20% Ulrich's 3,577 100% Journals counted are: ERIH fields active (not ceased) & published regularly identified as academic/scholarly by Ulrich’s published in a European country or U.S. Ulrich's 5,948 100% Venn diagrams plotted using: Littlefield & Monroe, Venn Diagram Plotter, US Department of Energy, PNNL, Richland, WA, 2004-2007.

11. Solutions

12. Norwegian model • National research documentation system • Submission • Universities submit bibliographic records and are responsible for data quality • Validation & standardization • Dynamic authority file of allowed publication channels • External peer review • National scale (<2/3 of authors from 1 institution) • 18,000 currently accepted • Thomson Reuters and Norwegian national library data imported • Authority file standardizes: • Names of publication channels • Document types • Institutional affiliations of authors • Institution names • Manual work also needed • Recognizes and addresses known accuracy problem in submitted data • Audit of Australian data (Butler & Visser) • Law study (Nederhof et al.)

13. Norwegian model continued • Counting – simple and transparent • All journals assigned to level 1 & 2 in field specific way, points assigned to pub types: • The point system is fair to all fields because scholars in each field decide what to assign to level 1&2 • Fractional counting (can be debated) • Recognizes and addresses known incentive problem • Australian composite index – just counting pubs increases publication in low quality journals (Butler)

14. Norwegian model commentary • Strengths • Fair to all fields in which written word predominates • SSH journals and monographs incorporated and weighted • Of the “four literatures” takes care of 3: international journals, books, national journals. • The fourth, “enlightenment” can be incorporated in the model if valued by society. • Non-text output could also be incorporated in this mechanism • creative writing publishers, exhibition and performance venues can be catalogued and weighted to assign points • Possible issues • Fractional/whole counting can be debated, but that’s a peripheral detail • National journals likely receive low weight, replicating results of citation analysis in WoS or Scopus • Cost – full cost includes that born by universities in submission and by agency in validation • Difficulty reaching international consensus on journal rankings • No impact measures – i.e. citations, this means simplicity, but also can be seen as a limitation

15. Recommendations • All countries implement the Norwegian model – resulting national evaluation databases are linked together • institutional submission & agency authority file, standardization and validation • ERIH could be used • Book publishers can be categorized and included • Non-text material - extend the principle of peer listing and ranking of venues/vehicles to: • Dance & play performance venues • Publishers of creative writing • Art and craft exhibition venues • building or manufacture for architects and designers • Establish a book database comparable to the journal article database • Support small European SSH scholarly journal publishers to enable their journals to be put online in a central infrastructure to be commissioned by ESF.

16. SSH online journal infrastructure recommendation • ESF builds and maintains an electronic full text SSH journal infrastructure for European SSH • Metadata fields built (author, institution, journal name etc.) • Electronic full text of all articles • Provides access to read 1 page at a time with no saving or printing allowed. • Sell articles cheaply for saving and download with revenue returned to publishers • The US National Research Council uses this model with its reports. • 6 month delay is used in PubMed to protect publishers subscriptions. But immediacy is of little consequence in humanities, PubMed idea must be modified • Requires a peer review process of journals to establish the list • Like ERIH, but with tougher standards to keep costs under control • Advantages • Public money goes to small European publishers • Used to overcome the obstacle to accessibility posed by fragmented publishing industry • Makes scholarship widely available worldwide • Because any full text electronic resource will be indexed by Google scholar, which also enables automatic page translation • Databases are enhanced, WoS and Scopus index the journals • Because infrastructure includes article metadata and passes on a clean file to db’s every month • Problems • May not be needed with WoS and Scopus expanding, further analysis needed.

17. Summary • Norwegian model extended • SSH electronic journal infrastructure for European SSH • It will always be possible to criticize coverage • Set up rules and aim to include everything that meets the criteria • Must be limited in some way, for example refereed, active, regularly appearing etc.

18. Problem: fragmented publishing industry Number of journals from publishers of different sizes Share of journals by publisher size 380 publishers - 588 journals Herfindahl index: 0.012 0.036 0.033 0.024 0.011 Publisher size: large – 10 or more small – 2-9 journals in Ulrich’s SSH tiny – 1 Journals counted are: SSH active (not ceased) & published regularly identified as refereed by Ulrich’s published in a European country or U.S.