Rethinking and Exploring New Ways of Measuring Research Outputs

1. ALTMETRICS: Rethinking and Exploring New Ways of Measuring Research Outputs Theng Yin Leng, PhD PI and Director, Centre for HEalthy& SustainableCitieS (CHESS) Wee Kim Wee School of Communication & Information Nanyang Technological UniversitySingapore 7 October 2015

2. This research is supported by the National Research Foundation, Prime Minister’s Office, Singapore under its Science of Research, Innovation and Enterprise programme (SRIE Award No. NRF2014-NRF-SRIE001-019). • Project Duration: Feb 2015 to Jan 2018 Downtown Singapore Acknowledgements Public Housing Singapore Nanyang Technological University

3. ALTMETRICSRethinking and Exploring New Ways of Measuring Research Outputs Challenges in Measuring Research Impact Outputs “The widely held notion that high-impact publications determine who gets academic jobs, grants and tenure is wrong.” (Michael Eisen 2012 http://goo.gl/t23fpr )

4. ALTMETRICSRethinking and Exploring New Ways of Measuring Research Outputs What is the BIG question? It seems quite obvious that the impact factor is not the best or even a good measure for the quality of a scientific article, a researcher or a research program. But it will take a concerted effort by universities, funding bodies, and researchers themselves to change habits. A collective change in evaluation procedures and personal behavior in academic publishing could boost the development towards the right direction. And it will take new experiments and new tools to develop optimal quality metrics. Both researchers and librarians have a powerful key role in this movement and I am interested how seriously and quickly they will make use of this role in the near future. “… it will take a concerted effort by universities, funding bodies, and researchers themselves to change habits.” http://blog.scienceopen.com/2013/12/journal-impact-factors-time-to-say-goodbye/

5. ALTMETRICSRethinking and Exploring New Ways of Measuring Research Outputs "In the future, tools like these may allow researchers to keep a “live CV” showing up-to-date indicators of their works’ impact next to each product they have produced. Funding organizations could keep timely portfolios of their grants impact across different populations and communities. Tenure, promotion, and hiring committees could assemble research teams whose members have different sets of “impact specialties,” just as sport managers might assemble teams of players with different physical and mental skill sets. Altmetrics also have great potential for supporting personalized recommender systems; citation-based versions of these have been created before [92], but suffer from the long lag between an idea’s insemination and citation; altmetrics could solve this problem. Sufficiently advanced recommendation tools allow the technical act of publishing to be decoupled from relevance and quality filtering [93], allowing every researchers to essentially consume their own private journal, peer reviewed organically by their scholarly network." arXiv:1203.4745v1 [cs.DL] 20 Mar 2012 Altmetrics in the Wild: Using Social Media to Explore Scholarly Impact Jason Priem*, Heather A. Piwowar**, Bradley M. Hemminger* National Information Standards Organisation (NISO) White Paper (June 2014) 5

6. Literature Survey: Papers (2010 – 2014) ALTMETRICS TRADITIONAL METRICS • We inputted the titles, abstracts and index retrieved from Google Scholar and Scopus. • Key words used: altmetrics, research, impact, social, citation, scientific, metrics, analysis, scholarly, data, media, publications, web, article, indicators, mendeley, articles, study, academic • We inputted the titles, abstracts and index retrieved from Google Scholar and Scopus. • Key words used: bibliometrics, analysis, citation, research, medical, journal, article, impact, publication, factor, information, literature, science, topic, publishing, periodicals, humans, index, scientific, human

7. Distribution of Papers in Altmetrics 1. Source: Das, A. K., & Mishra, S. (2014). Genesis of Altmetrics or Article-level Metrics for Measuring Efficacy of Scholarly Communications: Current Perspectives.Available at SSRN 2499467. [pubs]

8. ALTMETRICSRethinking and Exploring New Ways of Measuring Research Outputs Project Overview

9. ALTMETRICSRethinking and Exploring New Ways of Measuring Research Outputs

10. ALTMETRICSRethinking and Exploring New Ways of Measuring Research Outputs • Specific objectives • To carry out a literature survey of traditional metrics and new metrics, that is, altmetrics to measure research outputs. • To develop a framework and a set of hypotheses for cross-metric validation. • To develop new metrics on research impact that can be harvested from social media platforms (e.g. Twitter, Weibo, etc.). • To develop new research impact metrics for the different disciplines, broadly divided into: (i) “hard sciences” research which include sciences, computing and engineering; and (ii) “non-hard sciences” research humanities, arts, business and social sciences. • To develop new metrics to measure research innovation and commercialisation. • Deliverables • List of altmetricsto measure research outputs derived from social media, and applied to “hard sciences” disciplines versus “non-hard sciences” research. • Cross-metric validation findings to identify the effectiveness of the altmetrics compared to traditional metrics. • ARIA, an altmetrics system prototype, for researchers, universities, research institutes and policy-makers. • User interface guidelines for generating alternative interfaces to the ARIA prototype, supported by extensive evaluation. • Publications in internationally reputable journals and conferences. • Training of research manpower including graduate students and other research staff. • Organisation of a workshop on “Altemetrics” in Year 3.

11. Information Studies Computer Engineering • Prof. THENG Yin Leng (PI), WKWSCI, NTU • Ast/Prof. Joanna SIN (Co-PI), WKWSCI, NTU • Ast/Prof. WEN Yonggang (Co-PI), SCE, NTU • A/Prof. SUN Aixin (Co-PI), SCE, NTU • Ast/Prof. XIAO Xiaokui (Co-PI), SCE, NTU Communication Studies External Investigator/Advisor • Prof. Rich Ling (Co-PI), WKWSCI, NTU • Ast/Prof. Winson PENG (Co-PI), WKWSCI, NTU • Prof. Edie RASMUSSEN (Co-PI), UBC, Canada Industry Partners NTU ResearchOffice • Mendeley • Elsevier NTU Library • Ms Samantha ANG, Collaborator • Mr CHIA Yew Boon, Collaborator • Prof. Michael KHOR (Co-PI), PO, NTU

12. Industry Collaboration & Relevance • NTU is a development partner with Elsevier since Jun 2014 and for the next 2 years. • Hence, the Research Team will be working closely with NTU Research Office and NTU Library in collaboration with Elsevier and Mendeley to test and implement tools and metrics developed.

13. ALTMETRICSRethinking and Exploring New Ways of Measuring Research Outputs

14. ALTMETRICSRethinking and Exploring New Ways of Measuring Research Outputs Figure 1a. Interactions for WPs 1-5 leading to WP6

15. ALTMETRICSRethinking and Exploring New Ways of Measuring Research Outputs System Architecture of ARIA prototype (in WP6)

16. Pilot Work & Demo [click]

17. NOVELTY • Scenarios Of Use (A Multi-Disciplinary Approach) • Cross-Metric Validation (Traditional Metrics vs Altmetrics) • Framework and ARIA prototype Traditional metrics Scenario 1: Journal Level Scenario 2: Article Level Scenario 3: Individual Level Altmetrics [more] 17

18. SUMMARY • Motivation/Problem Statement: Research outputs have become very diverse and are increasingly being communicated and disseminated via Social Media. Traditional impact indicators, such as citation counts, are slow in measuring impact, they do not cover new and diverse research outputs, nor do they account for a wider non-academic audience. • Aim/Objectives: Investigate new approaches offered by Social Media to rethink and explore new ways to measure research outputs by comparing traditional metrics with Altmetrics (new metrics based on social media). • Deliverables: Design, develop and evaluate a framework and system prototype: “Altmetrics for Research Impact Actuation” (ARIA), offering new ways of computing altmetrics to measure research impact. • Novelty/Contribution to Research and Practice: Cross-metric validation across different disciplines including “hard sciences”, “non-hard sciences”, research innovation and commercialization. • Beneficiary: Researchers, universities, research institutions, policy-makers, industry and market tie-ups. • Scenario of use: Exploring altmetrics and comparing to traditional metrics. Almetrics Data Sources: Altmetrics Providers:

19. The Blind Men and the Elephant Multi-disciplinary and Inter-disciplinary Efforts Required

20. About ….. Stories of faculty & researchers and their impact (for P&T, CVs, translational research for societal and economic impact, etc) Altmetrics research: It’s also about good and responsible science& communication.

21. Prof. Theng Yin Leng • Principal Investigator • Director, Centre for HEalthy& SustainableCitieS (CHESS) • Wee Kim Wee School of Communication & Information Nanyang Technological University, Singapore • Email: tyltheng@ntu.edu.sg • Tel: (65) 6790-5834 • Dr. Helen Mojisola Erdt • Research Fellow • Centre for HEalthy& SustainableCitieS (CHESS) • Wee Kim Wee School of Communication & Information Nanyang Technological University, Singapore • Email: Mojisola.Erdt@ntu.edu.sg Thank You Contact Details More Info: • CHESS • http://www.wkwsci.ntu.edu.sg/Research/Pages/Research-Centres.aspx • http://www.chess.wkwsci.ntu.edu.sg • Altmetrics Project • http://www.wkwsci.ntu.edu.sg/Research/Pages/Altmetrics-Rethinking-and-Exploring-New-Ways-of-Measuring-Research.aspx • http://www.altmetrics.ntuchess.com

22. Additional Slides

23. ALTMETRICSRethinking and Exploring New Ways of Measuring Research Outputs Theng (lead), Wen, Rasmussen, Sin, Ling, RSO, Library Xiao (lead), Sun, Peng, Ling, Theng Sun (lead), Peng, Xiao, Khor, Theng Peng (lead), Ling, Sin, Rasmussen, Theng Theng (lead), Wen, Sun, Xiao, Sin, Rasmussen, RSO, Library Wen (lead), Theng, Khor, Ling, Peng, Rasmussen

24. WP1: Motivation • With the proliferation of scientific databases, new metrics (or altmetrics) are emerging to indicate research impact. Traditionally, citation is one key impact metric. Lately, a new metric, that is, the number of downloads (Bollen, Van de Sompel, Smith, & Luce, 2005), has started to attract attention. While there has been some research or analysis on factors that influence paper citations, there seems to be a lack of understanding of the relationships between, for example, citations and downloads. • In WP1, we aim to survey and compile a comprehensive list of traditional metrics (e.g. citation counts, impact factors, etc.) and altmetrics (e.g., opinions, social impact, etc.). We will then formulate a framework to conduct cross-metric validation, and ideally identify factors that could help to enhance research impact. [back]

25. WP1: Approach • Objectives: • To survey and compile a comprehensive list of traditional metrics (e.g. citation counts, impact factors, etc.) and altmetrics (e.g., opinions, social impact, etc.). • To formulate a framework to conduct cross-metric validation, and ideally identify factors that could help to enhance research impact. • Deliverables: • One conference paper and one journal paper. • A list of traditional metrics and altmetrics to measure research outputs derived from social media, and applied to “hard sciences” disciplines versus “non-hard sciences” research. • A framework to conduct cross-metric validation, and ideally identify factors that could help to enhance research impact. [back]

26. WP2: Motivation • Existing approaches (for impact actuation based on social media) rely on ad hoc statistics, e.g., • the number of times that an article is mentioned on Twitter (the higher the better) • the number of times that an article is downloaded • However, such statistics are rather inaccurate, e.g., • a bad article could be mentioned a large number of times on Twitter, but in a negative way • when people mention an article, they may be simply re-tweeting others’ posts, without understandings of what the article is actually about • one can easily increase the number of times that an article is mentioned, by tweeting it repeatedly • Therefore, we need a more systematic approach to evaluate research impact based on social media [back]

27. WP2: Approach • Collect large-scale data about people’s discussions of scientific articles on social media • Based on the data collected, develop mathematical models (referred to as information propagation models) about how research-related information is propagated in the social media • A toy example of models: • If an article has high impact, then it has X% probability to be mentioned by at least one high-profile tweet user, after which it has Y% probability to be re-tweeted by Z number of other tweet users, and the re-tweets have P% probability to have positive sentiments • Based on our models, devise methods to quantify the impact of articles, based on data from social media • A toy example: • We observe that an article is mentioned by 2 high-profile tweet users in the first month after its publication, after which it has re-tweeted by 100 other tweet users, but half of the re-tweets are negative. Based on our model, how likely is this article a high-impact work? [back]

28. WP3: Motivation • WP3 will focus on metrics to measure impact actuation for “hard sciences” research which includes sciences, computing and engineering. For example, one would like to understand the Societal Impact of Emerging Topics and Leading Researchers by looking into the life cycle of a specific technology or research. Another example would be the Research on Investment Estimation for National Research Programmes. One interesting metric, the entrepreneurial impact, was exemplified by a study that found that the total market capital of all the companies run or funded by living MIT alumni in 2009 is equivalent to the eleventh economy in the world (Roberts and Eesley, 2009). • In WP3, we would like to explore further different domains to establish alternative metrics for a local university like the Nanyang Technological University (NTU) as a test-bed, working closely with the NTU Research Support Office (RSO) and NTU Libraries. [back]

29. WP3: Approach • Objectives: • To discover and propose a list of altmetrics for the “hard sciences” disciplines. • Working with NTU RSO and Libraries, we will conduct cross-metric validation tests to investigate the effectiveness of the altmetrics compared to traditional metrics for the “hard sciences” disciplines. • Deliverables: • One conference paper and one journal paper. • A software tool to compute the altmetrics for the “hard sciences” disciplines. [back]

30. WP4: Motivation • Digital technologies promise new forms of space for action and interaction by individuals and communities, and open up new avenues of research making it possible to answer questions that were previously unanswerable with conventional and traditional research metrics on “non-hard sciences” research which include humanities, arts, business and social sciences. For example, one would like to understand the impact of a piece of work on society. • WP4 will focus on quantifying research impact on the “soft sciences” disciplines such as measuring the impact of investments in the arts. [back]

31. WP4: Approach • Objectives: • To discover and propose a list of altmetrics for the “soft sciences” disciplines. • Working with NTU RSO and Libraries, we will conduct cross-metric validation tests to investigate the effectiveness of the altmetrics compared to traditional metrics for the “soft sciences” disciplines. • Deliverables: • One conference paper and one journal paper. • A software tool to compute the altmetrics for the “soft sciences” disciplines. [back]

32. WP5: Motivation • In technology development, patent statistics are important innovator indicator to predict impact of technology that would transform the industry, leading to commercialisation. With the access to a huge volume of data, similar techniques could be applied to predict research trends in scientific pursuit. • In WP5, advanced machine learning techniques can be used to crunch the available data and better understand the model of research innovation. It can be further explored to predict future research trends, and important to the universities, research institutes and policy-makers. [back]

33. WP5: Approach • Objectives: • To discover and propose a list of altmetrics for innovation and commercialisation. • Working with NTU RSO and Libraries, we will conduct cross-metric validation tests to investigate the effectiveness of the altmetrics compared to traditional metrics for the innovation and commercialisation. • Using advanced machine learning techniques, we will develop a model of research innovation and a software tool to predict and visusalise future research trends. • Deliverables: • One conference paper and one journal paper. • A software tool to compute the altmetrics for innovation and commercialisation. • A software tool to model, predict and visualise research innovation and comercialisation. [back]

34. WP6: Motivation • Based onfindings from WPs 1-5 and the software tools developed, WP6 will focus on the design, development and evaluation of a system to allow users to obtain altmetrics on research impact actuation. • Using established usability inspection techniques, for example, heuristic evaluation and cognitive walkthrough, and use cases (or scenarios of use), we will conduct evaluation on the ARIA prototype in order to “gather information about the usability or potential usability of a system to improve features within an interface and its supporting material or to assess a completed interface” (Dix et al., 1997). This involves observing and interviewing 15-20 subjects. • Findings from our proposed suite of WPs in WP5 will form the basic understanding of user issues pertinent to the design of usable and useful elements on the displays (interfaces) of a self-management system to obtain measurements of research outputs. [back]

35. WP6: Approach • Objectives: • To design and develop an altmetrics system prototype called ARIA for researchers, universities, research institutes and policy-makers. • Using extensive user studies, we will evaluate alternative interfaces to the ARIA prototype for different users and stakeholders. • Deliverables: • One conference paper and one journal paper. • ARIA prototype with the respective software tools developed in WPs 2-5 to compute altmetrics to complement traditional metrics to measure research impact. [back]