Quality, trust and peer review: researchers’ perspectives 10 years on

1. Quality, trust and peer review:researchers’ perspectives 10 years on September, 2019 Survey results

2. Table of Contents

3. Back to contents Background and approach Research objectives are to: Examine the drivers and influences on the communication of scholarly research. This report is about peer review, assessment of research content and helping people outside the research community judge the quality of research evidence About the survey • 3133 researchers responded to a survey of 98160 individuals randomly selected from database of 3.6 million researchers (3.2 % response rate). • Survey tool: Co-branded (Elsevier and Sense About Science) online survey available in English only. Survey took 15 minutes to complete (median average). Fieldwork took place in May 2019. • Results: Responses have been weighted to be representative of the global researcher population by country (UNESCO 2014 data). Base sizes shown in this report are unweighted unless otherwise stated • Statistical testing: Maximum error margin for 3133 responses is ± 1.5% at 90% confidence levels. When comparing main group and sub-group we have used a Z-test of proportion to identify differences between the overall average and the sub-group (90% confidence levels). About Sense About Science • Sense about Science is an independent charity that challenges misrepresentation of science and evidence in public life. About Elsevier • A global information analytics business specializing in science and health helping institutions and professionals progress science, advance healthcare and improve performance for the benefit of humanity.  Differences are indicated by a tick . A green tick indicates the subgroup result is higher than the overall result while a red tick indicates it lower.

4. Back to contents EXECUTIVE SUMMARY

5. Back to contents EXECUTIVE SUMMARY Trust in peer review and communication Satisfaction with peer review has increased significantly since 2015 from 65% to 75%. The shift has been mainly from ‘neutral’ to ‘satisfied’. Peer review is widely seen to be improving the quality of research and keeping scientific communication under control., though only 38% believe it is well understood by the public The preferred format for peer review is two relevant reviewers either with or without input from a member of the editorial team; only 8% think AI/ML would qualify as peer review. Clear guidance on reviewing is considered most important for maintaining a healthy peer review system; only 28% think financial rewards for reviewers are needed. Researchers would like to see a metric of ‘reproducibility’ to help assess the quality of research outputs 86% of researchers doubt the quality of at least some of the research outputs they encounter. To compensate for this they check supplementary material/data carefully, read only information associated with peer reviewed journals or seek corroboration from other trusted sources. The main reasons outputs are considered untrustworthy are poor interpretation, lack of clarity of the peer review process and flaws in the methodology 70% of researchers doubt the quality of at least some of the research profiles they encounter. To compensate for this they use multiple platforms or visit websites of institutions. Reasons for untrustworthiness includeexaggeration of achievements, lack of verification and poor quality outputs. When using tools to find research outputs being able to adjust algorithms/parameters, transparency and interoperability are seen as effective ways to be directed to trusted content When communicating research outcomes to the public explaining research in lay terms is seen as the best way to help people outside the research community judge the quality of research The main causes of lack of public confidence in research evidence are seen to be misinterpretation or deliberate misrepresentation by the media as well as difficulty identifying high quality research.

6. Back to contents OVERVIEW OF RESULTS

7. SATISFACTION WITH PEER REVIEW: Has increased significantly since 2009 from 69% to 75%. Shift mainly from ‘neutral’ to ‘satisfied’. Overall, how satisfied are you with the peer review system used by scholarly journals? % satisfied 2019 75% satisfied 8% dissatisfied Base: All respondents, 2019 (n=3133), 2015 (n=1988), 2009 (n=4037), 2007 (n=3040)

8. ATTITUDES TO PEER REVIEW: widely seen to be improving the quality of research and keeping scientific communication under control. Only 38% believe it is well understood by the public Please indicate the extent to which you agree or disagree with the following statements Ordered by impact on overall satisfaction with peer review Base: All respondents (n=3133)

9. PEER REVIEW SYSTEM: Who should undertake peer review? Most (87%) believe an evaluation of research by two relevant researchers either with or without input from a member of the editorial team qualifies as peer review, only 8% think AI/ML would qualify as peer review. Which of the below qualifies as peer review; does it qualify when the evaluation of a research article is undertaken by: Base: All respondents (n=3133)

10. PEER REVIEW SYSTEM: Clear guidance on reviewing is considered most important for maintaining a healthy peer review system; only 28% think financial rewards for reviewers are needed. Which of the below do you consider most important for maintaining a healthy system of peer review? (select top 5) Base: All respondents (n=3133)

11. ASSESSMENT OF ARTICLES: Researchers would like to see a metric of ‘reproducibility’ to help assess the quality of research outputs When a reader is interacting with research outputs (e.g. research articles, preprints, data), what additional information is helpful for assessing those research outputs? Base: All respondents (n=3133)

12. TRUSTWORTHINESS OF RESEARCH OUTPUTS: Most researchers 62% trust the majority of research outputs they encounter but a third doubt the quality of at least half. To compensate for this they check supplementary material/data carefully, read only information associated with peer reviewed journals or seek corroboration from other trusted sources. Which of the following mechanisms do you employ to compensate for any lack of confidence you have in the content you are considering reading/accessing? Thinking about the various research outputs that you interacted with (or encountered) last week what proportion of the outputs would you consider trustworthy? 37% 62% Base: All respondents (n=3133) Base: All respondents that do not think all research outputs are trustworthy (n=2715)

13. REASONS RESEARCH OUTPUTS ARE REGARDED AS UNTRUSTWORTHY: for those researchers who rate at least some outputs as untrustworthy; the main reasons are lack of or low quality peer review, uncertainty if content is peer reviewed, poor interpretation and flaws in the methodology You indicated that some/all of the research outputs are trustworthy. Why do you think all/some of the content you encounter is not trustworthy? “Content that comes from predatory journals is not trustworthy. It is not rigorously peer reviewed. This diminishes the trust in scientific research. As scientists we need to be held to a high standard. The traditional peer review system does that." (Physician, USA, prefer not to say age) Inhibitors of trust Low quality peer review Not peer reviewed Unclear if peer reviewed Peer review scope "Not familiar with the journals or media/not sure what the peer review process might be, or how reviewers are selected" (Psychology, United States of America, 26 to 35) Forces driving volume of research Components of mistrust Pressure to publish "Experiments poorly designed, some analyses seemed suspect, areas that I know well improperly characterized" (Environmental Sciences, Canada, Over 65) Unsupported claims: Poor conclusions drawn New research outputs (preprints, data) Reduced focus on novel/ high-quality research Predatory journals "There is published Research biased by financial or other Support to the authors and not properly declared." (Medicine and Allied Health, Switzerland, 56 to 65) Growth in researchers Biases (in peer review, funding, negative findings not published) Errors: inflated statistical power/ grammatical/calculations New channels: (social media, media outlets) Methodological flaws "Authors often do not provide data/code/tools/proper description of the scenarios used for the evaluation contained in their papers. In particular, the correctness of code used for simulations reported in papers is often unverifiable." (Computer Sciences / IT, Brazil, 26 to 35) Open Science Lack of supplementary material

14. TRUSTWORTHINESS OF RESEARCH PROFILES: Most researchers (75%) trust at least the majority of research outputs they encounter, however 25% of researchers doubt the quality of at least some of the research profiles they encounter. To compensate for this they use multiple platforms or visit websites of institutions. Now thinking about the various research-related profiles that you interacted with (or encountered) last week (including researcher and institutional profiles), what proportion of the profiles would you consider trustworthy Which of the following mechanisms do you employ to compensate for any lack of confidence you have in the content you are considering reading/accessing? 25% 75% Base: All respondents that do not think all research –related profiled are trustworthy (n=1825) Base: All respondents that encountered research-related profiles in the last week (n=2588)

15. 15 Reasons research-related profiles are regarded as untrustworthy: among those that question the quality of at least some research output; the biggest problems are exaggeration of achievements, lack of verification and poor quality outputs You indicated that some/all of the research-related profiles (researcher and institutional profiles) are trustworthy. Why do you think some/all of the profiles you encounter are not trustworthy? "Both researchers and institutions tend to exaggerate their achievements." (Chemistry, Poland, 46 to 55) Exaggeration (self promotion, falsification) “The low quality of papers published" (Medicine and Allied Health, Indonesia, Over 65) Quality of research (reputation of associated journals, lack of metrics) "If I am contacted by someone with a profile disconnected from everybody, I do not trust it" (Engineering and Technology, Switzerland, 26 to 35) Unverified (references, endorsements) Out of date or poor profiles (errors, duplicate profiles, incomplete) "Many are updated infrequently. Others appear to be updated semi-automatically and contain errors." (Biological Sciences, United States of America, 36 to 45) Reputation (associated with the institution, author, country) Source of profile (Not reputable / open to edit) "I only trust top ranked outlets" (Earth and Planetary Sciences, Canada, Over 65) More comments from respondents can be found here

16. SOURCING TRUSTED CONTENT: being able to adjust algorithms/parameters, transparency and interoperability seen as most effective ways to be directed to trusted content Please indicate which of the below solutions you believe would be effective in ensuring you are directed to appropriate trusted content? Base: All respondents (n=3133)

17. COMMUNICATION TO THE PUBLIC: Explaining research in lay terms is seen as the best way to help people outside the research community judge the quality of research To help people outside the research community judge the quality of research articles they view, which of the following would be most helpful Base: All respondents (n=3133)

18. PUBLIC CONFIDENCE IN RESEARCH EVIDENCE: biggest problem seen to be misinterpretation or deliberate misrepresentation by the media as well as difficulty identifying high quality research. Thinking about public confidence in research evidence, how much, if at all, do you believe any of the following are a problem? Base: All respondents (n=3133)

19. Back to contents Attitudes to peer review Results by geographic region, country, broad subject area, age group and gender

20. SATISFACTION WITH PEER REVIEW: lowest in Western Europe and North America. Also lower than average in SSE/AH

21. ATTITUDES TO PEER REVIEW: SSE/AH least accepting of data and supporting materials being reviewed and least likely to think the time taken to review was reasonable

22. ATTITUDES TO PEER REVIEW: SSE/AH least likely to think peer review is well understood by researchers and the public

23. ATTITUDES TO PEER REVIEW: Only half of French researchers think the time taken for peer review is reasonable

24. ATTITUDES TO PEER REVIEW: only 14% of researchers in the UK and USA think peer review is well understood by the public

25. ATTITUDES TO PEER REVIEW: Western European researchers are least likely to believe peer review improves the quality of articles and think the time taken to review with last article was reasonable

26. ATTITUDES TO PEER REVIEW: researchers in North America, Western Europe and Latin America are least likely to think the public understand peer review well

27. ATTITUDES TO PEER REVIEW: Female researchers are least likely to want data and supporting materials to be reviewed

28. ATTITUDES TO PEER REVIEW: Female researchers and those aged 55+ least likely to think the public understand peer review

29. Back to contents Who researchers think should undertake peer review Results by geographic region, country, broad subject area, age group and gender

30. WHO RESEARCHERS THINK SHOULD UNDERTAKE PEER REVIEW: Earth/environmental science researchers more likely to consider post-publication commentary/ratings as peer review Which of the below qualifies as peer review; does it qualify when the evaluation of a research article is undertaken by:

31. WHO RESEARCHERS THINK SHOULD UNDERTAKE PEER REVIEW: SSE/AH more likely to expect peer review to involve review by a researcher or more than one editor before publication, but less likely to accept post-publication commentary as peer review. 20% of materials scientists think review by one editor is acceptable Which of the below qualifies as peer review; does it qualify when the evaluation of a research article is undertaken by::

32. WHO RESEARCHERS THINK SHOULD UNDERTAKE PEER REVIEW: Researchers in China less likely to expect peer review to be undertaken by at least one relevant researcher, but more likely to accept post-publication commentary/ratings as a valid form of peer review; the reverse view is found in the USA Which of the below qualifies as peer review; does it qualify when the evaluation of a research article is undertaken by:

33. WHO RESEARCHERS THINK SHOULD UNDERTAKE PEER REVIEW: Researchers in India most likely to accept post-publication commentary, review by a single editor and AI as forms of peer review Which of the below qualifies as peer review; does it qualify when the evaluation of a research article is undertaken by:

34. WHO RESEARCHERS THINK SHOULD UNDERTAKE PEER REVIEW: 37% of North American researchers consider review by 2+ editorial board members an acceptable form of peer review Which of the below qualifies as peer review; does it qualify when the evaluation of a research article is undertaken by:

35. WHO RESEARCHERS THINK SHOULD UNDERTAKE PEER REVIEW: Researchers from Africa are less likely to consider post-publication commentary as a valid form of peer review Which of the below qualifies as peer review; does it qualify when the evaluation of a research article is undertaken by:

36. WHO RESEARCHERS THINK SHOULD UNDERTAKE PEER REVIEW: Early career researchers slightly more likely to accept post-publication commentary and AI/ML as valid forms of peer review than older researchers Which of the below qualifies as peer review; does it qualify when the evaluation of a research article is undertaken by:

37. Back to contents How to maintain a healthy peer review system Results by geographic region, country, broad subject area, age group and gender

38. HOW TO MAINTAIN PEER REVIEW SYSTEM: Clear guidance for reviewers is considered the most important way to maintain peer review system across all subject areas. 38% of computer scientists want financial rewards for reviewers Which of the below do you consider most important for maintaining a healthy system of peer review?

39. HOW TO MAINTAIN PEER REVIEW SYSTEM: Clear guidance for reviewers is considered the most important way to maintain peer review system across all subject areas. Which of the below do you consider most important for maintaining a healthy system of peer review?

40. HOW TO MAINTAIN PEER REVIEW SYSTEM: Clear guidance for reviewers is considered the most important way to maintain peer review system across all countries. 38% of researchers in Russia want financial rewards for reviewers Which of the below do you consider most important for maintaining a healthy system of peer review?

41. HOW TO MAINTAIN PEER REVIEW SYSTEM: Clear guidance for reviewers is considered the most important way to maintain peer review system across all countries. 72% of researchers in the UK employer recognition. Which of the below do you consider most important for maintaining a healthy system of peer review?

42. HOW TO MAINTAIN PEER REVIEW SYSTEM: Researchers in North America and Western Europe more likely than average to want employer recognition; researchers in APAC least likely to want this Which of the below do you consider most important for maintaining a healthy system of peer review?

43. HOW TO MAINTAIN PEER REVIEW SYSTEM: 41% of researchers from the Middle East want financial rewards. Over half of researchers from Africa want formal training/quality benchmark and/or accreditation (e.g. CPD points) Which of the below do you consider most important for maintaining a healthy system of peer review?

44. HOW TO MAINTAIN PEER REVIEW SYSTEM: Early career researchers are slightly more likely to want financial rewards for reviewers (32% vs. 28% overall). 43% of females want greater diversity within peer review community (36% of males want greater diversity) Which of the below do you consider most important for maintaining a healthy system of peer review?

45. Back to contents ADDITIONAL INFORMATION FOR ASSESSING RESEARCH OUTPUTS Results by geographic region, country, broad subject area, age group and gender

46. ASSESSING RESEARCH OUTPUTS: 90% of chemists and life scientists want to see a reproducibility indicator When a reader is interacting with research outputs (e.g. research articles, preprints, data), what additional information is helpful for assessing those research outputs? % helpful

47. ASSESSING RESEARCH OUTPUTS: 57% Materials Scientists think news articles in popular press would be helpful When a reader is interacting with research outputs (e.g. research articles, preprints, data), what additional information is helpful for assessing those research outputs? % helpful

48. ASSESSING RESEARCH OUTPUTS: German researchers more likely to want signifier of reproducibility; researchers in USA more likely to want a clear signifier of content When a reader is interacting with research outputs (e.g. research articles, preprints, data), what additional information is helpful for assessing those research outputs? % helpful

49. ASSESSING RESEARCH OUTPUTS: Researchers in India more likely to find all indicators helpful; researchers in the UK least likely to find indicators useful (except for post-publication commentary and content signifier) When a reader is interacting with research outputs (e.g. research articles, preprints, data), what additional information is helpful for assessing those research outputs? % helpful

50. ASSESSING RESEARCH OUTPUTS: North American researchers are more likely to find signifiers to show content useful, while APAC researchers are more likely to find downloads, peer review ratings, number of collaborators and retweets/blog mentions useful When a reader is interacting with research outputs (e.g. research articles, preprints, data), what additional information is helpful for assessing those research outputs? % helpful