Peer reviewer training part I: What do we know about peer review?

1. Peer reviewer trainingpart I:What do we know about peer review? Dr Trish Groves Deputy editor, BMJ

2. What do editors want from papers? • Importance • Originality • Relevance to readers • Usefulness to readers and, ultimately, to patients • Truth • Excitement/ “wow” factor • Clear and engaging writing

3. Peer review • As many processes as journals or grant giving bodies • No operational definition--usually implies “external review” • Largely unstudied till 1990s • Benefits through improving what’s published rather than sorting wheat from chaff

4. What is peer review? • Review by peers • Includes: internal review (by editorial staff) external review (by experts in the field)

5. BMJ papers • All manuscripts handled by our online editorial office at http://submit.bmj.com • The website uses a system called Benchpress • Reviewers recruited by invitation, through volunteering, and by authors’ suggestions • Database also includes all authors • We monitor reviewers’ workload for BMJ • We rate reviewers’ reports using a 3 point scale

6. BMJ peer review process I • 7000 research papers, 7% accepted • approximate numbers at each stage: • 1000 rejected by one editor within 48 hours • further 3000 rejected with second editor • within one week of submission 3000 read by senior editor; further 1500 rejected • 1500 sent to two reviewers; then 500 more rejected • approx 1000 screened by clinical epidemiology editor and more rejected

7. BMJ peer review process II • 400-500 to weekly manuscript meeting attended by the Editor, an external editorial adviser (a specialist or primary care doctor) and a statistician.. • …and the full team of BMJ research editors, plus the BMJ clinical epidemiology editor • 350 research articles accepted, usually after revision • value added by commissioned editorials and commentaries

8. BMJ peer review process III • always willing to consider first appeals--but must revise the paper, respond to criticisms, not just say subject’s important • perhaps 20% accepted on appeal • no second appeals; always ends in tears; plenty of other journals

9. What we know aboutpeer review Research evidence

10. Peer review processes • “Stand at the top of the stairs with a pile of papers and throw them down the stairs. Those that reach the bottom are published.” • “Sort the papers into two piles: those to be published and those to be rejected. Then swap them over.”

11. Some problems • Means different things at different journals • Slow • Expensive • Subjective • Biased • Open to abuse • Poor at detecting errors • Almost useless at detecting fraud

12. Is peer review reliable?(How often do two reviewers agree?) NEJM (Ingelfinger F 1974) • Rates of agreement only “moderately better than chance” (Kappa = 0.26) • Agreement greater for rejection than acceptance Grant review • Cole et al, 1981 – real vs sham panel, agreed on 75% of decisions • Hodgson C, 1997 – two real panels reviewing the same grants, 73% agreement Are two reviewers enough? • Fletcher and Fletcher 1999 - need at least six reviewers, all favouring rejection or acceptance, to yield a stats significant conclusion (p<0.05)

13. Should we mind if reviewers don’t agree? • Very high reliability might mean that all reviewers think the same • Reviewers may be chosen for differing positions or areas of expertise • Peer review decisions are like diagnostic tests: false positives and false negatives are inevitable (Kassirer and Campion, 1994) • Larger journals ask reviewers to advise on publication, not to decide

14. Bias Author-related • Prestige (author/institution) • Gender • Where they live and work Paper-related • Positive results • English language

15. Prestigious institution bias Peters and Ceci, 1982 Resubmitted 12 altered articles to psychology journals that had already published them Changed: • title/abstract/introduction - only slightly • authors’ names • name of institution, from prestigious to unknown fictitious name (eg. “Tri-Valley Center for Human Potential”)

16. Peters and Ceci - results • Three articles recognised as resubmissions • One accepted • Eight rejected (all because of poor study design, inadequate statistical analysis, or poor quality: none on grounds of lack of originality)

17. How easy is it to hide authors’ identity? • Not easy • In RCTs of blinded peer review, reviewers correctly identified author or institution in 24-50% of cases

18. Reviewers identified (open review) – results of RCTs Asking reviewers to sign their reports in RCTs made no difference to the quality of reviews or recommendations made • Godlee et al, 1998 • van Rooyen et al, 1998 • van Rooyen et al ,1999

19. Open review on the web Various experiments and evaluations are underway…

20. What makes a good reviewer? – results of RCTs • Aged under 40 • Good institution • Methodological training (statistics & epidemiology)

21. What might improve the quality of reviews? • Reward/credit/acknowledgement? • Careful selection? • Training? • Greater accountability (open review on web)? • Interaction between author and reviewer (real time open review)?