Professional Development

Sponsored by OHSU Department of Public Health & Preventive Medicine, School of Nursing, & the Institute on Disability & Rehabilitation Professional Development Review & Critical Appraisal Skills for Peer-Reviewed Journals Aka “reviewing & responding to reviews” Presented by: Elena M. Andresen, PhD Chief, Disability & Health Research Group Institute on Disability & Development Oregon Health & Science University CDRC room 1272A Office: 503-494-2275 email: andresee@ohsu.edu

My Background, Limits, & Disclaimers • Professional Service • Editorial Boards J Gerontology; Dis Health J • Associate Editor Qual Life Res (Springer). Cumulative portfolio ~115 manuscripts • Frequent manuscript reviews: medical/clinical, public health journals (~ 8-15 manuscripts/year) • Frequent U.S. grant reviews: ad hoc & standing panels • Research & Scholarship • Author/Coauthor of 100+ peer-review publications • PI/Investigator 40+ national, state, international, & foundation grants & contracts

Background, Limits, & Disclaimers • Training: • No formal training/advice on reviewing for journals • No training on responding to journal reviewers • Over time, I observed expert reviews/reviewers,& found some nicely constructed papers to help me personally, & to share. • Most importantly: colleagues & collaborators provide exceptional role-modeling, constructive criticism, & encouragement

Your Experiences: Have you… • Coauthored a journal submission? • Been a lead (corresponding) author for a journal submission? • Performed manuscript reviewer for a peer-review journal? • Received harsh, inadequate, or highly useful feedback from reviewers, readers, instructors, or collaborators? Your experiences are valuable & add to the perspectives in this presentation.

Expectations Today • Content: Lecture slides, three pdf review examples (de-identifed) • Suggested readings:Many possible other resources & I have a few suggestions. • Availability:Email me or make an appointment if I can be of specific help.

Successful “Academic” Careers * These 4 components vary by setting. Review of your qualifications for jobs, grants, & promotions may include: • Research (funding & experience) • Productive scholarship (peer-reviewed papers, presentations, other scholarship) • Service (national & international professional work in your discipline) • Other career components for some • Teaching • Clinical duties * In reality, these are part of most professional positions

And… There is a feedback loop • Teaching & training ideas are partly derived from your research experience & publications • Trainees may seek you out based on your peer-reviewed scholarship • Research funding depends, in part, on your peer-reviewed publication productivity • Invitations to professional service depend partly on your being “known” via peer-reviewed scholarship Scholarly peer-reviewed success stimulates success in the other activities. Professional service provides insight that feeds scholarship & research.

Part 1: Becoming a successful journal peer-reviewer

Peer Reviewing • Myth:If you review (for a specific journal) your manuscripts are more likely to be accepted • Most journals use management software systems where this information would not be available to reviewers, & editors may be unlikely to examine authors/reviewers service in their final decisions • Reviewers may be “blinded” to author identity (journals vary) • Not a Myth:Constructing reviews of good & bad manuscripts helps you write & submit

Peer Reviewing • Not a Myth:review quality & reviewer performance is rated in some journal management systems. This may affect your future service for the journal. • Example: for Quality Life Research & other journals, editors (who search for & invite reviewers) examine… • Reviewer’s average days to submit reviews • Average editorial rating of reviews’ quality • More descriptive issues; e.g., total invitations & completed reviews; declined, agreed, uninvited & terminated reviews.

Personal Peer Reviewing Benefits • Another good outcome of journal reviewing: learning to respond to reviews! (Part 2) • Suggested reading on peer review: Cummings & Rivara. Responding to reviewers comments on submitted articles. Arch PediatrAdolesc Med 2002; 156: 105-107. Cummings & Rivara. Reviewing manuscripts for [our journal]. Arch PediatrAdolesc Med 2002; 156: 11-13. Schroter et al. What errors do peer reviewers detect, & does training improve their ability to detect them? J R Soc Med 2008;101: 507-514. Schroter et al. Differences in review quality & recommendations for publication between peer reviewers suggested by authors or by editors. JAMA 2006; 295: 314-317. Committee on Publication Ethics (COPE) documents (see website http://www.publicationethics.org.uk/about)

Getting Started for Journal Peer Reviewing Get some practice with mentors Ask to work as a co-reviewer. Discuss your practice reviews, compare yours to theirs, etc. Ask if your mentors will share an especially helpful review they received. Volunteer if the opportunity arises Ask mentors & colleagues to recommend you & don’t let them down. Introduce yourself to editors (& associate editors) at meetings Look at favorite target journals for editor/board names.

Getting Started for Journal Peer Reviewing Read articles, expectations, & suggestions from journal editors Search your favorite journals for recent editorials about reviewing & expectations, & for good studies about review quality, etc. Be prompt & thorough when you are asked & you agree to review We will spend more time on this.

Selection of Reviewers Register at the journal (reviewer) website Complete name, title, institution(s) Expertise: Be comprehensive. Editors look for reviewers by keywords & categories Usually this is a list that you check off for your interests & discipline or experiences It would be rare that one of these sites has categories that fit you perfectly, so add text if you have the opportunity. However, the management systems probably uses only the check-box information when an editor is searching. The added text may be used after looking at a possible reviewer record in more detail.

Selection of Reviewers Register at the journal (reviewer) website Don’t add more categories of expertise than is reasonable – you might refuse more invitations this way, because you will be sent invitations that are far outside your comfort level. Refusals may be recorded in your reviewer record. Keep a record of journal reviewer information; e.g., your login name & password, & update your data as it changes when you move from graduate student status to a new institution, email address, etc. Track your reviews. Someone, usually your supervisory, will want a report

Selection of Reviewers Answer invitations promptly, & be helpful with your suggestions for other reviewers if you decline A quick refusal is nearly as good as a quick acceptance Each delay in a reviewer response can affect the overall time that a manuscript is in review. When you say no, or automatically when some number of days expire on your invitation, the system moves to the next listed (pending) reviewer & sends out a request (& notifies the managing editor). Manuscripts are delayed by a number of review factors: reviewer delays in responding to invitations are part of total review time (journal decisions to authors)

General Guidance Read the abstract to see broad issues of what you are asked to review Usually part of the initial invitation There are multiple reviewers (2-4), so you do not need to know everything about the manuscript to provide a good review You may provide more on methods than content / topic You may provide more clinical or basic science input then research methods Set a reasonable timeline & keep to it The sooner you start the better It takes the same time no matter when you do the review

General Guidance Read Editors’ guidelines for review & format Do you use a check list of attributes in addition to a text review? Go to the tab on “submit review” if it is your first time to see what they ask for. May provide orientation. Are you required to use a numbered format (I recommend you do so even if not required), & do you need to provide manuscript details like section/row numbers for your comments? Start a structured WORD document name it by journal & use their system manuscript number, e.g., QualLifeRes9977.doc Personally: I review on a paper copy & make notes directly on it before starting a WORD file

General Guidance Often you are asked for two review formats A private summary of your recommendations about publication & special issues. Then the full review for the authors, where you typically do not say if you recommend publication. It may help to construct 2-3 sections & then write in your content (draft comments) as you read, e.g. Mandatory changes (or major comments) Minor changes (or recommended comments) Editorial suggestions (please be generous!)

Specifically Say something nice! (The Golden Rule) Consider your role: provide editorial decision support, constructive criticism, & improve science quality. Provide an accurate study description, setting; highlight benefits of (design or importance of the question, etc.) Address major comments & questions typically a list: use complete sentences, specifics from the manuscript. Consider: Both strengths & weaknesses Originality Sample (appropriate to question) Study design Measures & analytic methods Theory Conclusions/discussion/interpretation

Specifically Minor & important Unclear vs. flawed aspects of the manuscript Literature & references (Thorough? Current?) Use of terms & language that match the journal Tables & figures (lack of clarity, e.g., titles, complete words, footnoted effectively, etc). Suggest other editorial issues for clarity, e.g., reduce jargon & abbreviations

Specifically As noted, some comments don’t go to authors; summarize additional issues only for the editor(s). Frank advice to the editor, or if your review & advice conflict (Review: pleasant & constructive. Advice: reject) Example: a naïve manuscript, that didn’t follow scientific construction or format. The topic & sample (& apparently available data) would strengthen our field. All the details went to authors. I told the editor why I had not rejected it. Example: a reviewer did not find many faults with general analysis & construction of the manuscript, but noted to the authors that there were few new elements in the manuscript. In the comments to the Editor, s/he noted that the manuscript was quite duplicative of other (un cited) studies.

Next Set the review aside briefly (a day) Edit the review for clarity & writing Be as clear with your review as the authors should be with their manuscript Re-read the manuscript & make final edits Send the review promptly & confirm that it was registered Sometimes electronic journal tracking sites are imperfect, & sometimes reviewers are imperfect Typically you receive an email thanks, but you can check the journal review website to see if you still have a review “pending”

Examples (anonymized) Reviews Link to document with manuscript reviews for three different journals: “example manuscript reviews.pdf” These are my reviews with different review formats. Each has been edited to remove aspects that might identify the study, locations, etc., so forgive the multiple places where I insert “letters” replacing details like conditions, measures, etc.

Part 1 Reflections & Discussion Angola: young man with polio, WHO. http://www.who.int/features/2005/disability/en/ Courtesy of the Colorado Home Initiative & the Florida Office on Disability & Health http://fodh.phhp.ufl.edu/

Think “Oregon social & political styles.” Consider the process, & reviewers, as important diverse players in your scholarly environment. As in Oregon, you don’t always agree with what others say & do around you, but you are open to supporting a “better Oregon” & their right to express themselves. Part 2: Responding to journal reviews of your manuscripts

Responding to reviewers: part of methods of successful publication • Arguments with reviewers may be futile • Feel free to disagree, nicely, describing why • Use an appreciative, thoughtful style • “We thank the reviewer for catching this misplaced decimal…” • “Editing recommendations result in clearer language …” • “The Reviewer asked an important question prompting ad hoc analyses… although the results did not change our conclusions, the process provided an important addition to the discussion section…” • “The analysis suggested by the Reviewer is an interesting variation of our choice. We considered this with our statistical advisor, & feel that the use of non-parametric methods better suites our non-normal data & sample size & now reference this in…”

Responding to reviewers • Be thorough – summarize each reviewer comment & describe what you did & where • ‘There are no page limits to response letters’ • Cummings & Rivara 2002 • I once sent a 9-page response with new tables & analyses (not added to the revised paper) suggested by reviewers • I thanked them for suggestions prompting a more thorough discussion section & added the ad hoc analyses to methods, & to results & interpretation • If you do not have room, for page limits, add a comment in the revisions (data available from authors) or similar.

Learning from Journal Peer Reviewing Experience • These somewhat arbitrary suggestions are based on my experience & colleagues who are journal editors • Reviewing others’ work helps you to be a productive (successful) scholar • A few tips & inside experiences follow. • As a reviewer & editor, I often make similar suggestions to authors

Successfully Communicating to Reviewers: Tables –drop dead easy • Tables should “stand alone” (from colleague Fred Wolinsky@Iowa & APA criteria) • E.g. use variables in columns that someone can understand • E.g., use a full informative title with data set, topic, sample, etc. • E.g., label columns, with numbers where needed. • Be clear about data, %’s, means, variances, etc. • Use leading zeros (0.06) • Use informative footnotes liberally • Avoid abbreviations if not absolutely needed for brevity (use consistent shorter terms, e.g. not a full question or clinical description: use an informative full word)

Table Example (uninformative) Table 2: Risk factors for falls * p<.05

Table 2: Risk factors for incident falls among 888 African Americans aged 50-64 during two years of follow-up + Weighted odds ratio from binary logistic regression adjusted by all other model variables * p ≤ 0.05 ** p ≤0.01 *** p ≤0.001 (ns= not statistically significant p > 0.05) # Measured as ≥ 9 points, Center for Epidemiologic Studies Depression Scale (CESD) § 6 items summed (difficulty walk ¼ mile; walk up & down 10 steps; stand 2 hours; stooping; lift 10 lbs; push large objects)

Personal Editor Experience • Somewhat of a myth: Your suggestions for good reviewers will be the basis for selecting & inviting reviewers of your paper • Editors may assume you select colleagues even if they are not currently at your institution • An editor told me he took a random sample of submitted papers & tracked web data on reviewer/authors relationships (he said there were lots). He used author suggestions to build the reviewer database, sending them invitations & asking them to add details about themselves at the journal reviewer website (but rarely used them for the submitted manuscript).

Final Thoughts & Suggestions • Your science is always your first concern • Consider “packaging” as a task you perform to maximize the outcome of your science • Consider your overall career goals early, & don’t apologize for considering these as you prepare, submit, & resubmit your work • Don’t go against your core values & make decision that feel like you are “selling out” • But push your comfort limits a bit as you develop your career, & don’t forget how important it is to communicate your work effectively!

Final Thoughts & Suggestions • Use both successful journal experience & failures to inform future decisions, not to direct your decisions • knowledge is power • Use coauthors & colleagues for information & feedback for submissions & resubmissions • Consider criticisms in light of “style” vs. “content.” Style is the prerogative of the first author & the journal • Develop patience & perseverance • Keep repackaging good science if you do not first succeed! • It is common to select an inappropriate first target journal – consider target journals carefully

Thank youView from the Oregon Health & Science University

Professional Development