Archived File

1. Archived File The file below has been archived for historical reference purposes only. The content and links are no longer maintained and may be outdated. See the OER Public Archive Home Page for more details about archived files.

2. Evaluation of Peer Review Pilots Dr. Andrea Kopstein Director, Office of Planning, Analysis & Evaluation, CSR June 8, 2009 National Institutes of HealthU.S. Department of Health and HumanServices

3. Sample of Pilots Evaluated • Two Stage Review • Prebuttal • 1-9 Scoring

4. Two Stage Review Expected Benefits: • Provide specific expertise for a wide range of scientific areas • Enable better quality discussion by using a smaller number of stage two reviewers • Increase consistency for scoring • Emphasize overall significance and impact while preserving the dynamic of reviewer discussion • Expand the potential reviewer base. Mail reviewer stage • Two stages can simplify management of dyads and conflicts.

5. Types of Applications in Two Stage Review Pilot • Bioengineering Research Partnership (BRP). • Small Business Innovation Research (SBIR). • ~70% response rate among both Stage 1 and Stage 2 Reviewers

6. Key Evaluation Findings • Willingness to serve as a stage one and/or a stage two reviewer in the future: • Majority were willing to participate at either level of review. • 82.4%Stage One Reviewers willing to be either a Stage One or a Stage Two Reviewer. • 63.9% Stage Two Reviewers willing to be either a Stage One or Stage Two Reviewer. • What format reviewers would chose for the review of their own applications: • 63.9% of Stage Two reviewers put two-stage review as their first choice and about half of Stage One Reviewers ranked two-stage review as their first choice format of choice for their own applications

7. Key Evaluation Findings • Question regarding having more assigned reviewers per application: • Majority felt this review format had a positive effect on the availability of scientific and technical expertise (70.6% of Stage One and 72.2% of Stage Two reviewers). • Did Stage One Reviewer critiques inform Stage Two review of applications: • Over 2/3rds of Stage Two Reviewers found the Stage 1 critiques to be very helpful. • 63.9% Stage Two Reviewers thought this review format allowed them to spend less time reviewing and preparing comments.

8. Key Evaluation Findings • 84.7% Stage One and 69.4% Stage Two reviewers responded that two-stage review format would be appropriate for all grant application mechanisms.

9. Editorial Board Reviews: Status • SBIR • BRP • T-R01 • Challenge Grants

10. SBIR Prebuttal Pilot • Meeting 1: • 33 of 47 applications prebuttal submitted (70%) and the prebuttal made a difference in only 2 of the 33 (6%). • Meeting 2: • 14 of 18 applications prebuttal submitted (78%) and the prebuttal made a difference for none of the 14 (0%). Only 5 of 14 prebuttals addressed a factual error (36%). • Meeting 3: • 17 of 19 applications prebuttal submitted (89%) and the prebuttal made a difference for one of the 17 (6%). Zero of 17 prebuttals addressed a factual error ( 0%). • Meeting 4: • 57 of 71 applicants (80%) submitted a prebuttal; only 3 addressed factual errors (5%), with errors all being minor rather than substantive. “The remainder essentially submitted a “mini” Introduction, as if a resubmitted application.”

11. SRO Prebuttal Evaluation • 200 words is too short for a prebuttal: • forces the applicant to use shorthand and the prebuttal becomes less coherent • Prebuttal helps good applications: • not badly written applications • For meetings with large numbers of applications, the prebuttal concept is too much work -“unworkable.” • The prebuttal concept has “public relations (PR) “ value. • Applicants make additional promises in their prebuttal submissions and although the SRO told reviewers not to let the additional promises affect scoring, some reviewers may have been affected. • The new “bulleted” critiques should result in fewer errors.

12. Reviewer Prebuttal Evaluation • Reviewers who wrote open ended responses were strongly in favor of having this option. • Prebuttal option “is a major improvement to the review process.” • Stage 2 Reviewers expressed need for a longer prebuttal. • Some remarked that the 200 word limit favored applicants with only one concern. • Prebuttal should be up to a single page in length. • Reviewers were concerned about the time allocated for applicants who wanted to submit a prebuttal. • They are in favor of allowing more than the 3 days (including the weekend) allowed in this pilot for submission of the prebuttal.

13. How did having a prebuttal opportunity alter the quality of peer review?

14. Survey Summary of 1-9 Scoring System Pilot • 33 Reviewers participated in 2 meetings that piloted 1-9 scoring. • 22 completed the survey for a 67% response rate.

15. Key Evaluation Findings • Virtually every question directly pertaining to the 1-9 scoring system revealed positive responses. • 1-9 scoring system should be useful to applicants for interpreting the written comments they receive. • The new 1-9 scoring system was not deemed to be an additional burden when compared to the old 1-5 scoring system. • 100% of respondents indicated they were either “Very Satisfied” (68%) or “Somewhat satisfied” (32%) with the 1-9 scoring system.

16. How well were you able to communicate the differences in impact or merit of the applications you reviewed using the new 1-9 scoring system?

17. How helpful were the verbal descriptors (Exceptional to Poor) in your determination of the initial and final scores?

18. What effect do you feel the scoring of the specific criteria had on helping you determine the overall impact (merit) of the applications reviewed?

19. What is your overall level of satisfaction with a peer review process that includes the 1-9 scoring scale?

20. WHAT’S NEXT • TR01 • NIH External and Internal Constituency Surveys: • Applicants • Reviewers • SROs • PO • Etc.