The review process ( and responding to reviewer feedback )

1. The review process (and responding to reviewer feedback) Rick Gibbons June 20, 2014

2. Center for scientific review (CSR)vs. NIH

4. Reviewing @ CSR • Initial Review Group (aka “Study Section;” SRG) • 1. People • SRO • Chair (voting member) • Program Official* (with Institute) • Assigned Reviewers (primary, secondary, tertiary) • Voting Members (24 - ~42) • *maybe

5. Review Process @ CSR • 2. Process • New / early investigators grouped • Initial scores presented by assigned reviewers • Reviews presented in order • Discussion • Final vote by ARs • Others vote as well – stay within parameters (hi/lo score); if not, why

6. FEEDBACK FROM CSR • 3. Feedback • Overall summary from each reviewer • Criterion scores • Discussed? • Impact score • Percentile • Impact score vs. percentile

8. Scores

9. Criterion Scores

10. Weighting for each criterion

11. Paylines 19922013 NIDA Success 40% 21% Payline (%ile) 28 8 NCI Success 33% 20% Payline 22 9 “Zone of Consideration”: 10 – 20%ile

12. Success: Experienced

13. Success: new

14. Should I Respond? • Yes. • 70% likelihood of better score • In 2010: 3 times as likely to get funded • 11.5% success rate for initial vs. 35% for resubmission • How? • Appeal? • … unlikely to succeed (and if it does, then back to study section) • Take a look at a different study section; • 25% of funding decisions would have been different with a different panel (Johnson, 2008).

15. Resubmissions OLDNEW A0 A0 A1 (w/intro) A1 A2 A0 (no intro) A1 (etc.)

16. Ratings • Significance: • Investigator(s): • Innovation: • Approach: • Environment: Reviewer 1 • 5 • 1 • 6 • 6 • 1 Reviewer 3 • 1 • 1 • 1 • 2 • 1

17. Resume and Summary of Discussion In this very strong application, the investigators propose collecting a seventh wave of data from participants in the Family and Community Health Study (FACHS). This study will expand the original study in several important ways to examine how early stress experiences and cumulative stress predict continued substance use and risky health behaviors for African American young adults through their late 20s. The investigators have been mostly responsive to previous critiques. There was a divergence of opinion among reviewers about the potential overall impact of this study.

18. RESUME AND Summary of discussion (cont.) While most reviewers indicate the application is likely to have an extraordinarily high impact on the field and larger social world, a minority of reviewers were unsure about the impact due to limitations of the methodology. The strengths of the application include the broad significance of the work, outstanding and productive research team, use of high quality longitudinal data with excellent retention, addition of biomarker data and biological indicators of substance use, and the addition of community level data and a White comparison sample. The application is weakened by some limitations in the approach and an ambitious and underdeveloped analysis plan. Overall, the strengths of the application outweigh the weaknesses.

19. Critique 1: Overall Impact This is important work, and the team is well positioned to conduct it. In the resubmission, the investigators manifest a serious commitment to addressing prior weaknesses. They clearly thought through their options, and arrived at a reasonable solution to the absence of data on White people and the addition of team members with quantitative expertise. These are laudable changes. The resubmission, however, fails to provide a persuasive argument on why Wave 7 is necessary. Questions remain over the comparability and viability of the White sample. Some issues raised in the previous review have gone unaddressed. Too much material is devoted to early sections of the application at the expense of later, more critical ones. The absence of clearly delineated changes to the resubmission makes finding the improvements difficult.

20. Critique 3: Overall Impact In this application, the investigators propose to conduct a seventh wave of data in the Family and Community Health Study (FACHS). The ongoing study has followed samples of African American children in Iowa and Georgia starting at age 10 onward. The initial sample represented nearly 900 families. To date, the project has examined an array of factors predicting health risk behaviors (e.g., substance use) and health outcomes as well as social cognitive predictors of these (particularly perceived racial discrimination), not only in the target children but their parents and siblings. The study has documented the cross-over effect, indication that African Americans begin to use substances later than do Whites, but they also are less likely to mature out of substance use. This next wave of data collection would allow the investigators to test this cross-over effect in the best way possible: with data that track the participants into early adulthood. Moreover, the proposed work expands the original study in 6 important ways, not only simply moving forward with the initial study goals (which themselves remain important) but also in expanding to include 1) a White comparison sample, 2) biomarker data for genetic analyses, 3) biological indicators of substance use, and 4)additional community level data relevant to the outcomes of interest. The proposed data will also account for age-related changes in the lives of the target participants, incorporating 5) romantic partner reports and 6) data about the target children in parental roles as many of the original targets are now parents.

21. Critique 3: Overall Impact (cont.) The project is likely to have an extraordinarily high impact on the field as well as the larger social world. Understanding the effects of perceived racial discrimination on health and wellness in minority groups are an important policy problem and the investigators have taken pains to tie the science here to the real world context of their participants. The proposed work is immensely expansive in its scope and promises to provide the kind of evidence necessary to understand the problems addressed as well as to guide interventions. The major strengths of the application include the ambitious research plan, the expertise of the collection of scholars who will carry it out, he proven track record of the research team, and the very high broad significance of the work. The key weaknesses are best viewed as imperfections or limitations. Certainly, although similar to the original FACHS samples in some ways, the Oregon sample is likely to differ in ways beyond race/ethnicity. It is clear that the investigators are aware of these issues and will make the best use of these data. In addition, some aspects of the application remain a bit underdeveloped, such as the integration of statistical procedures that account for interdependence in the dataset. Importantly, the team that is in place suggests that these limitations will be addressed by, perhaps, the very best scholars in the field.

23. 1. Significance • Strengths: • Excellent progress report; very productive group. • Questions about health disparities are sorely in need of better data. This application promises to continue to generate those data. • Weaknesses: • Questions about what an additional wave of data collection will bring remaining unpersuasively answered. 2. Investigator(s) • Strengths: • Outstanding team. • Very productive in executing this study thus far. • Investigators are uniquely suited to continue the research they have been doing over the past several years. • Adding Dr. Kenny to the team gives added strength. • Weaknesses: • None noted.

24. 3. Approach • Strengths: • The resubmission represents an earnest attempt to address weaknesses in the prior submission. • More detail on measures and data analysis is helpful. • Weaknesses: • The absence of delineations for text revisions is regrettable. Locating the many improvements to the application was a challenge. • Heavy use of italicized words and phrases further inhibited the task of discerning changes. Though some resubmitted applications use italics to indicate changes, apparently this application employs italics to underscore particularly salient points. Excessive italicization, however, may have served to obscure, rather than to clarify what is special about the planned work. • The potential lack of comparability between the FACHS and OYSUP samples is worrisome. That both studies are up for renewed funding is another concern. If OYSUP is not awarded, how does this affect FACHS? • Timeline is too sketchy. This is particularly troublesome owing to the absence of a timeline being cited in the last review. • Too much space is taken up with background material and the progress report, at the expense of procedure. The absence of detail on procedure was cited in the prior review. • In the prior review, issues about heterogeneity and interdependence were noted. These are insufficiently addressed in the resubmission.

25. 4. Innovation • Strengths: • The addition of more detail on innovation demonstrated the team’s intent to respond to the prior review. • Weaknesses: • Notwithstanding arguments put forth in the applications, innovations seem minor. 5. Environment • Strengths: • Excellent settings. • Weaknesses: • None noted.

26. 1. Significance • Strengths: • Understanding racial differences in substance use and abuse is an extremely important scientific and social goal. • Understanding how perceived racial discrimination influence health risk behavior is an important goal for social science and policy makers. • The breadth of the data collected, including self and other ratings as well as biological measures of substance use and genetic markers, suggests that this work will influence a broad range of fields. • The inclusion of this next wave of data collection will allow for a true test of the processes identified thus far in the earlier waves. Such data are needed to fully understand the cross-over effect. • The capacity to follow the original participants into their young adult lives, as romantic partners and parents surely enhances the significance of the work. • Weaknesses: • The inclusion of the White (Oregon) sample without very specific plans to deal with the differences that likely exist between this sample and the FACHS samples.

27. 2. Investigator(s) • Strengths: • The team itself is a clear strength of the application. Principal Investigators Gibbons and Gerrard are productive scholars who have made important contributions to the scholarly literature of direct relevance to the proposed project. • A large number of Co-Investigators and consultants with direct expertise specific to the FACHS data and/or the specific complex issues included in this wave of data collection, including HIV prevention (Fisher), genetic epidemiological research and GXE effects (Todorov), tobacco use and prevention (Wills), stress, discrimination and genes and biological processes in African American samples (Brody, Cutrona) analyzing biomaterials (Philibert), and the complex quantitative task that is implied in these rich data (Todorov, Kenny). • The inclusion of Co-Investigators Andrews and Hampson from ORI to collaborate on the integration of the Oregon dataset with the FACHS samples. • Weaknesses: • None noted.

28. 3. Innovation • Strengths: • The FACHS is innovative in that it focused initially on Black participants who did not reside solely in urban environments. • The inclusion of biological as well as psychological measures. • Adding community level variables. • Applying more sophisticated quantitative analytical tools to these complex data. • Including genetic markers to fully contextualize the psychosocial variables and processes. • Adding bio-measures of substance use. • Examining, now, multigenerational processes. • Incorporating a White comparison sample. • Weaknesses: • None noted.

29. 4. Approach • Strengths: • Relatively large samples, studied longitudinally. • The very rich data already collected about these individuals. • The fact that, on the variables of interest, these is indeed enough variability to fully explore the central aims of the project. • The addition of a White/comparison sample. • The ambitious range of data to be collected about these individuals, including not only self-reports and other (i.e. parent, romantic partner, sibling) reports but also, now, genetic measures, bio-measures for substance use, and additional measures pertaining to the community level aspects of the participants’ lives. • Weaknesses: • Although the inclusion of the Oregon sample is an innovative step, it is possible that the differences between the samples (beyond, race/ethnicity) will be difficult to surmount. • This is clearly a massive undertaking and it is possible that the plan is over ambitious.

30. 5. Environment • Strengths: • Adequate facilities and resources are available for this research. • Weaknesses: • None noted.