Common Pitfalls in Research Proposals and Suggestions on How to Avoid Them Aloen L. Townsend, Ph.D. Research Methodo

1. �Common Pitfalls in Research Proposalsand Suggestions on How to Avoid Them�Aloen L. Townsend, Ph.D.Research Methodology ColloquiumMandel School of Applied Social SciencesMarch 31, 2010

2. This presentation is intended to help you write stronger research grant applications So, feel free to ask questions or add comments at any time

3. NIH Review Experiences I�ll Draw On Social Psychology, Personality, and Interpersonal Processes (SPIP) Study Section; Risk, Prevention and Health Behavior Integrated Review Group, National Institutes of Health (most recently March 2010) Special Emphasis Panel/Scientific Review Group 2009/10 ZRG1 HDM-A (58) R [challenge grants], National Institutes of Health Special Emphasis Panel on Predoctoral Fellowships for Minorities and Persons with Disabilities (RPHB3-02), National Institutes of Health Mental Disorders of Aging Review Committee, National Institute of Mental Health Life Course and Prevention Research Review Committee, Subcommittee on Aging, National Institute for Mental Health

4. Foundation Grant Review Experience I�ll Draw On Research Grant Program, Alzheimer's Association Research Grant Program, The Retirement Research Foundation Alzheimer's and Related Diseases Research Award Fund, Commonwealth of Virginia National Review Committee, Enhancing Personal Autonomy of Elderly Individuals in Long Term Care Initiative (Phase II), The Retirement Research Foundation

5. Some Other Grant Review Experiences Institute on Aging and Social Work (funded by NIA, OBSSR, and the John A. Hartford Foundation) Intramural Faculty Research Award Program, School of Social Work, University of Minnesota Pilot Grant Program, University Memory and Aging Center Pilot Grant Program, CWRU Review Committee, Cancer Survivorship Research Initiative, Case Comprehensive Cancer Center ADVANCE Opportunity Grant Review Committee, Academic Careers in Engineering and Science (ACES) Program, CWRU

6. Today I�ll focus on the most common mistakes that come up in the grant applications I�ve reviewed and suggest some tips for avoiding them Overall Impact Specific Aims Significance Innovation Conceptual Framework(s) Design Measures Analysis Plan Timeline Depending on time and audience interest: Investigators, Environment, Budget, Overall Organization and Format

7. Overall Impact and Significance February 2010 Extramural Nexus handout Two critical challenges for successful NIH application Overall impact is a rating that reviewers give, not a section of the application (but your application needs to build this case) �Takes into consideration, but is distinct from, core review criteria� Preliminary overall impact scores are used in some Study Review Groups (SRGs) to organize the study section review You need to be clear and explicit about your proposed study�s potential impact and significance

8. Common Pitfalls Related to Overall Impact Different reviewers weight different criteria differently Different mechanisms require different considerations for judging overall impact, so understand your mechanism R03 more emphasis on conceptual framework and general approach R21 more emphasis on conceptual framework, level of innovation, and potential to significantly advance knowledge or understanding Reviewers doubt the project�s ability to successfully achieve its aims Additional review criteria (e.g., Protections for Human Subjects) raise serious concerns

9. Common Pitfalls Related to Overall Impact(continued) You don�t convince reviewers there will be �a likelihood of sustained, powerful influence on the research field(s) involved� Likelihood (i.e., probability), according to NIH, �is primarily derived from the investigator(s), approach and environment criteria� Sustained powerful influence, according to NIH, �is primarily derived from the significance and innovation criteria�

10. Specific Aims What do you intend to do? A critical part of the application; may be the only part that some reviewers read and is often the first part that assigned reviewers read (along with the abstract) Form an overall impression in the reviewers� mind, for better or worse One page Concisely states �the goals of the proposed research� and summarizes �the expected outcome(s), including the impact that the results of the proposed research will exert on the research fields involved.� Succinctly lists �the specific objectives of the research proposed�

11. Common Pitfalls Related to Specific Aims Not concise and succinct; oblique and takes too long to make its points Critical problem being addressed is not identified Long-range goal(s) not stated Too many aims, overly ambitious; not realistic Aims are not logically connected to each other; not cohesive Entire study rests or falls on the first aim Aims are not innovative Aims appear unlikely to be achievable Aims omit essential steps

12. Common Pitfalls Related to Specific Aims(continued) Vague hypotheses, not testable Aims not tailored to the funding mechanism Key constructs not defined Aims do not (clearly) fit with the proposed design Aims do not (clearly) fit with the conceptual framework Aims do not (clearly) fit the proposed target population Expected outcome(s) unclear Impact on the field not clear or not persuasive

13. Research Strategy Research Strategy includes* Significance Innovation Approach *12 pages for R01, 6 pages for R03, R21

14. Significance �Explain the importance of the problem or critical barrier to progress in the field that the proposed project addresses� �Explain how the proposed project will improve scientific knowledge, technical capability, and/or clinical practice in one or more broad fields� �Describe how the concepts, methods, technologies, treatments, services, or preventative interventions that drive this field will be changed if the proposed aims are achieved�

15. Common Pitfalls Related to Significance Omits seminal prior work (e.g., studies, theories, services, interventions), omits relevant work by study section members, is outdated, or too narrowly focused on one discipline Poorly organized, poorly focused, and/or confusing; contains irrelevant information The argument that project aims address an important problem or a critical barrier to progress in the field isn�t persuasive Lacks active, strong, direct language Does not clearly identify critical gaps in �scientific knowledge, technical capability and/or clinical practice� that this study could address Does not document the magnitude or seriousness of the problem in compelling ways For NIH, need to quantify impact of disease on health, society, economy, but statistics alone rarely persuade

16. Common Pitfalls Related to Significance(continued) Fails to articulate significance of the proposed work in the context of gaps or limitations in existing research, theory, services, interventions, etc. Know the controversies, issues, and unasked/unanswered questions in your field Argues for the significance of a problem, but not the significance of the proposed solution References are not carefully selected Describes but doesn�t synthesize, summarize, critically evaluate Does not clearly articulate connection to the funder�s mission (e.g., �to extend healthy life and reduce the burdens of illness and disability,� for NIH) Forgot to check out recently-funded competition

17. Conceptual Framework(s) or Model(s) May be described under Significance, Innovation, and/or Approach

18. Common Pitfalls Related to Conceptual Framework(s) Lack of fit (or unclear fit) between conceptual model(s) and aims, hypotheses, design, sample, measures and analysis plan �Heritage� and innovative features, if any, not clear No context � how does conceptual model fit with and/or extend existing research and theory? Failure to mention competing or alternative frameworks Not state of the science, outdated theories Doesn�t fit with proposed intervention (e.g., doesn�t cover mechanisms of change), if relevant

19. Common Pitfalls Related to Conceptual Framework(s) � (continued) Too complicated (occasionally, too simplistic) Too narrowly-restricted to one discipline Fails to adequately address key features of proposed project (e.g., longitudinal design, minority population) No visual diagram, confusing diagram, diagram inconsistent with the guiding theory Contradicts other parts of the application (e.g., specific aims and hypotheses) Omits key constructs Goes beyond proposed project (e.g., depicts a long-range research agenda) No rationale provided for this choice

20. Innovation �Explain how the application challenges and seeks to shift current research or clinical practice paradigms� �Describe any novel theoretical concepts, approaches or methodologies, instrumentation or interventions to be developed or used, and any advantage over existing methodologies, instrumentation, or interventions� �Explain any refinements, improvements, or new applications of theoretical concepts, approaches or methodologies, instrumentation, or interventions�

21. Common Pitfalls Related to Innovation Not innovative! It�s already been done (worst of all, by a member of the study section) Innovative features aren�t clearly identified Why a feature is innovative (e.g., compared to existing methodologies, measures, etc.) isn�t clear Fit between the problem, gap, or limitation that the innovation seeks to address and the innovation isn�t tight What this innovation will give us isn�t clear Innovative feature wasn�t reflected in specific aims, when it should have been

22. Common Pitfalls Related to Innovation(continued) PI fails to consider all aspects of the proposed research for innovation No expertise on the research team to successfully implement the innovation Too innovative (too risky, too radical)

23. Preliminary Studies Preliminary Studies, if any (for new applications), can go in any of the three sections of the Research Strategy. Most often, they have been showing up under Approach. Provide preliminary support for significance, proposed aims and hypotheses, methods, measures, study design, sample Establish capabilities of the investigators (not just the PI)

24. Common Pitfalls Related to Preliminary Studies Not clearly and explicitly connected to the proposed research Described but not synthesized or summarized Focus on mechanics rather than critical results Preliminary studies by key personnel are omitted Limited links drawn (e.g., omit whether a preliminary study demonstrates ability of the research team to collaborate or the PI�s competence?) No preliminary studies included (if appropriate to the funding mechanism and the stage of the study) No preliminary evidence of intervention feasibility, acceptability, or efficacy, if relevant

25. Approach �Describe the overall strategy, methodology, and analyses to be used to accomplish the specific aims� �Discuss potential problems, alternative strategies, and benchmarks for success anticipated to achieve the aims� �If the project is in the early stages of development, describe any strategy to establish feasibility, and address the management of any high risk aspects of the proposed work� �Point [out] any procedures, situations, or materials that may be hazardous to personnel and precautions to be exercised�

26. Common Pitfalls Related to Approach Inconsistent with Specific Aims Inconsistent with Significance Inconsistencies within or between sections under Approach (e.g., measures and analysis plan) Investigators lack relevant expertise Not state of the science Fatal flaws (e.g., fail to include control group if relevant) Fail to clearly delineate responsibilities and timeline Potential limitations not identified, potential problems not anticipated, and/or no consideration of alternatives

27. Common Pitfalls for Selected Elements Under Approach Study Design and Procedures Sample Measures Analysis Plan Timeline

28. Common Pitfalls Related to Study Design and Procedures Not tightly connected with specific aims, hypotheses, conceptual framework, sample, measures, and analysis plan Does not adequately address threats to internal and/or external validity Overly ambitious or questionable feasibility No letters of support for critical elements of the design (e.g., agreement of recruitment or data collection sites) Boiler-plate description, not tailored to the proposed study Concerns about human subjects protections (e.g., confidentiality)

29. Common Pitfalls Related to Study Design and Procedures (continued) Measurement occasions (number, timing) not well-justified Not state of the science Not adequately resourced or over-resourced (e.g., money, time, personnel, equipment) Rationale lacking for proposed choices Inadequate description of research settings where data will be collected or services/interventions will be delivered Inadequate information about procedures for participant assignment to condition, if experiment or intervention No plan to address potential adverse effects or legal responsibilities of data collection, if relevant

30. Common Pitfalls Related to Study Design and Procedures (continued) No plan described for training and monitoring of data collectors, data abstracters, raters, interventionists, etc., if relevant Inadequate plan for combining data collected from different sources or different methods, if relevant Inadequate description of experimental conditions or intervention arms, as well as control or comparison groups, if relevant Timing, frequency, duration, and sequencing of intervention(s) or experimental condition(s) not clear, if relevant

31. Common Pitfalls Related to Study Design and Procedures (continued) Rationale for all experimental or intervention conditions and all control or comparison groups not clear Intervention(s) too complicated Weak design for disentangling active ingredients in intervention Questionable feasibility and acceptability of intervention and/or plan for assessing feasibility and acceptability is weak Questionable generalizability from data collection site(s) -- particularly if a single site

32. Common Pitfalls Related to Study Sample Lacks clear definition of and rationale for target population Lack of expertise on research team related to target population Inadequate information about recruitment settings and procedures, sample selection inclusion and exclusion criteria No consideration of potential limitations or bias of proposed sample or sampling procedures No consideration of potential sampling problems and alternative strategies If a longitudinal design, no strategies for sample retention or sample replenishment No power analysis for sample size; proposed sample size not well-justified (too small or too large)

33. Common Pitfalls Related to Study Sample(continued) No (or inadequate) special recruitment strategies for enhancing representation of underrepresented populations Proposed sample size unlikely to yield statistically significant results, either in the sample as a whole or in key subgroups; concerns about low power Weak sampling design (e.g., convenience sample) Overlooks issues related to sample identification, recruitment, retention of settings as well as individuals, if relevant Concerns about human subjects protections (e.g., vulnerable populations)

34. Common Pitfalls Related to Measures Not state of the science Lack evidence for reliability and validity Concerns about cultural (or age) validity and/or cultural (or age) invariance Poor fit with specific aims and hypotheses, conceptual framework, study design, sample, or analysis plan Measures omitted for some constructs Single-item or nominal-level measures when better alternatives exist Lack of expertise on research team related to proposed measures

35. Common Pitfalls Related to Measures(continued) Lack of clarity about level of aggregation, if any Lack of clarity about timing and source of measures Inattention to threats to measurement reliability (e.g., diurnal fluctuation in biological markers; memory biases) Fidelity measures, if relevant, missing or weak Measures not clearly and closely tied to intervention content, process, and intended outcomes, if intervention Questionable feasibility and acceptability of measures Too many measures; possibility of participant burden or fatigue not adequately addressed

36. Common Pitfalls Related to Measures(continued) �Untested� new measures Measures not suitable for the mode of data collection Strategies for reducing measurement error not incorporated Inadequate attention to measurement limitations (e.g., all self report) and possible alternatives No pretesting or pilot-testing of measures, if relevant

37. Common Pitfalls Related to Analysis Plan Inconsistent with specific aims and hypotheses, conceptual framework, design, sample, measures Generic, boiler-plate; not tailored to proposed study Not clearly and explicitly organized by specific aims and hypotheses Omits essential elements (e.g., screening for violations of assumptions, measurement development) Fails to control for key confounds or covariates Not state of the science; too simplistic Overly ambitious; of questionable feasibility Concerns about statistical power and effect size

38. Common Pitfalls Related to Analysis Plan(continued) Lack necessary expertise on research team Under-resourced in budget (e.g., effort, people, software, etc.) Poorly organized Too complicated (e.g., 5 types of analysis when 2 or 3 will suffice) No (or inadequate) discussion of analytic challenges and limitations and no (or inadequate) alternative plans No plan for assessing interrater reliability, if relevant No plan for synthesis or integration of mixed methods data (e.g., quant + qual, self-report + biological) Inadequate detail about qualitative analysis plan, if relevant

39. Common Pitfalls Related to Analysis Plan(continued) No evaluation of psychometrics No plan for controlling Type I error Statistical assumptions unlikely to be met and no alternative plan No analytic plan for missing data and/or attrition No analytic plan for establishing baseline comparability between groups (regardless of random assignment)

40. Common Pitfalls Related to Timeline Unclear Overly ambitious Omits key tasks Doesn�t leave adequate time for dissemination and publication