Piloting Your Research to Success: The Why, What and How of Pilot Study

1. Piloting Your Research to Success: The Why, What and How of Pilot Study Nancy A. Baker, ScD, OTR/L University of Pittsburgh, Pittsburgh PA Susan L. Murphy, ScD, OTR University of Michigan and VA Ann Arbor Health Care System, MI

2. Conflict of Interest / Disclosure Statements Nancy A. Baker has nothing to disclose. Susan L. Murphyhas nothing to disclose.

3. Getting Funded: Why Pilot Data are Important for Grant Applications Nancy A. Baker ScD, MPH, OTR/L

4. Objectives: • Define pilot studies • Examine the relationship between pilot studies and the main research project

5. Pilot Studies • Preparatory studies designed to testthe science, process, management, and resources of subsequent larger studies. • Pilot studies tend to test piecesof a project that will inform the conduct of the larger study • This is in contrast to ‘pilot trials’ • Pilot studies are vital for developing larger studies Charlesworthet al., BMC Medical Research Methodology 2013, 13: 78; Moore et al., CTS. 2011;4:332-337; Thabane et al. BMC Medical Research Methodology. 2010;10:1

6. Phases of Intervention Trials Campbell, M. et al. BMJ 2000;321:694-696

7. Preclinical - Theory • Systematic Reviews (All relevant disciplines) • Literature • Qualitative studies Campbell, M. et al. BMJ 2000;321:694-696

8. Phase I - Modeling • Mechanistic studies • Descriptive studies • Qualitative studies • Identify outcome measures Campbell, M. et al. BMJ 2000;321:694-696

9. Phase II - Exploratory • Refine intervention/ control • Feasibility of protocols • Train personnel • Test outcome measures • Preliminary effect • Qualitative studies Campbell, M. et al. BMJ 2000;321:694-696

10. Phase III – Definitive Trial • Main trial • Multi-site • Sample size Campbell, M. et al. BMJ 2000;321:694-696

11. Phases of Intervention Trials Pilot Studies Campbell, M. et al. BMJ 2000;321:694-696

12. Pilot Studies Aims, research questions and design should address issues that affect future trials Moore et al., CTS. 2011;4:332-337; Thabaneet al. BMC Medical Research Methodology. 2010;10:1

13. Process Data • Data on the feasibility of processes Tickle-Degnen. AJOT. 2013;67:171-176

14. Resources Data • Data on the availability and sufficiency of resources Tickle-Degnen. AJOT. 2013;67:171-176

15. Management Data • Data on Human and Data optimization Tickle-Degnen. AJOT. 2013;67:171-176

16. Scientific Data • Data on intervention/control safety, dose, effect, and variance of effect Tickle-Degnen. AJOT. 2013;67:171-176

17. Pilot Studies • Provide specific information for designing future studies • Technical aspects • Develop preliminary data • Foster investigator development • Ensures that main trial appropriate and feasible

18. Getting Started with Pilot Studies: Appropriate Designs, Interpretation and Reporting Strategies Susan L. Murphy ScD, OTR/L

19. Objectives • To discuss research design considerations for pilot studies • To examine interpretation of pilot studies • To discuss reporting of results of pilot studies • To discuss funding opportunities

20. Pilot Studies are Important • Research studies that are adequately powered to test hypotheses occur because all or some elements of them were tested previously • Pilot studies allow researchers to know facts that will be important in conduct of larger trials before research stakeholders commit resources (Tickle-Degnen, 2013)

21. Design Considerations • Scope needs to be carefully considered • The optimal pieces to test should be selected based on an understanding of the theory, research evidence, and other sources of pilot data that support the larger study • In other words, ‘where is the science at?’

22. Art of Designing Pilot Studies • Rooted in understanding of where science is at for your research question • Scope suited to available resources • Prioritization of the pilot study aims • Set criteria for success when assessing feasibility

23. Examples: Phase I -- Modeling • Define intervention components • Which are most relevant to the outcome?

24. Example of Phase I Pilot Studies • LTG: To design an knowledge translation (KT) intervention for rehabilitation therapists in acute care to use evidence in practice • STGs: Determine what to include in intervention, optimal intervention delivery • Possible research methods? • Focus groups • Semi-structured interviews • surveys with health professionals/supervisors

25. Focus Groups from Health Care Professionals

26. Interpretation of Findings • Researcher designed an intervention including: • Tools for electronic access of research • Assessing quality of research • Identification of potentially applicable assessments for use in acute care • Researcher worked with clinical setting • to support initiative for EBP training • to allow time and space for intervention

27. Phase II: Exploratory studies/trials • Define trial and intervention design • Prioritize elements to test

28. Some pieces that can be tested in pilot studies: • Is your project feasible/acceptable to participants? • subject recruitment, procedures/intervention not overly burdensome, are people compliant)? • Are the risks understood and planned for (adverse events)? • Are you (and team) able to carry out such a project? • ability to coordinate, collect and analyze the data • Is there support for your intervention or its components? • Do you have procedures to ensure treatment fidelity? • Are the study instruments you have chosen to use appropriate?

29. Establishing criteria for success

30. Prioritizing pieces affects design • 1 group versus 2 group design • Example • Goal: test out intervention protocol and fidelity of therapists -- 1 group design • Goal: examine burden and drop-out rates with and without tx -- 2 group design

31. Example: KT Intervention Feasibility Study • KT intervention components are established (Phase I study) • Goal of this study to examine feasibility of delivering the intervention and treatment fidelity • Research design? • One group pretest/posttest

32. KT Intervention Feasibility Study • Researcher plans to enroll 20 participants from OT acute care unit • Group intervention that will occur biweekly for three months at lunch hour • The clinical setting has agreed to support the intervention including extended time for lunch

33. Measuring Chosen Feasibility Aims Treatment Fidelity Article--Bellg et al. Health Psychology, 2004, 23: 5: 443-451.

34. Power and Pilot Studies • Power analysis for a pilot study? • Important that study has enough sample to allow test of feasibility • Some review groups look for this power analysis, disagreement in literature about appropriateness of powering • Should estimating power be a study aim? • Caution is recommended • pilot study effect size ≠ population effect size Kraemer HC, et al. Arch Gen Psychiatry 2006; 63: 484-489.

35. Interpreting and Reporting results • Important to establish a-priori criteria for success • Process Examples • Recruitment--At least 70% of all eligible patients can be recruited (POET pilot study Chou et al., 2009) • Follow-up—Complete follow-up in at least 95% of all recruited subjects • Feasibility of Intervention Example • Low perceived levels of fatigue, anxiety, and perceived difficulty during task training of people with PD (Canning et al., 2008)

36. Pilot Study Characteristics • Exploratory • Test of feasibility of processes • Test of issues with resources or management • Safety/dose considerations • Data generating • Not for hypothesis-testing • May or may not include an attention or control group

37. From a Talk by Kelli Allen PhD “How to get funding and resources to conduct pilot studies” Funding opportunities

38. Funding Opportunities: Grants for “Stand-Alone” Pilot Studies Slide Source: Allen, K. “How to get funding and resources to conduct pilot studies” • NIH & AHRQ • R03s (some just for new investigators, some do not allow pilot trials) • R21s (require higher level of innovation) • Minority and other Supplements • Department of Veterans Affairs • Foundations • Rheumatology Research Foundation (targeted disease areas) • American Occupational Therapy Foundation • National Psoriasis Foundation • Foundation for Physical Therapy • American Federation on Aging & Arthritis National Research Foundation • Scleroderma Foundation • Internal Institutional Awards

39. Grants for “Stand-Alone” Pilot Studies: Pros and Cons Slide Source: Allen, K. “How to get funding and resources to conduct pilot studies” • Pros: • Resources for novel data collection • Experience with grant writing • Establishes funding track record • Cons • Time delay with review process (tends to be shorter with internal awards) • Funding rates • Limited budget • Some provide minimal support for investigator • Some funding periods brief

40. Funding Opportunities: Career Development Awards Slide Source: Allen, K. “How to get funding and resources to conduct pilot studies” • NIH & AHRQ – Various K-Awards • Institutional Career Development Awards (CTSA, T32) • Department of Veterans Affairs • Rheumatology Research Foundation • Scientist Development Award • Investigator Award • Robert Wood Johnson Foundation • Other Foundations and Professional Societies

41. Career Development Awards: Pros and Cons Slide Source: Allen, K. “How to get funding and resources to conduct pilot studies” • Pros: • Protected research time for several years • Allows for well developed pilot data • Resources for novel data collection • Experience with grant writing • Establishes funding track record • Formal mentorship and training • Cons • Time involved in submission and review process (shorter for internal) • Funding rates

42. Other Resources for Pilot Studies: Publicly Available Data Slide Source: Allen, K. “How to get funding and resources to conduct pilot studies” • Are there publicly available data sets that support aspects of your proposed research question(s)? • Examples: • Osteoarthritis Initiative • National Health and Nutrition Examination Surveys • National Health Interview Survey • Health and Retirement Study • CDC Behavioral Risk Factor Surveillance System • Mining Electronic Health Records • Many others!

43. Publicly Available Data: Pros and Cons Slide Source: Allen, K. “How to get funding and resources to conduct pilot studies” • Pros: • Free (usually) & often immediately available • Often large samples with strong generalizability • Can help establish publication track record • Cons • May not exactly answer your question • No resources for analysis or protected time • Does not provide grant writing practice or establish funding track record

44. Other Resources for Pilot Studies: Data from Mentors / Collaborators Slide Source: Allen, K. “How to get funding and resources to conduct pilot studies” • Secondary analyses of existing data • Adding variables to ongoing projects • Collecting qualitative data related to new project • Networking: • Reach beyond your immediate study team and institution • Who is doing work in your area(s) of interest, and how might you collaborate?

45. Summary & Considerations:Which Road to Take? Slide Source: Allen, K. “How to get funding and resources to conduct pilot studies” • Primary Consideration: What kind of pilot data will result in a competitive grant application? • Consult with mentors, collaborators, program officer • What resources do you have? • Internal pilot / career awards? • Data from ongoing studies? • Statistical support? • What resources do you need? • Salary support? • Mentorship? • Training?

46. References Campbell M, Fitzpatrick R, Haines A, Kinmonth AL, Sandercock P, Spiegelhalter D, Tyrer P. Framework for design and evaluation of complex interventions to improve health. BMJ 2000, 321:694-696. http://www.ncbi.nlm.nih.gov/pmc/articles/PMC1118564/pdf/694.pdf Charlesworth G, Burnell K, Hoe J, Orrell M, Russell I. Acceptance checklist for clinical effectiveness pilot trials: a systematic approach. BMC Medical Research Methodology 2013, 13: 78. http://biomedcentral.com/1471-2288/13/78 KianifardF, Islam MZ. A guide to the design and analysis of small clinical studies. Pharmaceutical Statistics 2011, 10:363-368. doi:10.1002/pst.477. Lancaster GA, Dodd S, Williamson PR. Design and analysis of pilot studies: recommendations for good practice. Journal of Evaluation in Clinical Practice, 10, 2, 307-312. Leon AC, Davis LL, Kraemer HC. The role and interpretation of pilot studies in clinical research. J Psychiatr Res, 2011, 45: 626-629. Moore C, Carter RE, Nietert PJ, Stewert PW. Recommendations for planning pilot studies in clinical and translational research. Clinical and Translational Science 2011; 4:332-337. doi: 10.1111/j.1752-8062.2011.00347.x Shanyinde M, Pickering RM, Weatherall M. Questions asked and answered in pilot and feasibility randomized controlled trials. BMC Medical Research Methodology 2011, 11: 117. http://biomedicalcentral.com/1471-2288/11/117 ThabaneL, Ma J, Chu R, et al. A tutorial on pilot studies: the what, why, and how. BMC Medical Research Methodology 2010, 10:1 http://biomedicalcentral.com/1471-2288/10/1 Tickle-Degnen L. Nuts and bolts of conducting feasibility studies. AJOT 2013, 67; 171-176. doi 10.5014/ajot.2013.006270.