Development of a Psychometrically Sound Self-Assessment Tool for PBRN Best Practices

1. Developing a Psychometrically Sound Instrument of Best Practices Specific to the PBRN Context Victoria Neale, Kendra Schwartz, Monina Bartoces, Jason Booza, Joseph Monsur, Marilyn Wayland: Wayne State University; and Brenda Hudson: Indiana University DISCUSSION ABSTRACT RESULTS • Analytic Strategy: • 1) Essential Phase: • Content Validity Coefficients (CVCs) identified the items that a majority of 76 PBRN experts rated as “essential” research best practices. • 12/59 items had negative CVCs and were eliminated; 47 items were retained. • 2) 360 Phase: • a) Cronbach’s coefficient alpha computed by role of respondent: • Directors/Research Directors (n = 33): α = 0.89 • Staff/Research Assistants ( n = 34): α = 0.93 • Community-based Staff ( n = 37): α = 0.94 • All respondents combined (n = 104): α = 0.93 • b) Factor Analysis • 31 items loading on 5 factors retained based on a priori criteria: eigenvalue> 1; factor loading > 0.45; accounting for > 5% of variance. • Factor 1: Data Management (10 items): overall α = 0.88 • Director/Research Director: α = 0.65 • Staff/Research Assistants: α = 0.83 • Community-based staff: α = 0.89 • Factor 2: Study Supervision (4 items): overall α = 0.85 • Director/Research Director: α = 0.91 • Staff/Research Assistants: α = 0.71 • Community-based staff: α = 0.80 • Factor 3: PBRN Policies (9 items): overall α = 0.83 • Director/Research Director: α = 0.73 • Staff/Research Assistants: α = 0.87 • Community-based staff: α = 0.85 • Factor 4: Study Management (5 items): overall α = 0.81 • Director/Research Director: α = 0.90 • Staff/Research Assistants: α = 0.54 • Community-based staff: α = 0.82 • Factor 5: Ethical Considerations (3 items): overall α = 0.63 • Director/Research Director: α = 0.63 • Staff/Research Assistants: α = 0.69 • Community-based staff: α = 0.59 • Objective: To develop a self-assessment tool for PBRNs to evaluate and improve their research culture. • Design: Multi-step formative process to develop consensus on best practices. • Setting: Internet-based survey data collection. • Participants: 3 respondents per 42PBRNs estimated the extent to which the PBRN followed 59 research best practices. • Outcome: A psychometrically-sound self-assessment tool of research best practices specific to the PBRN context. • Innovation and Significance • To date, the national research enterprise has functioned in a decentralized fashion, resulting in duplicative or undocumented processes, without acknowledged best practices. • This study used collective wisdom of experienced survey researchers and practice- based researchers to identify essential PBRN-specific research best practices. • The final product will be a self-assessment checklist of best practices that PBRNs can use to assess their current research culture. • The checklist has the potential to improve PBRN research culture by encouraging PBRNs to prioritize activities that strengthen the research professionalism of their organization. • By promoting research professionalism, this project has the potential to contribute to greater public willingness to participate in research that can build the evidence- base of primary care. • Next Steps: • 1) Develop Standard Operating Procedures (SOPs) for Each Best Practice • 2007-2008 funding from CTSA Community Engagement Supplemental Award: Laura Mae Baldwin is Co-Investigator; Victoria Neale and Rowena Dolor are consultants. • We will conduct a PBRN survey in early 2009 to: a) determine routine procedures and SOPs currently used by PBRNs; and b) ask PBRNs for insights into how SOPs could best be implemented in their setting. • 2) Disseminate and Evaluate the Utility of the Best Practices and their SOPs • Recruit PBRNs as test sites to implement and evaluate the SOPs. • Evaluate utility of the best practices and SOP model for improving PBRN research professionalism. • Determine motivators for PBRNs to engage in the self-assessment process and to use the Best Practices and their SOPs for continuous quality improvement. DISCUSSION RATIONALE INTRODUCTION • The practice-based research network (PBRN) movement has resulted in decentralized community-based research, which raises particular challenges to research integrity. • While practice-based primary care research offers the advantage of greater external validity, it also presents challenges that include: implementing standardized research protocols in decentralized settings; training and monitoring of ethical research practices among the community-based clinic staff; and data management and quality control issues. • To date, there is no PBRN-specific infrastructure to promote the integrity of studies conducted in decentralized settings. METHODS Progress to Date: 1) An extensive list of research integrity best practices was developed from the published literature and suggestions from PBRN experts. The wording for these PBRN-specific best practices was reviewed by 20 survey research experts recruited from the NAPCRG Consultants Directory. 2) 59 research best practices were organized into 7 domains: (study launch; study orientation; study supervision; data management & analysis; dissemination; professional training/development; and PBRN management). 3) “Essential Phase”: 76 PBRN experts rated the 59 items as essential or not. 4) “360 Phase”: 3 respondents from 42 PBRNs evaluated the use of 59 research best practices in their network. Respondents were in the role of : a) PBRN director/research director; b) network coordinator or experienced research assistant; and c) community-based staff person.