Rationale for Project

1. EVALUATION OF THE EFFECTIVENESS OF AHRQ’s GRANT SUPPORTED RESEARCH ON HEALTHCARE COSTS, PRODUCTIVITY, ORGANIZATION, AND MARKET FORCESOVERVIEW OF PROJECT FINDINGSby Marsha Gold, Sc.D., Project DirectorTimothy Lake, Ph.D., Deputy DirectorKate Stewart, Ph.D., ResearcherTara Krissik, M.P.P., ResearcherDecember 2008This project was funded by the Agency for Healthcare Research and Quality

2. Rationale for Project • While AHRQ’s statutory mission explicitly includes support for grant funded research on healthcare costs, productivity, organization, and market forces, the agency does not systematically track the work it funds in this area and what has been accomplished through it. • Purpose of this project is to address this gap and identify how AHRQ may improve the visibility and impact of this research.

3. Project Details • MPR evaluation began in October 2006; the two phased project was completed in December 2008. • Work was guided by an Expert Panel: Sharon Arnold (AcademyHealth), John Christianson (University of Minnesota), Paul Ginsburg (Center for Studying Health Systems Change), Robert Helms (AEI), and Gail Wilensky (Project Hope) • Michael Hagan was the AHRQ project officer.

4. Key Questions of Interest 1. Since the late 1990s, what grant research has AHRQ funded that relates to healthcare costs, productivity, organization, and market forces? 2. How are the research findings disseminated to public and private decision makers and what factors contribute to their use? 3. What is AHRQ’s role in supporting research in this area and how does it compare with that of others, such as NIH and private funders? 4. What actions, if any, could enhance AHRQ’s efforts to track, disseminate and encourage use of these research findings?

5. Components of the Study - I • Descriptive Analysis: Reviewed AHRQ administrative data on all grants, and conducted indepth analysis of a stratified random sample of nine grants. • Conceptual Framework: Using relevant social science literature, identified pathways for translation and key variables that are likely to influence use of research. • Case Studies: In depth description and analysis of 7 grants that illustrate diverse ways in which study findings reach target audiences and influence policy debates; based on PI and user interviews.

6. Components of the Study - II • Comparative Funder Analysis: Comparison of AHRQ’s infrastructure and funding for grants in this area to that of NIH (government) and HCFO (private sector). • Federal Research Translator/User Interviews: Identify how and when this type of research gets used, and awareness of AHRQ research. (Included CBO, GAO, MedPAC, FTC, DOJ, and selected others). • PI Survey: Web based survey in July-August 2008 to all 149 grantees (70% response). Detail on research focus and key findings, dissemination modes and preferences, uses made of results, and PIs interaction with policymakers. Also obtained feedback on AHRQ’s role and performance and how AHRQ compares to other funders.

7. Components of the Study - III

8. KEY FINDINGS: WHAT HAS AHRQ FUNDED IN THIS AREA BETWEEN 1998 – 2006?

9. Number of New AHRQ Grants for Research on Healthcare Costs, Productivity, Organization, and Market Forces Funded, 1998-2006 Fiscal Year Source: MPR analysis of AHRQ Administrative Data.

10. AHRQ Funding for Research on Healthcare Costs, Productivity, Organization, and Market Forces, 1998-2006 Fiscal Year Source: MPR analysis of AHRQ Administrative data. Note: Includes funding for new grants and continuing grants that were funded in 1998 or later. The combined spending over the period was $81.4 million, of which $12.6 million was for three PO1 grants. 47 RO3 grants under $100,00 accounted for about $4 million in spending.

11. Characteristics of Funded Studies • Of the 149 studies: • 97 studies had organizations as the unit of analysis. Studies of hospitals and health plans were particularly common. • 37 studies examined consumer behavior. • 15 studies examined markets or purchaser behavior. • 75 percent of the 149 were national in scope. • Multiple outcomes studied including quality (53%), use (44%), cost (38%) and access (19%), among others.

12. Nature of Research PI survey indicates that findings commonly examine how specific outcomes of care are influenced by: • Economic factors (e.g., provider payment, insurance coverage) • Organizational characteristics (e.g., nurse leadership, volume) • Systems and markets (e.g., HMO penetration, capacity constraints) • Patient characteristics and preferences (e.g., percent minority, DNR orders) Findings address questions of substantial policy interest today about influences on the performance of health care system.

13. Key Findings: How are Findings Disseminated and What Factors Contribute to Use

14. Dissemination May Occur Through Diverse Pathways - I • We identified 10 pathways that differ in means used to communicate findings and the role researchers, intermediaries and users play in the process. Pathways include: • Traditional: “Big Bang,” Gradual Accumulation and Diffusion, Gradual Communication and Formal Syntheses • Effective Use of Intermediaries: Researcher as Messenger/Expert, Formal Intermediary Broker, Press Publicizes • User Guided: User Directed Syntheses, User Partner Grant Review, User Commissioned Studies, Researcher as User

15. Dissemination May Occur Through Diverse Pathways - II • Some pathways involve researchers directly engaging with users and others involve intermediaries that help synthesize, apply or communicate the findings. • Diverse pathways will be effective in different circumstances; effectively reaching diverse audiences for the work enhances the use of well targeted high quality research.

16. PI Survey Shows Preference for Publication as Dissemination Strategy but Use of Multiple Modes Source: MPR Survey of AHRQ-Funded Principal Investigators. Note: Responses based on 97 responding PIs (70% of grantees surveyed).

17. Support from Host Institution for Dissemination Limited Source: MPR Survey of AHRQ-Funded Principal Investigators. Notes: N = 91. These analyses exclude 6 respondents who did not respond to any questions about dissemination support.

18. Mean Percentage of Time PIs Report Interacting with Policymakers Source: MPR Survey of AHRQ-Funded Principal Investigators.

19. The Impact of Research on Policy - I • Research is one of several influences on decision-making. Importance of research likely to vary with the topic, user, environment and other factors. • Some research may have an immediate impact but most accumulates and gets applied to relevant topics as issues arise. • Accumulation of research contributes to a “research reservoir”—use more likely if findings readily available to potential users and applicability of findings clear.

20. The Impact of Research on Policy - II • Accumulation of findings across multiple studies adds weight to findings. • Some forms of communication are likely to be stronger in reaching users than others. • Researchers can enhance usefulness by thinking carefully about the key findings and identify the main “message” of the study (“elevator test”).

21. PI Perception of Outcomes Relevant to Their Grants Source: MPR Survey of AHRQ-Funded Principal Investigators. Note: N = 85.

22. Lessons from Case Studies on Ways to Enhance Use • Develop relationships with potential users and involve them early • Be aware of how results may be relevant to different policy decisions and the timing of those decisions • Identify where research “fits” in stream of research • Develop expertise—and reputation for it

23. They make extensive use of this type of research They are familiar with ongoing work and consult the literature as time allows They value timeliness but also place a high value on quality and objectivity Regulatory entities find general studies of markets useful in providing context for case specific analysis They see critical gaps in the existing body of research in this area What Federal Research Translators Told Us

24. KEY FINDINGS: WHAT IS AHRQ’S ROLE IN CONTEXT OF NIH, HCFO, AND OTHER FUNDERS?

25. Researchers view AHRQ as a major source of funding for this research but have turned elsewhere as AHRQ funds have become less available. Low funding levels for this kind of research at AHRQ was the most common problem cited by PIs. PIs view AHRQ’s oversight over grants more positively than the agency’s work on dissemination and communication. While NIH processes mirror those of AHRQ, HCFO has more emphasis on dissemination, and researchers score HCFO higher than other funders on technical assistance Federal translators perceive AHRQ as more a primary resource for databases and clinical research than for this type of research. General Findings Across the Project

26. Share of Respondents Who Consider Various Funding Organizations to be Major, Moderate, Minor, or Not a Source of Funding for Research on Health Care Costs, Productivity, Organization, and Market Forces (percentages unless otherwise noted) Source: MPR Survey of AHRQ-Funded Principal Investigators. Note: N = 96.

27. Ratings of AHRQ Technical Support Source: MPR Survey of AHRQ-Funded Principal Investigators. Note: Mean scores calculated based on respondent ratings of AHRQ technical support, where 1 = Excellent; 2 = Very Good; 3 = Good; 4 = Fair; 5 = Poor

28. Comparison of Technical Assistance Provided by AHRQ, NIH, HCFO, and Others Source: MPR Survey of AHRQ-Funded Principal Investigators. Note: Mean scores calculated based on respondent ratings of funders’ technical support, where 1 = Excellent; 2 = Very Good; 3 = Good; 4 = Fair; 5 = Poor.

29. KEY FINDINGS: ACTIONS THAT COULD ENHANCE USE OF RESEARCH FINDINGS

30. AHRQ currently is not highly visible as a source of research on health care costs, productivity, organization and market forces despite a legislative mandate for it and $81 million spent on grants for this kind of research from 1998 through 2006. Our findings show that the research AHRQ has funded addresses critical aspects of health system performance that are important to decision-makers. Research AHRQ has funded has had an impact but mainly project by project at the initiative of the investigator. AHRQ can better track its work and leverage its investments to make its research more visible, make findings more accessible, and enhance their use. Conclusions From the Project

31. Use our study as basis for ongoing tracking of research. Create a visible place on the Agency website to communicate AHRQ’s efforts in this area. Create internal capacity for monitoring grant progress to identify emerging outcomes that may be of interest; reward staff for support in this area. Communicate with PIs why AHRQ would value ongoing reports of dissemination and use after grants end and develop systems to support such monitoring. Recommendations for AHRQ - I

32. Use our project to develop an annotated bibliography with abstracts of publications from AHRQ funded research in this area. Work with others on targeted syntheses of findings on topics which AHRQ has invested. Help others contact experts on given research topics through online contact information on the website or other means. Offer researchers a “webinar” on ways of thinking about, developing, and communicating “messages” from the study. Recommendations for AHRQ - II

33. Elicit more feedback from users on their needs to identify areas that can be supported by indepth, high quality research. Work with others to better articulate what this type of research is and its relevance to understanding health system performance and how to enhance outcomes. Recommendations for AHRQ (and Others) - III