De-Mystifying Evidence

1. De-Mystifying Evidence Kauai County, Hawaii June 17, 2011 Presenter: Eric Albers, Ph.D. Evaluator, CAPT West Resource Team **Slides adapted from Dr. Carol McHale’s 2010 West Region RTEP Meeting

2. Learning Objectives • Understand CSAP’s guidance document Identifying and Selecting Evidence-Based Interventions • Know how to select and/or adapt appropriate evidence-based interventions • Know how to move innovative and community-based programs/initiatives along the continuum of evidence towards becoming an evidence-based interventions • Understand how to use and create logic models • Understand how SAMHSA’s Service to Science Initiative operates, including its nomination process

3. Who Am I?

4. Who Am I?

5. Who Are You?

6. Who Are You?

7. What is the CAPT? WEST Montana Wyoming Tribal Leaders Council CENTRAL Nooksack Indian Tribe Confederate Salish and Kootenai Tribes Red Lake Nation Ogala Sioux Tribe NORTHEAST Leech Lake Band of Ojibwe Northern Arapaho Tribe Little Traverse Bay Band of Odawa Indians Lower Brule Sioux Tribe Great Lakes Inter-Tribal Council California Native American Health Center WA ME Grand Traverse Band ND MT VT MN NH OR WI NY MA Northern Mariana Islands SD ID RI MI WY CT PA IA NJ DE NV IN OH IL NE UT MD WV Marshall Islands CA CO VA District of Columbia KS KY MO NC Palau TN AZ OK SC AR NM SOUTHEAST AL GA MS Federated States of Micronesia Tohono O’odham Nation Puerto Rico LA TX Pueblo of Acoma Hawaii First Nations Community Health Source FL AmericanSamoa Winnebago Tribe Cook Inlet Tribal Council US Virgin Islands Cherokee Nation OK Inter-Tribal Consortium (ITC) Guam Dena Nena Henash (TCC) AK SOUTHWEST Cohort V Partnerships for Success Northeast: ME,NY,NH,VT,MA,CT,RI,PA,NJ,MD,DE Southeast: KY,VA,TN,NC,SC,MS,AL,GA,FL, District of Columbia, Puerto Rico, US Virgin Islands Central: ND,SD,MN,IA,WI,IL,MI, IN,OH,WV, Lower Brule Sioux Tribe, Great Lakes Inter-Tribal Council, Little Traverse Bay Band of Odawa Indians, Grand Traverse Band, Ogala Sioux Tribe, Leech Lake Band of OjibweSouthwest: NE,CO,KS,MO,NM,OK,AR,LA,TX, Cherokee Nation, Winnebago Tribe, First Nations Community Health Source, Pueblo of Acoma, Oklahoma Inter-Tribal Consortium West: WA, MT,OR,WY,CA,NV,UT,AZ,AK,HI, California Native American Health Center, Northern Arapaho Tribe, Montana Wyoming Tribal Leaders Council, Tohono O’dham Nation, Confederate Salish and Kootenai Tribes, Nooksack Indian Tribe, Dena NenaHenash, Cook Inlet Tribal Council, Northern Mariana Islands, Guam, Marshall Islands, Micronesia, American Samoa, Palau Cohort IV Cohort I Cohort III Cohort II

8. Who Is Involved?

9. Discussion: What is “Evidence-Based”? Evidence-Based Science-Based Evidence-Informed Model Programs Best Practices Proven Practices

10. Evidence-Based Interventions (EBIs) • Included in Federal registries • Reported (with positive effects on the primary targeted outcomes) in peer-reviewed journals • Documented effectiveness using other sources of information

11. Advantages: Using Federal Registries • Concise descriptions • Documented ratings of evidence • Practical information • One-stop convenience

12. Challenges: Using Federal Registries • Offers a limited number of interventions • Includes interventions that are most easily evaluated using traditional scientific methods • Some information may be dated • May be confusing to the consumer

13. Advantages: Using Peer-Reviewed Journals • Presents detailed findings/analyses to determine whether there is adequate evidence that an intervention works • Provides authors’ contact information to facilitate further discussion (e.g., appropriateness of intervention for target audience, current research, how to appropriately adapt) • In some cases, include meta-analyses

14. Challenges: Using Peer-Reviewed Journals • Up to reader to interpret results and assess strength of evidence and applicability to specific population • Limited detail regarding activities and practical implementation issues, materials needed • Not all local communities have easy access to peer-reviewed journals or the knowledge or experience to search for interventions

15. Using “Other Information Sources” Four Guidelines • Grounded conceptually in theory of change or logic model • Similar to formally recognized EBIs • Demonstrated effective implementation with a consistent pattern of positive outcomes • Judged appropriate by panel of informed prevention experts

16. Advantages: Using “Other Information Sources” • Enables State and community planners to consider interventions that do not currently appear on Federal lists or in peer-reviewed literature but have potential • Provides opportunity for locally developed or adapted interventions to receive funding as an evidenced-based program (if they have adequate documentation of evidence)

17. Challenges: Using “Other Information Sources” • Increased responsibility for selection of  appropriate programs and practices • Potential confusion and/or biases in the implementation of a transparent decision-making process for reviewing candidate interventions • Potential differing opinions of what is considered evidence makes the establishment of agreed-upon criteria for documentation to support intervention selection a challenge

18. When to Use These Guidelines • When no appropriate interventions are available through primary sources to address: • Assessed community need • Population served • Cultural and community context

19. Process to Select Best Fit Prevention Interventions

20. Identifying and Selecting Evidence-Based Interventions • CSAP Website:http://prevention.samhsa.gov/evidencebased/evidencebased.pdf • CAPT Area of SAMHSA Website: http://captus.samhsa.gov • SAMHSA’s Health Information Network (SHIN): http://www.samhsa.gov/shin • Print copiesavailable: HHS Publication No. (SMA) 09-4205

21. SAMHSA/NIDA Recommended Guidance • Details a “best process” for selecting interventions • Sets a “high bar” for standards of evidence to guide selection decisions • Recognizes that selection decisions are “context dependent”

22. Utilizing Logic Models Discussion: • What is a logic model? • How can the logic model benefit my program?

23. Community Logic Model Outcomes-Based Prevention • Guides selection of strategies that fit conceptual logic model • Addresses salient risk and protective factors and underlying conditions • Drives positive outcomes related to the priority problem

25. Courtesy of University of Wisconsin-Extension

26. Utility Checklist

27. Feasibility Checklist

28. When to Use These Four Guidelines When no appropriate interventions are available through primary sources to address: • Assessed community need • Population served • Cultural and community context To document innovation and emerging evidence-based practices – “Service to Science”

29. SAMHSA’s Service to Science Initiative

30. Next Steps Discussion