DHDSP Outcome Indicators for Policy and Systems Change: Indicators 101

1. Hilary K. Wall, MPH DHDSP Grantee Meeting September 12, 2011 National Center for Chronic Disease Prevention and Health Promotion DHDSP Outcome Indicators for Policy and Systems Change: Indicators 101 Division for Heart Disease and Stroke Prevention

2. Indicators Project Team Hilary Wall, Applied Research and Translation Team (co-lead) Rachel Barron-Simpson, Evaluation and Program Effectiveness Team (co-lead) Eileen Chappelle, EPET Rashon Lane, EPET Evaluation TA providers Project Officers

3. Session Outline What are indicators? DHDSP Outcome Indicators for Policy and Systems Change Available tools Using Indicators in program evaluation FAQs

4. What are indicators? Specific, observable, and measurable characteristics that show progress a program is making toward achieving a specified outcome. Measuring Program Outcomes: A Practical Approach. Alexandria, VA: United Way of America; 1996

5. Why are indicators important? Indicators tell us specifically what to measure to determine whether desired objectives have been met. For any outcome there are usually many possible indicators, but some are more appropriate and useful than others http://pdf.usaid.gov/pdf_docs/PNABY214.pdf

6. Example How have people’s driving habits changed? (i.e. Are people driving less?)

7. DHDSP Indicators In 2005, DHDSP started an indicators project to develop common evaluation measures for use by all doing heart disease and stroke prevention (HDSP) efforts DHDSP created the Outcome Indicators for Policy and Systems Change (Indicators) Four priority areas Four settings Support the use of evidence to guide the planning and outcome evaluation of state HDSP programs

8. DHDSP Indicators Summary books with ~60 indicators/priority area (the “menu”) Evidence-based logic models and outcome indicators for each priority area Expert panel reviews for each priority area Consumer Reports-type ratings to allow for flexibility in local tailoring Individual indicator profiles

9. Logic Model for Controlling High Cholesterol

10. Indicators Across Time Short-term Outcomes Health Care Systems/Provider Changes Worksite Changes Community Changes Intermediate Outcomes Individual behavior changes Medication adherence Average and control of BP/Chol Long-term Outcomes Longer-term morbidity, death, and cost

11. Consumer Reports-like Ratings

12. Indicator Number Designation

13. “There are too many Indicators!”

14. Indicator Cycle of Use

16. Using indicators in evaluation

17. Steps in HDSP Program Planning & Evaluation Select an ultimate outcome that is achievable given the timeframe, resources, political climate, etc. Sketch a logic model pathway leading to the ultimate outcome

18. DHDSP Controlling HBP Logic Model

19. Steps in HDSP Program Planning & Evaluation Select an ultimate outcome that is achievable given the timeframe, resources, political climate, etc. Sketch a logic model pathway leading to the ultimate outcome Select relevant outcome indicators along the pathway

20. Example Pathway of Indicators

21. Steps in HDSP Program Planning & Evaluation Select an ultimate outcome that is achievable given the timeframe, resources, political climate, etc. Sketch a logic model pathway leading to the ultimate outcome Select relevant outcome indicators along the pathway Construct project outcome objectives Implement evidence-based activities that will accomplish these objectives (http://www.cdc.gov/DHDSP/programs/nhdsp_program/docs/ABCs_Guide.pdf) Design and conduct an appropriate evaluation

23. Indicator Operationalization Once a state has selected indicators and created an evaluation plan, indicators must be “operationalized” Logical thought process that tailors generic indicators Second nature to epidemiologists Often a missing step in state HDSP program evaluations

24. Indicator Template Key concepts Intended population Recommended measures Data source Numerator Denominator Time period Frequency of collection Unit of analysis Level of analysis Representativeness Measurement issues Limitations Responsibility for collection & analysis

25. Indicator Operationalization Involve diverse stakeholders Explore the literature Explore existing data sources Conduct an environmental scan Use the indicator template Brainstorm Call for help when you get stuck Put formal data agreements in place

26. Steps in HDSP Program Planning & Evaluation Select an ultimate outcome that is achievable given the timeframe, resources, political climate, etc. Sketch a logic model pathway leading to the ultimate outcome Select relevant outcome indicators along the pathway Construct project outcome objectives Implement evidence-based activities that will accomplish these objectives Design and conduct an appropriate evaluation Reflect on experiences and share results

27. Frequently asked questions

28. Q: How many indicators do I need to have for each project? A: There is no set number. A state should pick enough indicators to reflect the logical pathway of what they are trying to accomplish.

29. Q: How does CDC want states to define [any given indicator]? A: CDC has no expectations for how states define indicators for their context. States should define indicators so that they best showcase the impact of their initiatives.

30. Q: Does my state need to use all of the indicators listed for a given strategy in the ABCS guidance document? A: No. Many of the strategies in the guidance document have several possible indicators that could be used to monitor progress.

31. Q: What indicators should states use to monitor sodium-related work? A: Sodium is a strategy for Controlling Blood Pressure so indicators should be pulled from that priority area but may also likely include practice-based indicators. See the Indicators Spotlight: Sodium Reduction tool for more info.

32. Q: Where can I find the Indicators online? A: An early draft of the Controlling High Blood Pressure indicators can be found online: http://www.orau.gov/hsc/hdspinstitute/2008/presentations/Session13-25-HDSPIndicators.pdf. The other Indicators are only available via PDFs. An Indicators Website will be rolled out in 2012.

33. Q: Do grantees have to incorporate short- term, intermediate, and long-term indicators into their evaluation plans? A: Grantees should include these types of indicators in their plans. However, they should focus their data collection efforts on gathering information to address short-term and intermediate outcomes.

34. Q: My state is implementing a QI intervention. The quality of care indicators have not been released. How do we proceed with the evaluation? A: The Improving Quality of Care Indicators are for hospital-based QI. If you are doing ambulatory care QI, use the Controlling High Blood Pressure or Cholesterol Indicators.

35. Q: Our state is implementing an innovative intervention. None of the short-term indicators would apply to our effort. Do we still have to use the indicators? A: Innovative interventions that do not have outcomes documented in the scientific literature should definitely be evaluated by using practice-based indicators developed for your initiative’s context.

36. Q: If my state gathers all the necessary indicators, are we done with our evaluation? A: No, collecting information only on DHDSP outcome indicators is not sufficient for a complete evaluation. A full program evaluation would likely also include a process component designed to gather and compile lessons learned, program implementation challenges, and unexpected findings, among other features.

37. Other Indicators for Chronic Disease Evaluation

38. For More Information WK10A – Demonstrating Public Health Reach and Impact Using Key Indicators Wednesday, September 14 9-11:30am 3-5:30pm (repeated) RT8 – DHDSP Indicators: Tailoring Indicators for Your Context through Operationalization Wednesday, September 14 1-2:30pm

39. Thank you! Hilary Wall hwall@cdc.gov National Center for Chronic Disease Prevention and Health Promotion Division for Heart Disease and Stroke Prevention