1. Change Starts Here. The One about Demonstrating Change ICPC National Coordinating Center This material was prepared by CFMC (PM-4010-075 CO 2011), the Medicare Quality Improvement Organization for Colorado under contract with the Centers for Medicare & Medicaid Services (CMS), an agency of the U.S. Department of Health and Human Services. The contents presented do not necessarily reflect CMS policy.

2. Recap: measurement for IC-4 • Time series outcomes • Effect on root cause/driver • Success of the intervention • Rates; scores; rating scales • Best-fit line or other signal indicating improvement • What to do about outcomes not well portrayed as time-series • Intervention implementation • Reach/dosage of an intervention • Who was affected? • Counts • Rates among eligible population (offered, refused, completed)

3. Recap: suggested approach • Map out a detailed, community-level logic model of the intervention strategy. • Select and operationalizeoutcomes and processes from the logic model. • Develop and enforce the system for tracking implementation and outcome. • Effectively report time series data.

4. Recap: timing and duration When will improvement be detected? • Considerations • How long should it take to observe an effect? • What should the effect look like? • IC-4: ≥4 quarters of data within 18 months of community engagement • Ensure that the measurement period includes pre-intervention baseline data. • Measure frequently • The more data points, the better. Monthly indicators lend themselves to run/control charts.

5. What’s this all about? • Purpose • Confidently demonstrate that the interventions were effective. • Link interventions to observed changes in readmissions. • Validate or revise the logic model based on short-term outcomes.

6. Be thoughtful and careful • Pitfalls • Waiting too long to begin the process • Checking progress infrequently • Going it alone • Advice • Rapid cycle improvement • Measure frequently, revise accordingly • Analytic support • NCC support

7. Tracking outcomes • Time series • Run chart or control chart (≥12 data points) • Trend line (fewer data points) • Cross-sectional; cohort • Group comparison • Intervention vs. no intervention • Beneficiary-level • Difference between pre- and post-intervention

8. Detecting improvement Does the intervention have an effect? • Run chart: special cause • Co-occurrence with intervention deployment • Best-fit trend line: statistical significance • Between-group or pre-/post-intervention differences achieved and sustained

9. Time series: run charts • Run chart • Data points overlaid against the median • Simple patterns suggest ‘special cause’ • Runs(consecutive points above or below median) • Number of runs • Length of run • Consecutive data points continually increasing/decreasing • Very high or low point • Resources: • Perla, Provost & Murray (2011) • http://qualitysafety.bmj.com/content/20/1/46.full.pdf

10. Run charts Perla, Provost & Murray (2011)

11. Time series: best-fit line • Best-fit line among available data points • Cochran-Armitage test • Statistical significance: slope of the line is different from zero • Requires analytic support • Resources: • SAS documentation • SAS paper: Liu (2007) • www.lexjansen.com/pharmasug/2007/sp/sp05.pdf

13. Cross sectional and cohort data Not well portrayed as time series, per se… • Group comparisons • Separate plot for each group on the same time-series graph • Pre- and post-intervention change • e.g., Patient Activation Measure • Consider ways to make it a group comparison • Pre-intervention vs. post-intervention scores • Sample of individuals who did not receive intervention • Requires analytic support

14. What if we don’t see improvement? • Refer to the logic model • External factors • Challenged assumptions • What factors influenced the outcome? • What adjustments could have been made? • What other outcomes may have been measured?

15. Document what was learned • Regardless of success or failure • Context • What made the intervention successful or unsuccessful in your setting/community?

16. Additional resources • Toolkit – measurement http://www.cfmc.org/caretransitions/toolkit_measure.htm • Runcharts: Perla, Provost & Murray (2011) http://qualitysafety.bmj.com/content/20/1/46.full.pdf • ICPCA NCC contact: Tom Ventura tventura@coqio.sdps.org 303-784-5766

17. Questions? CO-ICPCTechnical@coqio.sdps.org The ICPC National Coordinating Center – www.cfmc.org/caretransitions Change Starts Here.