1. Performance Reports Andy Bindman MD Department of Medicine, Epidemiology and Biostatistics UCSF

2. Calculating an Individual’s Risk • Solve the multivariate model incorporating an individual’s specific characteristics • For continuous outcomes the predicted values are the expected values • For dichotomous outcomes the sum of the derived predictor variables produces a “logit” which can be algebraically converted to a probability (enat log odds /1 + enat log odds )

3. Aggregating to the group level • Sum observed events (eg deaths) for sub-group • Sum expected probability of events for same • Probabilities used to calculate expected events derived from entire data set and applied to individuals in sub-group (eg defined by care site) • The overall expected number of events must equal the observed number of events but this need not be the case at the level of subgroups

4. Comparing observed and expected outcomes to assess quality • Observed events or rates of events • Expected events or rates of events • Better quality implied when observed is lower than expected (worse quality when observed higher than expected)

5. Risk adjusted rates • Standardizes rates across sub-groups so that they can be directly compared with a single number • Observed rate/expected rate of subgroup x overall observed rate

6. Observed CABG Mortality Rates, NY 1989-1992

7. Calculating Expected CABG Mortality Rates in New York by Year • Pool all 4 years of CABG patients • Develop risk adjustment model for CABG patients • Apply risk adjustment model for CABG patients sub-grouped by year to determine expected number of deaths for each year • Divide expected number of deaths by number of cases per year to get expected death rate

8. Observed and Expected CABG Mortality Rates, NY 1989-1992

9. Annual Risk Adjusted Mortality Rate for CABG in New York • Observed rate per year/expected rate per year X average death rate over 4 year period (3.1)

10. Observed, Expected and Risk-Adjusted CABG Mortality Rates, NY 1989-1992

11. What Happened with CABG Surgery Over Time in New York? • Operated on sicker patients • Observed mortality rate declined over time • Risk adjusted mortality rate declined even more • Did quality of CABG care improve over time?

12. NY CABG Risk Adjustment Model • Well designed model C index = .787; Hosmer-Lemeshow chi square p=.16 • Mortality is not a subjective outcome- hard to fake • Gaming might be possible with coding some predictors

13. Interpreting Risk Adjusted CABG Outcomes • Public reporting on hospital CABG mortality began in 1989 • Low volume hospitals had higher mortality rates and some stopped performing CABG over time • Process indicators of cardiac care (beta blocker post MI) also improved in NY hospitals over time • Hospitals documented more co-morbidities over time resulting in inflated expected death rates • Some sick NY cardiac patients operated on in NJ

14. Applying Results To Providers • Possible to aggregate observed and expected rates of events to hospital, physician, or some other provider level grouping • Statistical problems arise when total number of expected events are small • Minimum of five expected events per group as a rule of thumb

15. Naming Names • Assigning assessments of quality to specific providers increases the stakes • Need to demonstrate validity of analytic approach

16. Reporting Results • Public reporting vs internal quality improvement • Data users tend to want gradations of quality along a continuum (excellent to poor) • However denoting those within a 95% confidence interval of the expected as average is less sensitive to noise in data

17. Bootstrap Procedure:Deriving Confidence Intervals • Multiple (e.g. 1000) random samples of same size of original derived from original sample with replacement • Calculate expected rate for each “new” sample • Create frequency distribution of expected rates • Empirically derive 95% CI (950 of 1000 centered around mean)

19. Consistency in the evidence • Differences between observed and expected may be due to things other than ‘quality’ • Are the results consistent over time • Are results consistent with prior expectations such as volume-outcome relationships • Confirmation through very different types of evidence is a major goal- “external validation”

20. Observed / Expected Mortality

21. Volume-Outcome • Relationship between high volume providers and better outcomes • Most often studied in relationship to procedures • Consistent with notion that practice makes perfect

22. Hospital Volume and CABG Mortality in California Hospitals Using Registry, 2000-02

23. External validation of data • Link hospital discharge data with CABG registry data • Looking for missing cases, deaths and highly predictive risk factors • 221 cases in discharge data not reported to registry • 26 additional deaths (498 total) • 63 undercodes and 123 overcodes of cardiogenic shock • 29 overcodes of salvage (51 total) • Direct auditing • Deaths • Highly weighted predictors particularly if subjective

24. Growing Number of Quality Initiatives Provide Opportunity for Cross Comparisons • AHRQ Quality Indicators • JCAHO ORYX Hospital Core Performance • CMS Hospital Quality Alliance • National Quality Forum (NQF) • Leapfrog • NCQA HEDIS

25. JCAHO - Hospital AMI indicators Process elements: Aspirin at arrival, Aspirin at discharge, ACEI for LVSD, Smoking cessation counseling, Beta-blocker at discharge, Beta-blocker at arrival, Thrombolytic within 30 minutes, PCI within 120 minutes

26. Who uses these reports and how • Patients • Slow to catch on • More important for those without other ways to judge quality • Managers • Aim to improve quality to avoid “naming and shaming” • Payers (health plans) • Selective contracting • Pay for performance

27. How much do these reports matter? • California has lowered isolated CABG mortality by ~1% (from 3% to 2 %) during public report period • Approximately 20,000 procedures per year • Reduction from 600 to 400 deaths • Average survival ~5 years • Even if half the change is due to gaming, 500 life years saved

28. My Reflections on Performance Reports • View the risk adjusted estimates as ‘yellow flags’, not ‘smoking guns’ • Risk models will probably improve with ability to add more clinical data available through electronic records • Research may not need to be perfect to bring about public health benefits • Attempts to improve quality need to consider unintended consequences on access/efficiency

29. Outcomes Research Opportunities • Validate risk adjustment models for new conditions • Are health care outcomes changing over time and if so why? • How can performance reports on health outcomes be used to create better health care quality?