Evaluating and Selecting Effective Measures _____________ Session # 6 Jimmy Anthony

1. Evaluating and Selecting • Effective Measures • _____________ • Session # 6 • Jimmy Anthony • Senior Director, Delta Dental of Arkansas

2. Learning Objective(s) Participants will gain knowledge in: • Purpose of setting measures • Types of measures and indicators • Evaluating measures given the endless possibilities of measurable data. • Role of reporting and data • Things to think about looking forward

3. Disclosure and Conflict of Interest Declaration I declare that I have a financial interest/arrangement or affiliation with the corporate organization offering financial support or grant monies for this continuing dental education program, or I do have a financial interest in any commercial product(s) or service(s) I will discuss in the presentation.

4. Purpose of setting measures System of measurement that includes: • An item being measured (what) • A method of measurement (how) • The valued associated with the metric (why) • Measures are all around us • Measure change • Drives insights and questions • Identify needs • Measure performance against evidence-based standards

5. Types of Measures Operational performance measure • Key measures used to measure operational performance, such as turn-around times completion percentage rates Process measure • Measures and assesses the steps in a process that lead to an outcome Access measure • Provides insight into the timely use of services to achieve the best outcomes Utilization measure • The outcome of the interaction between health professionals and patients. Also known as the quantification of the use of services and site where care is provided Outcome measure • Patient centered, high-level outcomes of particular importance

6. Evaluating Measures • Measures and indicators are not the same thing • How to find the “right” one? • Start with the end in mind, with specific objectives • Be aware of alternatives • Know and target drivers of change/outcomes • Understand weaknesses • Be aware of your data and system constraints • More is not better • The hardest work should always be in the beginning • Pre-determined measures may be available, but may not align with your program • Clarity is key

7. Role of Reporting and Data • A metric is only as good as the data behind it • System constraints are a key challenge • Reporting definitions should be created when the metric is developed • Data collection methods should be planned when measures are developed

8. Looking forward • Multiple organizations are taking deeper dives into dental, the future is exciting. • Understanding how different groups arrive at recommended measures is key, paying special attention to: • The goal of the measure, and how it aligns with your specific program or project’s objectives • The historical and current state of your program • Your current system and/or data constraints • Key nuances of your program not contemplated by the recommended measure • Unintended consequences of implementing the measure

9. Questions

10. Jimmy Anthony Jimmy has worked in the dental health services industry for 12 years, spending time in a myriad of functions. His first 9 years were in financial services, working in underwriting and data analytics. During this time Jimmy developed a broad and deep expertise in evaluating and analyzing risk, developing measures to ensure accurate measurement of the health and status of key programs, and how to effectively use data to understand and evaluate overall performance of key components in the oral healthcare delivery system. Jimmy has spent the last 3 years dedicated to Delta Dental of Arkansas’s Medicaid Dental Managed Care program, Initially working to evaluate and respond to the State’s Request for Proposal, then following through with implementation and ongoing management of the program’s day-to-day operations. Serving in these capacities gives him a unique understanding of how to effectively use measures to evaluate the health and status of a key objectives.

11. Contact Information Jimmy Anthony Senior Director, Delta Dental Smiles Operations Delta Dental of Arkansas 1513 Country Club Road Sherwood, AR 72120 janthony@deltadentalar.com (501) 993-6530

12. Performance and Quality Measurement • Understanding Difference and       Selecting the Right Measures • _______________ • Session # 6 • DeDe Davis • MCNA

13. Learning Objective(s) Participants will gain knowledge in: • Criticality of early involvement towards measure design and goal setting • Understanding of the use of clinical practice guidelines in measurement • When and how there is opportunity to voice concerns

14. Disclosure and Conflict of Interest Declaration I declare that neither I nor any member of my family have a financial arrangement or affiliation with any corporate organization offering financial support or grant monies for this continuing dental education program, nor do I have a financial interest in any commercial product(s) or service(s) I will discuss in the presentation.

15. Background to the Ripple Effect • In 2010 the CMS Oral Health Initiative established two performance goals, to be accomplished over five years by FFY 2015: • Goal #1 – Increase by 10 percentage points the proportion of Medicaid and CHIP children ages 1 to 20 (enrolled for at least 90 days) who receive a preventive dental service. • Goal #2 – Increase by 10 percentage points the proportion of Medicaid and CHIP children ages 6 to 9 (enrolled for at least 90 days) who receive a sealant on a permanent molar tooth. • Simultaneously in 2010, CMS made a public commitment to work along side provider organizations such as the Dental Quality Alliance (DQA) to develop additional oral health measures which would later be included in the Medicaid and CHIP Child Core Set.

16. State Action Plans • CMS’ Oral Health Initiative called for action plans from states outlining how they would achieve the goals • States built in a variety of intervention strategies some of which included: • Expanded school based programs • Collaboration with medical providers • Member outreach initiatives • Financial incentives or penalties at the plan and/or provider level

17. Public Progress Reports • Utilizing the CMS 416 results, the American Dental Association’s Health Policy Institute released a 2015 report on oral health which included the following statement: • In 2013, 14 percent of Medicaid children ages 6 through 14 in the United States received a dental sealant on a permanent molar. This ranges from 6 percent in Ohio to 24 percent in Massachusetts.

18. Introduction of the DQA Sealant Measure • The DQA Sealant Measures (6-9 and 10-14 year olds) are endorsed both by the American Dental Association (ADA) by the National Quality Forum (NQF) in 2014 • CMS announces the inclusion of the DQA’s Sealant Measure into the Medicaid and CHIP Child Core Measure Set beginning in 2015.

19. Issues Affecting Quality Improvement • While the DQA and CMS 416 Sealant Measures had variations in continuous enrollment, enrollee risk, and tooth ID requirements, they both presented like issues with measure design that made it difficult to demonstrate improvement: • Both measures maintained children in the denominator year over year regardless if all eligible molars had been previously sealed, restored, or extracted. • Neither of the measures allow for a hybrid option in which plans could collect and report clinical data from the dental record that might not be included in claims data, i.e. sealants placed in school based programs, placed prior to the child being enrolled, or paid for by another primary carrier.

20. How big is the problem? DQA Denominator Analysis – TX Medicaid Ages 6-9 Benefit Eligible – Numerator Non-Compliant DQA – Numerator Non-Compliant Not Benefit Eligible – All Four Molars Previously Sealed, Restored, Or Extracted Not Benefit Eligible – All Four Molars Previously Sealed DQA – Numerator Compliant

21. Individual State Negotiations • MCNA and other vendors were forced into negotiations with states to allow for alterations in the measure denominator that would remove kids without any eligible teeth to seal. • State reactions varied in 2017: • Louisiana – adjusted CMS 416 specifications for required performance improvement project but maintained the measure as designed for contractual goals with financial penalties. • Texas – included the DQA measure in Pay for Quality goals with financial risk stating if the measure design was good enough for the NQF they saw no reason to deviate. • Florida – maintained measure specificity citing the DQA’s measure inclusion in the CMS Child Core Set

22. Approach the DQA with Concerns • Throughout the Fall and Winter of 2017 numerous conference calls were held with the DQA to voice payor concerns with using the DQA measure for pay for quality or other quality improvement activities. • Data and analysis were presented both by MCNA and DentaQuest to demonstrate the impact and need for change. • The DQA’s measurement subcommittee voted not to make changes to the measurement design given its current endorsement with the NQF.

23. NQF Annual Evaluation • The DQA’s Sealant Measures were up for annual evaluation in 2018. In February, MCNA submitted public comment to the NQF citing the denominator concerns and lack of consistency with clinical practice guidelines. • The NQF was not able to come to a consensus on maintaining the measure’s endorsement and requested additional information from the DQA to support the measure’s validity. • The DQA did not make change in the measure and their responses to the NQF were not significant enough to support the measure passing the validity testing and as a result it has since lost its endorsement.

24. DQA Annual Evaluation • The DQA conducted its 2018 Annual Evaluation of measures and upon the recommendation of its measurement subcommittee they voted to maintain the measure in its original form citing the adoption by CMS in the Child Core Set. • The DQA was not willing to issue a statement that the measure should not be used for quality improvement and in reverse re-enforced that it was designed for quality improvement despite all evidence provided.

25. Where Are We Today? • CMS announced during the May Oral Health Technical Advisory Group that they would be retiring the DQA Sealant measure for 6-9 year olds. • The DQA has convened a workgroup for development of a new sealant measure and will vote for adoption at their June 14th meeting. • New measure is better but we still have opportunity for refinement with integration of data from the dental records. • The Journal of the American Dental Association (JADA) has accepted an article on the concerns with sealant measurement co-authored by members of a research grant that will be published in an upcoming issue.

26. A Look at Louisiana…What Does Your State Look Like? • The old measure gives the appearance of declining utilization in sealants. • The new measures demonstrate that the overall rate of children receiving sealants on their molars, including all four molars, is increasing year over year. • Both new measures demonstrate statistically significant improvement and at a rate with that originally targeted in CMS’ Oral Health Initiative.

27. Lessons Learned • State Medicaid and CHIP programs are under-reporting progress in oral health care. • Entities such as the NQF will listen when presented with challenging information such as clinical practice guidelines. • All measures go through an annual review process and public comment period – voices in numbers are loud!

28. Questions

29. DeDe Davis DeDe Davis is a proven executive with a diverse background in clinical and non-clinical health care operations across Medicaid, Medicare, and commercial lines of business. DeDe operates in partnership with the Chief Dental Officer to implement and oversee all quality improvement plan activities and ensures utilization program goals are met in all MCNA markets. DeDe performs expert analysis of member utilization data to identify areas for opportunity in quality improvement and designs targeted performance improvement projects (PIPs). She represents MCNA with state external quality review organizations, collaborating with them during their review of MCNA's performance improvement plans. DeDe also represents MCNA during meetings of the Dental Advisory Committee, collaborating with committee members to sharpen MCNA's focus on quality and continue to build strong relationships with network providers. DeDe has over 28 years of health care operations and quality improvement experience, including serving as Vice President of Health Plan Operations and Associate Vice President of Quality Improvement and Operations for Passport Health Plan and Passport Advantage. In 2015, she joined MCNA as Vice President of Dental Management and Quality Improvement to ensure we continue to provide innovative quality improvement initiatives.

30. Contact Information DeDe DavisVP, Dental Management and Quality ImprovementMCNA Dental Plans200 West Cypress Creek RoadSuite 500Fort Lauderdale, FL 33309Office: (800) 494-6262 ext. 361Email: dedavis@mcna.netWeb: www.mcna.net

31. Performance and Quality Measurement • South Carolina Tests New Measures • _______________ • Session # 6 • Amy Martin & Stephen Boucher • MUSC and SC Medicaid

32. Introduction Amy Martin, DrPH, MSPH Professor & Director Division of Population Oral Health Chair, Department of Stomatology James B. Edwards College of Dental Medicine Medical University of South Carolina Steve Boucher Senior Advisor to the Deputy Director of Health Programs Director of Managed Care Operations & Division of Dental Services SC Dept of Health & Human Services Additional Analytical Team Members: Gerta Ayers – SC Medicaid Heather Kirby – SC Medicaid Abby Kelly – MUSC Chris Veshusio – MUSC Amah Riley - MUSC

33. Background Benefit Environment • South Carolina Medicaid’s dental benefit structure is fee-for-service • Claim processing, utilization management & provider training & outreach Program Imperative • The Duke Endowment’s Comprehensive School-Based Oral Health Program Expansion Initiative • Great opportunity to explore (a) how pediatric dental care might be reimbursed differently to reward evidence-based prevention in population-based settings; and (b) how to operationalize measures of value.

34. First Draft of Value-Measures in School-Based Settings • ER Visits • Sealants • Preventive Service Utilization • Categorical Risk Improvements

35. Sealant Measures Public Health Surveillance • Basic Screening Survey, National Oral Health Surveillance System • All states that participate in CDC Health Plan Performance Assessment • Evaluation of contract • Not relevant in all states Quality of Provider Care (Adherence to Standard of Care) • Applicable in all states (theoretically) • Measure of provider performance

36. Assumptions Assumption 1: • Must use validated data as that is (a) accessible, (b) affordable, (c) validated/consistent across all states. (Medicaid claims data) Assumption 2: • Measuring provider performance controls for extraneous influences such as eligibility and health plan policies because provider behavior should be guided by standards of care, not reimbursement levels or eligibility. Assumption 3: • Measuring quality is assessing provider adherence to a standard of care. • “When presented with the opportunity to seal an ‘eligible’ tooth, did the provider deliver services in keeping with the clinical standard of care?” (Provider performance measure)

37. Design – Feasibility Assessment Purpose: Conduct feasibility assessment for sealant quality measure using single county data. • Step 1 – Develop definitions & inclusion criteria • Step 2 – Build a data dictionary that delineates claims codes for dataset construction • Step 3 – Document any data management issues • Step 4 – Create the study cohort • Step 5 – Run a descriptive analysis • Step 6 – Preliminary examination of face validity

39. “Test” County: Charleston • More than 400K residents • Nearly 20% less than 18 • Only 65% White alone • 1300 bridges

40. Definitions • A Dental Provider was identified by his/her rendering NPI. • Timeframe of inquiry is State Fiscal Year 2018. • Patient cohort was defined as kids aged 8 years at any point in time during the reporting period. The cohort only included children enrolled for all 12 months in SFY 2018. • prospective analysis was used to determine if a sealant was applied by the same dentist 3 months subsequent to providing a preventive dental visit. • Data Management Note: We ultimately had to look forward 9 months • Eligible teeth from patient cohort include first permanent molars: 3, 14, 19, and 30. Teeth previously filled, sealed or missing were excluded from the analysis. • Retrospective analysis was needed to determine tooth eligibility. • A Preventive Dental Visit was defined as any encounter that included prophylaxis, fluoride treatment, or an EPSDT visit. • If there was no evidence of sealant placement for eligible teeth during a preventive visit, we looked forward 9 months after that preventive visit for sealant placement during any visit (originally looked forward 3 months). • This should control for the design threat that either sealant placement wasn’t indicated at the time of the preventive visit for any reason, including deferred restorative care.

41. Data Dictionary

42. Data Dictionary • In cases where kids saw more than one dentist for preventive care, their eligible teeth were counted under each dentist. • In cases where kids saw the same dentist more than once for preventive care, eligible teeth were counted on the last preventive visit of the fiscal year.

43. Face Validity? Medicaid Claims Data SC Public Health Surveillance Data Prevalence Measure 3rd graders in six Schools (8 and 9 year olds) Charleston County residents 36% of cohort got sealants • Incidence Measure • 8 year olds enrolled in Medicaid • Charleston County residents • 23% of cohort got sealants

44. Data Observations • 75 unduplicated dentists that provided a preventive service to 8 year olds with 12 months of enrollment. • 89% of 8 year olds with a preventive service had 12 months of enrollment in the fiscal year. • Of 3,464 teeth identified as not eligible, 42.3% were marked not eligible at age 6 and 46% were marked not eligible at age 7. • Reminder – not eligible means previously filled, sealed, or extracted • 11% of eligible teeth were sealed. • When sealants were applied to eligible teeth, 81% were applied within 3 months, 85% were applied within 6 months, and 97% were applied within 9 months. • Why? Potential focus group with DMDs who fell in the 6 to 9 month cohort.

45. Output by Dentist

46. Distribution of Individual Dentists by % Eligible Teeth Sealed

47. Descriptive Analysis 9% of DMDs sealed 40%+ of eligible teeth when presenting for care

48. Next Analytical Steps • Improve validation with better comparison to public health surveillance data • Linked to Medicaid claims data • Develop comparison group (8 year olds, Medicaid, CC) • Run the code for all 46 counties • Aggregate data run at state-level

49. Next Operational Steps COMMUNICATION: • Facilitate discussions with key stakeholder groups on what the threshold of ‘quality’ is and state-level goals for dental providers PERFORMANCE REPORT CARDS: • Publish provider performance reports using peer group comparisons on prevention-based standards of care EDUCATION: • Educate low performing providers on evidence-based preventive standards of care • Educate parents on the importance of sealants (drive up demand for sealants) POLICY: • Explore reimbursement policies that incentivize evidence-based preventive standards of care…including upside & downside risk

50. Thank you!Questions?