Kathryn Andolsek MD MPH Kathryn.andolsek@duke.edu

2. Clinical Competency Committees: Transitioning our Learners in GME2013 Graduate Medical Education ConferencePredicting Effective Transitions to the NAS Palm Desert, CA – October 24, 2013 Kathryn Andolsek MD MPH Kathryn.andolsek@duke.edu

3. An Institutional Question? Are trainees at Loma Linda achieving the competency required for unsupervised practice that includes the delivery of safe, timely, equitable, effective and patient-centered care?

4. “So how do you know?”

5. Objectives • Optimize your CCC for learners, your program and institution • Use evidence from medical education research to make best choices for CCC • Conduct a gap analysis and determine your Next Steps to strengthen your CCC

7. Competency Based Education

8. Dreyfus and Dreyfus Model Expert/ Master Proficient Competent Advanced Beginner Novice Time, Practice, Experience Dreyfus SE and Dreyfus HL. A 1980 Carraccio CL et al. Acad Med 2008;83:761-7

9. Fully Competent A B C Finish PGY3 Start PGY1 Milestones and Trajectories “D” “E” D A’ E Lucey and Boote Milestones

10. Clinical Competency Committee (CCC) • A group of faculty (and others) trained in knowing learner expectations and the understanding of evaluations • Uniformly monitor trainee progression through the continuum of education using objective measures • assist the PD with the assessment of competency, need for remediation, and feedback

12. Participation CCC Quiz

13. Accreditation Council for Graduate Medical Education (ACGME) Requirements/Guidance Proposed Common Program Requirements Section V Comments accepted thru May 2013 Access at http://www.acgme.org/acgmeweb/Portals/0/PFAssets/ProgramRequirements/CPR_Impact.pdf Frequency Asked Questions Published December 2012 Access at http://www.acgme-nas.org/assets/pdf/NASFAQs.pdf FAQ Addresses: • Role of CCC in resident assessment • Qualifications for CCC membership • Preparation for CCC members • Milestones and the CCC • CCC and Remediation New CPR 2016 http://www.acgme.org/acgmeweb/Portals/0/PFAssets/ProgramRequirements/CPRs_07012016_TCC.pdf ACGME NAS Webinar

14. 1. The ACGME expects each program to form a CCC and develop its’ members by: • 2013 • 2014 • When the specialty begins to require milestones • The time of the first C.L.E.R. visit • Before the next Program Site visit • Before the next “Internal Review” • None of the above

15. 1. The ACGME expects each program to form a CCC and develop its’ members by: • 2013 • 2014 • When the specialty begins to require milestones • The time of the first C.L.E.R. visit • Before the next Program Site visit • Before the next “Internal Review” • None of the above

16. CCC implementation required by….. Proposed CPR Impact Statement: “The proposed effective date for these revisions is July 1, 2013. However, programs in Phase II of NAS will not be expected to be in full compliance with new requirement until July 1, 2014.” FAQ: ACGME Programs expected to form a CCC by June 2013 FAQ Addresses: • Role of CCC in resident assessment • Qualifications for CCC membership • Preparation for CCC members • Milestones and the CCC • CCC and Remediation Accessed February 20th 2013 at http://www.acgme-nas.org/assets/pdf/NASFAQs.pdf

17. 2. The minimum number of CCC members should be • 2 • 3 • 4 • 5

18. 2. The minimum number of CCC members should be • 2 • 3 • 4 • 5

19. Common Program Requirements 7-1-2013

20. http://www.acgme.org/acgmeweb/Portals/0/PDFs/FAQ/CCC_PEC_FAQs.pdfhttp://www.acgme.org/acgmeweb/Portals/0/PDFs/FAQ/CCC_PEC_FAQs.pdf

21. 3. The CCC members…. • Vote on each resident’s performance • Provide a consensus on each resident • Only consider residents who need remediation plans • Only review residents in their last year of training • None of the above

22. 3. The CCC members…. • Vote on each resident’s performance • Provide a consensus on each resident • Only consider residents who needs remediation plans • Only review residents in their last year of training • None of the above

23. Common Program Requirements 7-1-2013

24. Common Program Requirements 7-1-2016

25. Role of CCC in resident assessment • Members make consensus decision on the progress of each resident • CCC utilizes: • Existing resident assessment data • Faculty observations • Data from milestone assessment (beginning July 2013) Accessed February 20th 2013 at http://www.acgme-nas.org/assets/pdf/NASFAQs.pdf

26. 4. The CCC must include: • Patients • Nurses • Peer selected residents • Core faculty members • Program director • All of the above • C, D, and E

27. 4. The CCC must include: • Patients • Nurses • Peer selected residents • Core faculty members • Program director • All of the above • C, D, and E

28. Common Program Requirements 7-1-2013

29. Qualifications for CCC members • faculty members • Those who observe and evaluate residents; willing to make hard decisions • Others such as assessment specialists, non-MD educators, resident (?) • Webinar: Advisors/mentors excluded from discussion

30. http://www.acgme.org/acgmeweb/Portals/0/PDFs/FAQ/CCC_PEC_FAQs.pdf 7-19-2013

31. 5. According to the ACGME, faculty development needed by the CCC members includes: • Knowing their potential legal liability • Giving “bad news” to the resident after the milestone determination has been reached • Reaching a common agreement of milestones narrative meaning • Determining how many assessments are needed for any given milestones • Applying QI improvement principles to the evaluation process • Knowing the best remediation strategies for certain milestone suboptimal performance • All of the above • None of the above

32. 5. According to the ACGME, faculty development needed by the CCC members includes: • Knowing their potential legal liability • Giving “bad news” to the resident after the milestone determination has been reached • Reaching a common agreement of milestones narrative meaning • Determining how many assessments are needed for any given milestones • Applying QI improvement principles to the evaluation process • Knowing the best remediation strategies for certain milestone suboptimal performance • All of the above • None of the above

33. Faculty Development for CCC members(…plans to develop training resources for CCCs) • Evaluation is a core faculty competency but most will need additional training in evaluation process • Discuss milestone narratives & reach common agreement of meaning • Evaluation process training: • How to aggregate data • How to interpret evaluation data • Venue for faculty member discussion about resident evaluation: • How many assessment needed for each milestone • Data quality • Application ofQI principles as part of evaluation process Accessed February 20th, 2013 at http://www.acgme-nas.org/assets/pdf/NASFAQs.pdf

34. 6. A resident rotates on another specialty service. That specialist evaluates them as performing poorly. The CCC should…. • Use the grade provided by the specialist as their grade in assessing resident performance against the milestones • Not consider the evaluation as it came from a different specialty than the program • Take the evaluation and apply it with other data to the resident’s program milestones • Vote whether the evaluation seems accurate and should be included in the overall review of the resident’s performance

35. 6. A resident rotates on another specialty service. That specialist evaluates them as performing poorly. The CCC should…. • Use the grade provided by the specialist as their grade in assessing resident performance against the milestones • Not consider the evaluation as it came from a different specialty than the program • Take the evaluation and apply it with other data to the resident’s program milestones • Vote whether the evaluation seems accurate and should be included in the overall review of the resident’s performance

36. Milestones and the CCC • CCC will take data from evaluations and apply them to the milestones to mark progress of residents. • The CCC will have the advantage of knowing how each of the specialists evaluated residents and can apply that knowledge as it marks the residents’ progress on the milestones. Accessed February 20th 2013 at http://www.acgme-nas.org/assets/pdf/NASFAQs.pdf

37. 7. Pilot assessments on the milestones have found that the first time the evaluation is done, it takes approximately _____for each resident. • 5 minutes • 10 minutes • 20 minutes • 30 minutes • 45 minutes • Up to an hour

38. 7. Pilot assessments on the milestones have found that the first time the evaluation is done, it takes approximately ____ for each resident. • 5 minutes • 10 minutes • 20 minutes • 30 minutes • 45 minutes • Up to an hour

39. Time required: milestones and the CCC • It takes a significant amount of time (up to an hour) to conduct the milestones assessment for each resident the first time the evaluation is done. • Subsequent evaluations take less time. Accessed February 20th 2013 at http://www.acgme-nas.org/assets/pdf/NASFAQs.pdf

40. Remediation and the CCC • Goal: identify residents who are not progressing • CCC sets performance thresholds (until national benchmark data available) • For early years- compare resident performance on milestones to peers in program • When deficiency identified, interventions considered (may or may not be role of CCC?) • Mentor • Additional readings • Skill lab experiences • Added rotations • Extend education • Counseling to consider another specialty/profession Accessed February 20th 2013 at http://www.acgme-nas.org/assets/pdf/NASFAQs.pdf

41. Benefits of CCC • For the resident • Offers insight and perspectives of a GROUP of faculty • Serves as an early warning system if resident fails to progress (opportunity for remediation) • For the program director • CCC outcomes shared with resident so Program Director can be advocate for resident learning • For the faculty/program • Utilizes objective assessment measures making it easier for evaluators • Chance to simplify assessment tools; evaluation system • Opportunities to improve programs - curriculum, rotation schedules, supervision, mentorship (educational QI)

42. Evidence based: What the literature says…. • Group conversations more likely to uncover deficiencies in professionalism for students(Hemmer et al 2000) • Group assessments improve inter-rater reliability and reduced range restriction in multiple domains (Thomas et al 2011) • CCC helps ensure decisions are not arbitrary or capricious AND serves as checks and balances (especially identifying marginal resident) (Wu et al 2010, Schwind et al 2004, Williams et al 2005) • No evidence (that) individuals dominate discussions or gang up (Williams et al 2005) • 18% of resident deficiencies requiring remediation only became apparent through group discussion; average discussion 5 minutes per resident, range 1-30 minutes (Schwind et al 2001) • Amount of time spent discussing a resident is frequently a measure of the severity of the problem/concern (Sanfrey et al 2012)

43. CCC: The Legal & Practical Legal precedent suggests Courts will support educational institutions decisions about resident progression and graduation when: • Decisions are reasonable (e.g. made at routine faculty meetings) • There is a shared understanding of expectations • Residents with performance issues are given Notice (informed of deficiency) and Opportunity to Cure (improve) (Missouri v. Horowitz 1978) • Decisions are made “conscientiously and with careful deliberation” • Taking into account learners entire record (Univ. of Michigan v. Ewing 1985) Are your CCC proceedings and documentation “peer review protected”?

44. CCC Components, Practical Processes & Procedures • Membership: How many members? Who is eligible/optimal? For how long? • Who will serve as the CCC Chair and for how long? • When, how often and how long should the CCC meet? • What is the role and responsibility for each member? • How will learner expectations be determined? What are the shared set of standards/thresholds? • What are objective (& varied) measures of performance? What assessments (existing or future) could be used? • How are findings determined? • How will findings be conveyed to Program Director? (who makes ultimate decision) • How/by whom will findings be conveyed to learners? • What happens when CCC determines learner is performing sub-optimally? • What faculty development is required for CCC members? • How will you describe this process to your program?

46. Pre-Conference Survey Results • Electronic survey link sent on February 19th • 123/270 responded (46%) • THANK YOU!

48. Think about…CCC Components, Processes & Procedures • Membership: How many members? Who is eligible/optimal? For how long?