Milestones: Are You Ready for July 1 st ?

1. Milestones: Are You Ready for July 1st ? Kirk Lalwani, MD, FRCA, MCR Associate Professor, Chair, Resident Evaluation and Clinical Competence Committee, Anesthesiology and Perioperative Medicine. On behalf of the AMIGOS (Anesthesiology Milestones Implementation Group Operatives)

2. Disclosure • No conflict of interest • Some of my slides are borrowed from the faculty educational resources on the ACGME website

3. Implementing Milestones Objectives: • Outline the Next Accreditation System (NAS) • Define a Milestone • Review our Milestones strategy • Discuss assessment of Anesthesiology Milestones

4. Abbreviations • NAS- Next Accreditation System • CLER- Clinical Learning Environment • ACGME- Accreditation Council for Graduate Medical Education • RRC- Residency Review Committee

5. The Next Accreditation System

6. Why ‘Next Accreditation System’ (NAS)? “Self-regulation is a fundamental professional responsibility, and the system for educating physicians answers to the public for the graduates it produces.” “The Next GME Accreditation System – Rationale and Benefits” Nasca T.J., Philibert I., Brigham T., Flynn T.C. N Engl J Med 2012; 366:1051-1056

7. Goals of NAS • Strengthen resident development in • Professionalism • Communication Skills • Systems Based Practice • Practice Based Learning • Enhance public accountability • More explicit definition of a good physician (Milestones) • Patient safety (Clinical Learning Environment Review Program) • Reduce burden required for accreditation

8. ‘Next Accreditation System’…in a Nutshell • Accreditation on the basis of educational outcomes • NAS : Advance from an episodic ` biopsy ’ model to annual data collection or ` continuous oversight ’. • RRCs will measure compliance through the evaluation of annual program data elements.

9. The Next Accreditation System Continuous Observations Promote Innovation Assess Program Improvement(s) Identify Opportunities for Improvement Program Makes Improvement(s)

10. The Next Accreditation System • We will now have annual data collection • Trends in annual data • Milestones • Resident and Faculty Surveys • Resident and Faculty Scholarly activities • Case Log Data • ITE / ABA Examination pass rates

11. Clinical Learning Environment Review Visits • Oversight of • Transitions of Care • Duty Hours Policy • Fatigue Management and Mitigation • Education and Monitoring of Professionalism • Involvement in Institutional Quality Improvement and Safety initiatives • Supervision policies Additional information available at: http://www.acgme-nas.org/CLER

12. Milestones

13. What are Milestones? Specific behaviors, attributes, or outcomes in the six general competency domains to be demonstrated by residents during residency. (ACGME and Specialty Boards) Skill and knowledge-based development that commonly occurs by a specific time.

14. Why ???? • ACGME requires it • Program Accreditation

15. Who, When, and How???? • All residents in all ACGME-accredited programs • Subspecialty fellows from July 2015 • New assessment methodology as well as existing tools • 6-monthly reporting of Milestones to ACGME in January and July • Reporting starts July 2014

16. Expected Benefits of Milestone Assessments For Residents • Clarify expectations • Feedback should identify specific areas to work on • Earlier identification of under-performers • Aspirational goals for residents who exceed expectations

17. Expected Benefits of Milestone Assessment For the Program • Curriculum development • Accreditation requirements • Earlier identification of under-performers For the Public • Better definition of what a physician can do at the completion of training • Program is accountable to a common standard • Possible use for board certification

18. Milestones Development Strategy Pragmatic Program implements new milestone aligned tools, which help residents understand expectations Comprehensive Program changes assessment system, invests time in faculty development and education to improve rater expertise and CCC validity Minimalist CCC assigns milestone levels using existing tools

19. Process: June 2013

20. General Strategy“Good now is better than perfect later….” • Priorities • Ease of use • Should not be burdensome for residents, faculty, RECCC or education office staff • No PAPER ! • App for ‘on-the-go’ assessments - procedures, H&P • Implementation feasibility • Ready by July 2014 • Integrate with existing system (Evalue) • Ensuring adequate data to assess Milestones comprehensively • A Milestone for every competency • 17/25 Milestones chosen • Work in progress, as refinements will be made and new tools added over time

21. Which? Which Milestones did the AMIGOS select?

22. Medical Knowledge (MK): 1 • MK1- Knowledge of biomedical, clinical, epidemiological, and social sciences as outlined in the American Board of Anesthesiology Content Outline • ITE • AKT • ABA Basic Examination • ACLS certification • Direct Clinical Observation

23. Patient Care (PC): 8 of 10 • PC1-Preanesthetic Evaluation, Assessment, and Preparation • PC2- Anesthetic choice and conduct • PC3- Periprocedural pain management • PC4- Management of perianesthetic complications • PC5- Crisis management • PC6- Triage and management of critically ill patient in a non-operative setting • PC7- Acute, chronic, and cancer related pain consultation and management • PC8- Technical skills: Airway management • PC9- Technical skills: Monitoring and Equipment • PC10-Technical skills: Regional anesthesia

24. Example of PC-8 Assessment • PC-8 (Technical skills: Airway management) • Direct Clinical Observation (DCO) • Daily evaluation form • Airway management checklist • Objective Structured Clinical Examination (OSCE) / Objective Structured Assessment Test (OSAT) • Simulation / Standardized Patient (SP)

25. Professionalism (Prof): 3 of 5 • Prof1- Responsibility to patients, families, and society • Prof2- Honesty, integrity, and ethical behavior • Prof 3- Commitment to institution, department and colleagues • Prof4- Receiving and giving feedback • Prof 5- Responsibility to maintain personal, emotional, physical and mental health

26. Example of Prof-1 Assessment • Prof 1 (Responsibility to Patient, Family, and Society) • 360 degree evaluations • Patients • Healthcare team members • Peer evaluations • Administrative staff • Direct Clinical Observation (DCO) • Daily evaluation form • Written comments

27. Interpersonal and Communication Skills (ICS): 2 of 3 • ICS1- Communication with patients and families • ICS2- Communication with other professionals • ICS3- Team and Leadership skills

28. Practice Based Learning (PBL): 2 of 4 • PBL1 Incorporates quality improvement and patient safety initiatives into personal practice • PBL2- Analysis of practice to identify areas in need of improvement • PBL3- Self-directed learning • PBL4- Education of patient, family, students, residents and other health professionals

29. Systems Based Practice (SBP) 1 of 2 • SBP1- Systems-based approaches to patient care • SBP2- Coordination of patient care within the healthcare system

30. How will we Assess Milestone Levels? • Milestones are a summary of how a resident is progressing based on information from other sources • Milestones are not assessment tools to be used for gathering detailed information • They do not replace end-of-rotation forms, simulation, 360’s, Daily evaluations etc. • More pieces of data allow for more precision • There will be a minimum number of evaluations required for procedural assessments

31. Clinical Competency Committee (RECCC) Self Evaluations Case Logs Mock Orals End-of-Rotation Evaluations Professionalism Indicators like pharmacy, billing, evals etc Unsolicited Comments ITE Sim Lab Student Evaluations Nursing and Ancillary Personnel Evaluations Clinical Workplace Evaluations Clinical Competency Committee OSCE Patient/ Family Evaluations Peer Evaluations Assessment of Milestones

33. Anesthesiology Milestones • Level 4 is the Graduation Target • Not every resident will achieve Level 4 in every Milestone • Residents required to substantially meet most Milestones at Level 4 • Residents are not expected to achieve Level 5 during residency • Residents may achieve a level of competency in specific Milestones sooner than expected

34. Milestone Template

35. General Competency Subcompetency Developmental Progression or Set of Milestones Milestone

37. Instruments to Measure Milestones • Daily Evaluation Form for OR (Evalue) • Daily Evaluation Form for ICU (Evalue) • Daily Evaluation Form for Pain (Evalue) • Technical Skills forms (Evalue) (‘App’?) • Arterial line • CVP Line • Nerve Block / Epidural / Caudal • Airway Management • Peer evaluations • Assessment of Patient Interaction • Regional assessments • Simulation assessments

38. ‘QUASAR’ Project: Quality in Anesthesiology, Systems Assessment and Research

39. Resident ‘QUASAR’ Project: Quality in Anesthesiology, Systems Assessment and Research • Quality Improvement • Systems Based Practice • Practice-Based Learning • CA1- Background research, identify area of study • CA2- Identify quality measure to be studied and change to be implemented, study effects, refine • CA3- Write summary, submit abstract to national conference, or write a manuscript for publication

40. Summary • Next logical extension of competency-based evaluation • Mostly familiar, not burdensome • Levels 1 and 2 do NOT signify a ‘Fail’ grade • Evaluations will be more specific and focused • Direct clinical observation to evaluate procedures and patient interactions

41. We Need Your Data ! APOM NEEDS YOU

42. Acknowledgements • AMIGOS • Nicole Conrad • Oumou Diallo • Judy Freeman • Julio Gonzalez • Karen Hand • Izumi Harukuni • Amy Miller Juve • Diana Kim • Ed Kahl • Dean Lao • David Larsen • Kim Mauer • Michele Noles • Annie Riley • Peter Schulman • Katie Seligman • Chris Swide • Linda Wylie • David Wilson • Glenn Woodworth