Implementing the ACGME Outcome Project: Questions, Answers, Tips, and Traps

1. Implementing the ACGME Outcome Project:Questions, Answers, Tips, and Traps Doris A. Stoll, PhD RRC Executive Director

2. 2 Learning Objectives Recognize the value of outcomes-based education as a viable tool for improving GME Develop learning objectives and assessment tools for four of the general competencies Identify key challenges in implementing Outcome Project-related changes in their programs Identify resources available for assistance in implementing Outcome Project-related changes

3. 3 What is the ACGME? The Accreditation Council for Graduate Medical Education is an independent, private sector, voluntary, not-for-profit organization responsible for evaluating and accrediting residency programs in the United States. Currently, when you prepare for a site visit, when you�re frantically preparing those Program Information Forms, it�s like an examination of your program conscience. The program director, faculty, and staff should be asking: [SELECT] Do we comply with the Requirements? Do we have everything in place so that we can complete those Program Information forms perfectly? [SELECT] In most cases, that means, do we have goals and objectives and an organized curriculum upon which our program is based? [SELECT] Lastly, you�re asked, among other things, if the program evaluates itself and its residents?Currently, when you prepare for a site visit, when you�re frantically preparing those Program Information Forms, it�s like an examination of your program conscience. The program director, faculty, and staff should be asking: [SELECT] Do we comply with the Requirements? Do we have everything in place so that we can complete those Program Information forms perfectly? [SELECT] In most cases, that means, do we have goals and objectives and an organized curriculum upon which our program is based? [SELECT] Lastly, you�re asked, among other things, if the program evaluates itself and its residents?

4. 4 The ACGME Mission To improve the quality of health care in the United States by ensuring and improving the quality of graduate medical educational experiences for physicians in training. Before we get into what�s changing, I think it�s important that we first review what isn�t changing, just what is the ACGME and what do we do? Our mission is no less than [READ mission from slide]. How do we fulfill this critical mission?Before we get into what�s changing, I think it�s important that we first review what isn�t changing, just what is the ACGME and what do we do? Our mission is no less than [READ mission from slide]. How do we fulfill this critical mission?

5. 5 The current problem Increasing public concerns with quality and safety. Variable patterns of care that are not based on medical science. Poor quality of interpersonal �service.� Public encounters difficulty in assessing competence (initial and continuing ) and judging quality.

6. 6 The current mindset The SIMPLE questions: Does the program comply with the written Requirements? Does the program have established goals and objectives and an organized curriculum? Does the program have a process to evaluate its residents and itself?

7. 7 A new way of thinking The COMPLEX questions: Do the residents achieve the learning objectives set by the program? What evidence can the program provide to demonstrate that they do so? How does the program demonstrate continuous improvement in its educational processes?

8. In other words� How well do we learn what is being taught�

9. �how well do we practice what we learn?

10. 10 A new way of thinking Never confuse activity with productivity. Dee Hock But the new way of thinking, that changed mindset, has a different set of questions. While the RRC will still want to know if the program has goals and objectives, now [SELECT] the new question is whether or not the residents achieve the objectives? Remember, can �we see what we�re saying?� And it�s not enough for the program director to check �yes,� on a Program Information Form. [SELECT] The program will have to show evidence that goals and objectives are being achieved. And finally, [SELECT] how does the program use the results of its assessments to make improvements? Of course, there will still be requirements with which you must comply, but now, the purpose for that compliance should be much more important than simply whether everything is in place.But the new way of thinking, that changed mindset, has a different set of questions. While the RRC will still want to know if the program has goals and objectives, now [SELECT] the new question is whether or not the residents achieve the objectives? Remember, can �we see what we�re saying?� And it�s not enough for the program director to check �yes,� on a Program Information Form. [SELECT] The program will have to show evidence that goals and objectives are being achieved. And finally, [SELECT] how does the program use the results of its assessments to make improvements? Of course, there will still be requirements with which you must comply, but now, the purpose for that compliance should be much more important than simply whether everything is in place.

11. 11 A new way of thinking But the new way of thinking, that changed mindset, has a different set of questions. While the RRC will still want to know if the program has goals and objectives, now [SELECT] the new question is whether or not the residents achieve the objectives? Remember, can �we see what we�re saying?� And it�s not enough for the program director to check �yes,� on a Program Information Form. [SELECT] The program will have to show evidence that goals and objectives are being achieved. And finally, [SELECT] how does the program use the results of its assessments to make improvements? Of course, there will still be requirements with which you must comply, but now, the purpose for that compliance should be much more important than simply whether everything is in place.But the new way of thinking, that changed mindset, has a different set of questions. While the RRC will still want to know if the program has goals and objectives, now [SELECT] the new question is whether or not the residents achieve the objectives? Remember, can �we see what we�re saying?� And it�s not enough for the program director to check �yes,� on a Program Information Form. [SELECT] The program will have to show evidence that goals and objectives are being achieved. And finally, [SELECT] how does the program use the results of its assessments to make improvements? Of course, there will still be requirements with which you must comply, but now, the purpose for that compliance should be much more important than simply whether everything is in place.

12. 12 A new way of thinking How to make this transition to a competency based program for a community of: 724 sponsoring institutions 7,800 accredited programs 99,000 residents 27 Residency Review Committees covering 26 core specialties and 115 subspecialties Currently, when you prepare for a site visit, when you�re frantically preparing those Program Information Forms, it�s like an examination of your program conscience. The program director, faculty, and staff should be asking: [SELECT] Do we comply with the Requirements? Do we have everything in place so that we can complete those Program Information forms perfectly? [SELECT] In most cases, that means, do we have goals and objectives and an organized curriculum upon which our program is based? [SELECT] Lastly, you�re asked, among other things, if the program evaluates itself and its residents?Currently, when you prepare for a site visit, when you�re frantically preparing those Program Information Forms, it�s like an examination of your program conscience. The program director, faculty, and staff should be asking: [SELECT] Do we comply with the Requirements? Do we have everything in place so that we can complete those Program Information forms perfectly? [SELECT] In most cases, that means, do we have goals and objectives and an organized curriculum upon which our program is based? [SELECT] Lastly, you�re asked, among other things, if the program evaluates itself and its residents?

13. What is the value of a competency-based model of education? How can we implement the competencies? How can we assess the competencies?

14. 14 Project Principles Focus of general competencies that apply to all specialties Tendency to improve what we measure More flexibility to be creative More credibility for public accountability Use of the continuum of lifelong learning, e.g. ABMS maintenance of certification Focus on improvements in lieu of minimal thresholds

15. 15 WHAT? A long-term initiative To enhance residency education Through educational outcome assessment The Outcome Project is [SELECT] [READ slide]. Changes to a system as extensive as the one we just outlined, cannot and do not happen overnight. Full implementation is going to take years. The purpose of the Project is [SELECT] [READ slide], that is, we�re going to improve what we already do well, and in some cases, what we may not do as best as we can. How are we going to accomplish this improvement? [SELECT] We�re going to concentrate on assessing the outcomes of our efforts to educate residents. This may not strike you as new, but I assure you, it�s a different emphasis for us as the accreditors and possibly for you, the educators, as well. Let�s think about what this means.The Outcome Project is [SELECT] [READ slide]. Changes to a system as extensive as the one we just outlined, cannot and do not happen overnight. Full implementation is going to take years. The purpose of the Project is [SELECT] [READ slide], that is, we�re going to improve what we already do well, and in some cases, what we may not do as best as we can. How are we going to accomplish this improvement? [SELECT] We�re going to concentrate on assessing the outcomes of our efforts to educate residents. This may not strike you as new, but I assure you, it�s a different emphasis for us as the accreditors and possibly for you, the educators, as well. Let�s think about what this means.

16. 16 HOW? Project Activities Identifying what to measure Developing measurement tools Collaborating to find the answers The Project has three major activities: The first [SELECT] is to identify what it is that we�re going to measure. The second [SELECT] is to develop the tools that will help us measure what we�ve accomplished. And the third [SELECT] is to recognize that we can�t do this alone; we�re going to have to collaborate to find both what to measure and how to measure it. So, just what are we measuring?The Project has three major activities: The first [SELECT] is to identify what it is that we�re going to measure. The second [SELECT] is to develop the tools that will help us measure what we�ve accomplished. And the third [SELECT] is to recognize that we can�t do this alone; we�re going to have to collaborate to find both what to measure and how to measure it. So, just what are we measuring?

17. 17 Patient Care Compassionate Appropriate Effective For treatment of health problems For the promotion of health

18. 18 Medical Knowledge About established and evolving science Biomedical Clinical Cognate (epidemiological & social-behavioral) About application of this knowledge to patient care Ability to critically assess �new� knowledge

19. 19 Practice-based Learning and Improvement Investigation and evaluation of their own patient care Appraisal and assimilation of scientific evidence Improvements in patient care

20. 20 Interpersonal and Communication Skills Results in effective information exchange and teaming with Patients Their families Other health professionals Enhances therapeutic relationship

21. 21 Professionalism Manifested through A commitment to carrying out professional responsibilities Adherence to ethical principles Sensitivity to diverse patient populations

22. 22 Systems-Based Practice Manifested by Actions that demonstrate awareness of and responsiveness to larger context and system of health care Ability to effectively call on system resources to provide care that is of optimal value

23. 23 WHEN?

24. 24 Building a Support Network Shared experiences/interest groups National, regional and local conferences Poster sessions at Mastery Workshops RFP process � 70+ examples Web-based resource center for the competencies and their assessment

25. 25

26. 26 Assessment Tools (The Toolbox) 360� Evaluation Instrument Chart Stimulated Recall Oral Exam (CSR) Checklist Evaluation of Live or Recorded Performance Objective Structured Clinical Exam (OSCE) Procedure, Operative or Case Logs

27. 27 The Toolbox (continued) Patient Surveys Portfolios Record Review Simulations and Models Standardized Oral Exams Standardized Patients (SP) Written Exams (MCQ)

28. 28

29. 29

31. Education 101

32. Curriculum

34. 34 The Nuts and Bolts Learning Outcomes (educational goals) Learning Objectives

35. 35 Outcomes and Objectives Outcomes Can be described under a small number of headings Emphasize broad overview Knowledge and �metacompetencies� are embedded Objectives Are extensive and detailed Emphasize instructional intent at a lower and more detailed level Classified into discrete areas (knowledge, skills, attitudes)

36. 36 The Learning Outcome VII.C.4. Residents must be able to demonstrate interpersonal and communication skills that result in effective information exchange and teaming with other health care providers, patients, and patients' families.

37. 37 The Learning Objective Residents are expected to use effective listening skills and elicit and provide information using effective nonverbal, explanatory, questioning and writing skills.

38. 38 The Learning Objective Who�? Will do� How much�? Of what�? By when�?

39. 39 A Competency-based Objective At the completion of the PG-1 year, the resident will be able to diagnose and manage common ambulatory medical disorders, i.e., hypertension, diabetes, angina, COPD with minimal supervision.

40. 40 Levels of Cognition:Taxonomy of Knowledge

41. Skill Development Models

42. 42 Prototype Requirement Other important aspects of the requirement are that assessment results can and will be used to: achieve progressive improvement in residents� competence and performance to evaluate educational effectiveness of the program to make improvements in the program -Goal is for residents to attain the competencies and demonstrate them; goal is not for residents to have an opportunity to learn; the goal is for residents to learn, to attain competencies and to demonstrate them Major question now, what needs to be done and what is being done so that these requirements can be implemented?Other important aspects of the requirement are that assessment results can and will be used to: achieve progressive improvement in residents� competence and performance to evaluate educational effectiveness of the program to make improvements in the program -Goal is for residents to attain the competencies and demonstrate them; goal is not for residents to have an opportunity to learn; the goal is for residents to learn, to attain competencies and to demonstrate them Major question now, what needs to be done and what is being done so that these requirements can be implemented?

43. 43 Types of Evaluation Formative Improve performance Summative Note achievement

44. 44 Characteristics of good assessment Systematic Dependable Comprehensive Congruent Practical

45. 45 Characteristics of good assessment (continued) Makes professional practice more transparent Deconstructs the role of physician Clarifies levels of expertise by distinguishing functional levels

46. 46 Characteristics of good assessment (continued) Measures actual performance Identifies areas for improvement, i.e., self, others Satisfies reasonable requests for accountability

47. 47 Resources Where do we find them?

48. 48 How do we select the experiences to evaluate? Organize Categorize experiences broadly, from the simple to the complex Level required experiences

49. 49 How do we select the experiences to evaluate?(continued) Identify Diagnoses with high incidence/prevalence Cases with significant morbidity and mortality Diagnoses that are treatable and preventable Incidents where questionable management occurs Situations where improvement is needed and can be accomplished through improved education

50. 50 Assessment Tools (The Toolbox) 360� Evaluation Instrument Chart Stimulated Recall Oral Exam (CSR) Checklist Evaluation of Live or Recorded Performance Objective Structured Clinical Exam (OSCE) Procedure, Operative or Case Logs

51. 51 Assessment Tools (The Toolbox)(continued) Patient Surveys Portfolios Record Review Simulations and Models Standardized Oral Exams Standardized Patients (SP) Written Exams (MCQ)

52. 52 Why focus on Outcomes? Directly improves resident learning Relates to the �real world of work� Defines, focuses, and prioritizes content, methods, and assessment

53. 53 Keeps faculty close to the curriculum and the assessment methods Supports communication among and involvement of the faculty Why focus on Outcomes?(continued)

54. 54 Why focus on Outcomes?(continued) Capitalizes on the need for accountability (and opportunities) Links the parts to the whole, i.e., sections, divisions, and departments Develops multiple methods of assessment

55. 55 Goal-based Evaluation �determining to what extent educational objectives are realized by the program of curriculum and instruction, i.e., the degree to which behavior changes take place. Ralph Tyler 1949

56. 56 The Link with Assessment (4) Objective Residents should become proficient in the cost-effective diagnosis and management of common clinical problems. How to measure? Report on a patient management issue that incorporates principles of epidemiology, cost-effectiveness. Targeted reading by the resident that directly relates to cost-effectiveness.

57. 57 Use Multiple Evaluators Reduce bias Increase accuracy Enhance fairness

58. 58 Evaluate on Multiple Occasions Resident behaviors Differences in patients Variable clinical situations

59. 59 He that judges, without informing himself to the utmost he is capable, cannot acquit himself of judging amiss. John Locke 1690