Teaching and Evaluating Professional Competence: Tools and Models

1. Teaching and Evaluating Professional Competence: Tools and Models 2007 AACOM Annual Meeting Baltimore Maryland, June 29, 2007

2. Cast of Characters • Erik Langenau, DO, FAAP, FACOP • Program Director, Pediatrics • Maimonides Infants and Children’s Hospital of Brooklyn • Christine Black-Langenau, DO • Associate Program Director, Family Medicine • Lutheran Medical Center • Elizabeth Kachur, PhD • Medical Education Specialist • Medical Education Development • David Yens, PhD • Associate Professor, EDA • NYCOM/NYIT

3. AACOM 2007

4. Proving that… New York is the center of the Universe AACOM 2007

5. Professionalism 101 Elizabeth Kachur, PhD Medical Education Consultant Medical Education Development New York, New York

6. Professionalism 101 Learning Assessment • What is Professionalism? • How does it fit into AOA Competence Guidelines? • Can Professionalism be taught? • Can Professionalism be assessed? Assessment Drives Learning

7. Professionalism is the habitual and judicious use of … • Communication • Knowledge • Technical skills • Clinical reasoning • Emotions • Values • Reflection in daily practice … for the benefit of the individual and community being served Epstein/Hundert, 2002

8. Professionalism Arnold/Stern, 2006 Excellence Humansim Accountability Altruism Ethical & Legal Understanding Communication Skills Clinical Competence (Knowledge of Medicine)

9. How Students Operationalize Professionalism • Ginsberg et al. 2002: Focus group study at 3 medical schools • Six critical "issues" emerged: • Communicative violations(to or about patients or other health care professionals) • Role resistance(individuals chafing against constraints or expectations of their perceived roles) • Objectification of patients(ignoring patients or treating patients as vehicles for learning) • Accountability (of colleagues or patients, including avoiding patients, failing to disclose information or treat appropriately) • Physical harm(to patients or others) • Crossfire(being put in the middle of a struggle between superiors)

10. Osteopathic Philosophy & OMT Medical Knowledge Osteopathic Patient Care Interpersonal & Communication Skills Professionalism Practice-based Learning and Improvement Systems-based Practice Osteopathic Medical Competencies

11. Required Elements: • Knowledge • Humanism • Primacy of Patient Need • Accountability • Continuous Learning • Ethics • Cultural Competence Osteopathic Medical Competencies Osteopathic Philosophy & OMT Medical Knowledge Osteopathic Patient Care Interpersonal & Communication Skills Professionalism Practice-based Learning and Improvement Systems-based Practice

12. Implementation Approaches Patient Care, Interpersonal & Communication Skills, Professionalism A O A Osteopathic Philosophy & Medical Knowledge Practice-based Learning & Systems-Based Practice Improve evaluations, collect performance data for internal review – all competencies Use performance data for improvement, begin using external quality measures – all competencies A C G M E Define objectives, begin integration into learning – all competencies Involve community in quality definition, benchmarking

13. Can professionalism be taught? • Curricular formats are often didactic, removed from clinical setting, focus on abstract concepts • Educational interventions must: • Teach explicitly • Attend to hidden curricula, culture, role models, incentives • Include faculty development • Be specific to context & learning-stage • Reach throughout continuum of medical education • Span across all institutional levels (e.g., ecology) • Have adequate institutional support • Include a system of evaluation which reinforces teaching

14. Can professionalism be assessed? • Most tools are for research and program evaluation, have poor or unknown measurement properties • Need to assess elements of professionalism separately and as comprehensive construct > multi-method approach • Instruments may be better if they: • define professionalism as behaviors expressive of value conflicts • investigate the resolution of these conflicts • recognize the contextual nature of professional behaviors

15. Thank You!

16. What are the obstacles and barriers to evaluating professionalism? • Elements of Professionalism • Knowledge • Humanism • Primacy of patient need • Accountability • Continuous learning • Ethics • Cultural competency

17. Obstacles identified (summary of large group discussion) • Too many learning objectives within “professionalism”: 167 items at one school. How do we make sense of these individual items. • 360 degree evaluations: are they useful? Are they practical? Are they valid (especially with peer evaluations)? • Concept itself—nature of “professionalism varies with context (primary care vs. surgical care; basic sciences vs. clinical sciences) • SPs’ evaluations: are they valid for evaluating professionalism (they have been validated for med knowledge, communication skills, and other humanism components; but, how about professional behaviors)? • People don’t want to say “bad things” about others • Are we settling for “minimum competency” for professional behavior, and lowering our professional expectations? • People define professionalism in different ways. Is a pierced ear professional or not? Is a necktie essential or not?

18. How Is Professionalism Measured In Medical School? David Yens, PhD Associate Professor, EDA New York College of Osteopathic Medicine Old Westbury, New York

20. NYCOM APPROACH • Attendance at a professionalism curriculum development workshop by a NYCOM delegation in December, 2005 • Creation of the Professionalism Advancement in Curriculum and Education Project (“PACE”) under the leadership of VP and Dean Ross Lee, D.O. and Humayun Chaudhry, D.O., M.S. • Assessment of medical and clinical professionalism as part of the patient simulation program (Institute for Clinical Competence) • Identification of six proposed tasks to foster professionalism • Conducting a literature review and identify potential instruments for assessing professionalism • Creating and administering a survey to assess baseline knowledge and beliefs about professionalism at NYCOM

21. PACE Initiative • Incorporate professional assessment as part of training provided in the Institute for Clinical Competence for all OMS 1 and 2 students • Promote a process by which there is rewarding of achievements in professionalism to NYCOM students in all years of their undergraduate curriculum • Formulate a plan for NYCOM faculty and staff development on medical professionalism • Develop a process at NYCOM whereby OMS 1 and 2 students are asked to maintain medical professionalism narratives in the form of an anonymous, secure, web-based professionalism journal (modeled after Indiana University SOM) • Establish a mechanism where NYCOM students can recognize their peers for demonstrating and modeling exceptional behavior and professionalism • Create a “professional portfolio” for every incoming NYCOM student that will contain letters and anecdotes from student peers, faculty, and patients relating to professionalism

22. BASELINE DATA PRIOR TO START • Sources of Evaluation Instruments • ACGME • AAMC

23. Good Source of Instrumentswww.acgme.org/outcome/assess/profindex.asp

24. AAMC Resource

25. The Surveys • Provide baseline data • Survey titled “Assessment Of Perceptions Of The Environment” • Three parts: • Response to two ethical situations • General perceptions of faculty, administrators, and students • Observation of student and faculty behavior • Administered to classes of 2009 and 2010 during a class in fall, 2006 • Demographic information about age, gender, and specialty plans collected • IRB approval

26. SAMPLE ITEMS • 1. During a class-wide demonstration in the OMM training laboratory, you notice a male student place his arm around the waist of a female student and thank her for helping him with the “the procedure.” You sense that the student is made uncomfortable by the gesture. An appropriate first response would be which of the following? • A. Do nothing, on the basis that the student was simply showing his appreciation. • B. Report the incident to the lab instructor as an example of sexual harassment. • C. Tell the student that you thought the gesture was inappropriate and that you were made uncomfortable by it. • D. Ask the female student if the gesture made her uncomfortable. • E. Ask the female student if there are actions she would like you take on her behalf.

27. SAMPLE ITEMSPERCEPTIONS OF FACULTY AND PEERS

28. SAMPLE ITEMS

29. RESULTS • Class of 2010: n =152 (50%), M=43%, F=57% • Differences in item responses by gender – 1 • Class of 2009: n = 145 (48%), M=43%, F=57% • Differences in item responses by gender – 1 • Overall differences between classes: p<.01 • Differences between classes on several individual items • Class of 2010 somewhat more positive than class of 2009

30. What’s next • Modify the survey as needed • Seek other schools willing to use the survey to provide interschool comparisons • Using the survey results as a basis, pursue the six tasks originally proposed for the PACE project • Adapt and extend the survey for use during clinical and postgraduate years

31. THANK YOU

32. How Is Professionalism Measured In A Pediatric Residency Program? Erik Langenau, DO, FAAP, FACOP Residency Program Director, Pediatrics Maimonides Infants and Children’s Hospital of Brooklyn Brooklyn, New York

33. The Maimonides Infants and Children’s Hospital of BrooklynOSCE Curriculum Communication OSCE (PL-1 year) Culture OSCE (PL-2 year) Genetics OSCE (PL-3 year)

34. Multimedia Food Involved Faculty Pre-OSCE Workshop

35. Going for more food Timing Reading Scenario / Instructions Preparing for station tasks: 2 Minutes

36. Resident Instructions What you’re supposed to do in room

37. Playing video before encounter: 30 seconds

38. Well Trained Actor Receptive Resident Interview: 10 minutes

39. Actors trained to give relevant feedback Immediate Feedback by Faculty and Standardized Patients: 5minutes

40. Feedback Protocol Self-assessment 1st Then SP & Faculty Station Specific Teaching Points

41. Debriefing

42. Communication With Colleague http://www.devines.net

43. Type of Behavior Witnessed (N=88) Percent

44. How Is Professionalism Measured In A Family Medicine Residency Program? Christine Black-Langenau, DO Associate Residency Program Director, Family Medicine Lutheran Medical Center Brooklyn, New York

45. Reflective Learning Family Medicine

46. Reflective Learning • Team evaluation that takes place monthly with in-patient team • Moderated by an Attending and Psychologist • Two 15 min sessions with chief and then rest of team with joint debriefing afterwards • Each resident involved 3-4 times a year • Filed in residents’ portfolio

47. Achievements • Teamwork Skills • Academic Activities • Problems Solved • Problems Requiring Improvement • Individual Roles • Suggestions for future teams

48. Reflective Learning Activity:Examples of teaching and evaluating professionalism identified • Team cancelled a academic activity because of increased census • A consultant that did not see a patient • Team members complimented a senior for working every day even when sick

49. Advantages of Reflective Learning Activity • Offers tool to evaluate how residents • work with other team members and • utilize resources within the hospital. • Self-report of difficult cases provides feedback on resident performance (strengths/weaknesses) • Pinpoints areas that may need further evaluation or development • Provides teaching opportunity when other members of the system do not demonstrate professional behavior.

50. Now, it’s your turn!! • Let’s see how cleaver and creative we can be. • Please divide into small workgroups to share experiences and tools. • These workgroups should be diverse, with each group containing participants from different disciplines (basic sciences, clinical sciences, deans, researchers, clinicians, students) • Please identify a specific behavior or attitude to evaluate, and develop a strategy to measure that behavior