Creative Commons License

1. Creative Commons License Attribution-NonCommercial-ShareAlike 2.0 • You are free: • to copy, distribute, and display this presentation, and/or • to make derivative works • Under the following conditions: • Attribution. You must give the original authors credit. • Noncommercial. You may not use this work for commercial purposes. • Share Alike. If you alter, transform, or build upon this work, you may distribute the resulting work only under a license identical to this one. • See http://creativecommons.org/licenses/by-nc-sa/2.0/ for full license.

2. Portfolios in Medical Education: Bridging the Gap from Data to Discovery Lee A. Learman, MD, PhD Director of Curricular Affairs UCSF Office of Graduate Medical Education CREOG 2007 Education Retreat Rio Mar, Puerto Rico

3. Learning Objectives Participants will emerge able to: • define and list essential components of a learning portfolio • distinguish an evidence database, formative portfolio and summative portfolio • describe potential uses in GME, including the proposed ACGME learning portfolio

4. Portfolios for Assessment • Purposeful collections of evidence used by students to document and reflect on learning outcomes over time. • Potential uses: • Evidence database: by program vs. learner • Formative: coaching, feedback • Summative: decisions regarding progress Dannefer EF, Henson LC. Acad Med 2007;82:403.

5. Why Portfolios? • Exciting and innovative tool for resident learning and development • Already used in K-12, higher education, and multiple professions • Relieving burden while increasing accuracy for program directors • “added” or “lifted” - “eye of the beholder” • Building a community of practice for GME • Within and across specialties to raise the bar http://www.acgme.org/acWebsite/portfolio/cbpac_memo.pdf

6. Why Portfolios? • Needs identified in the literature: • Focus on complex tasks, integrated competencies • More feedback needed as learning objectives become defined and measured more accurately • Context-dependence: sample multiple sources • New ways to analyze, summarize all the data • Portfolios can meet these needs Dannefer EF, Henson LC. Acad Med 2007;82:403.

7. Self-Reflection & Portfolios Evidence Warehouse Learners Mentors Instructions on reflection A method for assessing reflection

8. Bridge from Data to Discovery Evidence Warehouse Reflection is scored as evidence of a specific competency or PBLI (summative) Learner receives, instructions and selects experience Discusses reflection with mentor (formative) Reflects and summarizes

9. Reflection at UCSF: Instructions • Describe the situation that “taught you the most” about [specific competency] • Describe challenge(s) faced, strategies used • Describe sources of feedback (people, data) • Relate the situation to previous similar ones • Include details to illustrate challenges you faced and lessons you learned • List conclusions re: strengths, opportunities for improvement, and use examples to justify conclusions Learman LA, Autry AM, Pliska L, O’Sullivan PS. WGEA Meeting 2006.

10. The Evidence Sent: Wed 1/10/2007 10:04 PM Subject: Proposal for HROB at SFGH Hello all!As many of you know, Thursday mornings at SFGH pose a challenge to even the most efficient of residents.  Juan and I were discussing the following minor changes to resident roles and Thursday mornings HROB clinic that could potentially make a major difference…Please look over these proposed changes. I would love your feedback – most of this was born from the question of how to make M&M a more consistently educational experience for all (i.e., how can the OB chief and R2 get to OB M&M on time?) -1)      All residents (including OB team) arrive at 5M on Thursdays at 8:00 am to start prepping charts; attending and fellow arrive at 8:30 to hear presentations of prepped charts. Conference should be finished by 9 am. If chart prep not complete and patients present at 9 am, at least 2 residents should leave conference to start seeing patients.2)      Night float chief helps to prep some charts on Wed night for Thursday morning so that conference can be finished by 9 am3)      OB Chief must leave clinic at noon to start M&M - supported by clinic attending4)      OB R2 next to leave, ideally at noon as well5)      Gyn R2 and Clinic Chief last to leave clinic, to cover those 12:30 must-see patients (Gyn R1 covering pager and consults if Gyn R2 needed after 12)6)      If >8 patients remain to be seen at 12:00, then OB R2 stays with the rest of the residents while OB Chief leads M&M alone. If needed, other attendings (mobilized by Dr.V) help to see patientsWithout compromising patient care for education, I'm hoping this may help both clinic flow and M&M utility. Thanks for your time!

11. The Evidence Sent: Thu 1/11/2007 9:38 PM To: OBGYN Resident Class of 2007 Subject: RE: Proposal for HROB at SFGH If all the rest of the chiefs agree with this plan, I'll forward a summary to the rest of the residents and we'll make a go of it next week. As for the main concern of folks arriving at an earlier time, 1) Gyn team is almost always finished rounding by 7:30 because the OR chief must be at OB board rounds. In the remaining half hour, juniors can complete a bit of work. 2) OB Chief will make sure that am rounds finish by 8 am (just as we make our best effort to do on Wed am for conference) and bring charts down from L&D, where at least some have been prepped by the NF Chief. 3) GYN OR Chief (covering for OB Chief) will remind juniors on GYN team to be at conference by 8am and help to get the OB team out of rounds by 8 am 4) GYN Clinic Chief will arrive at 5M at 8:00 instead of 8:30 5) Jeopardy and other clinic residents (OB intern) will arrive at 5M at 8:00 instead of 8:30. Dr. L's suggestion of ensuring that our ancillary services are aware that patients must be seen/in line to see an MD by 11:30-11:45 should also be addressed with M and G (Would you mind doing this, T?). G is aware and enthusiastic about the proposed plan - she has her chart preparations completed by Wednesday evenings, so it shouldn't be a problem to get the charts to the NF chief. Thanks to everyone for being so adaptable!

12. The Reflection Reflective Self-Assessment of Systems-based Practice Instructions: Select a clinical situation you remember during the past year that taught you the most about either practicing cost-effective health care that does not compromise quality of care OR assisting patients in dealing with healthcare system complexities (e.g., surmounting logistical barriers to optimal care – appointments, diagnostic tests). (1) Describe the setting and context including who was present. Setting: San Francisco General Hospital, Thursday morning moderate risk obstetrics clinic. Those involved in the system change are residents, faculty staffing the clinic, clinic staff and clinic flow nurse (2) What challenges did you face in practicing cost-effective healthcare or surmounting systemic barriers to optimal care? The challenge I hoped to address was optimizing the care of patients with complex obstetric problems in the outpatient setting. The major obstacles have been in existence for years: many patients requiring both clinical attention and ancillary services, within a limited time. One additional challenge is finishing the clinic at a time that would allow some or all of the residents to attend M&M conference, a major learning opportunity. (3) Describe what efforts you made to surmount the challenges. What past experiences did you bring to this situation? In the past, there has been a common feeling that “the clinic is just that way – it’s hopeless to try and change it.” I tried to separate myself from that because I wanted people to want to change! In conjunction with one of our faculty members, I came up with ideas on how residents could contribute time that did not risk violating duty hour regulations in order to improve clinic flow. For example, the night float team helping to prepare charts, residents arriving a half hour earlier to finish preparing charts, and then presenting to attendings two at once so that we could start seeing patients as soon as they were placed in rooms. I trusted that that every resident in our program would be interested in bettering the system for all. The proposal involved and that a small sacrifice on one rotation would mean educational benefit for them in another rotation (i.e., what goes around, comes around). I used this reasoning when I sent out the email proposal to my fellow chief residents and the program/clinic directors.

13. The Reflection (4) How did you obtain feedback and from whom? Include details providing evidence that you made cost-effective decisions without compromising the quality of care or that you succeeded in assisting the patient in dealing with system complexities? For the last two weeks, our goal of getting at least two residents to the M&M conference has been successful, but the rest of the residents are not having the opportunity to benefit from this educational conference as I had hoped. I have been asking the faculty who staff the clinic, residents, and the clinic flow nurse how the system has been working. The reviews have been mixed, and since it has been only been implemented for a short time, it’s difficult to determine whether or not this system change is actually beneficial. (5) List what conclusions you drew from situation regarding your strengths and opportunities for improvement, and use examples from the situation to justify your conclusions. I found that motivating people towards change was simple and that those involved were willing to attempt to motivate change themselves, once the activation energy was there. I felt that communicating the importance of this change was one of my strengths. However, putting emotional investment in it means that I have been disappointed that in the first two weeks it hasn’t seemed to work. (6) What changes, if any, do you plan to make if you face a similar situation in the future? I plan to encourage future residents to continue with this system and possibly find even more adjustments to improve its efficiency

14. Reflection Scenarios: ICS & P • Addressing challenging patients • Angry or frustrated • Worried, scared or guarded (including DV) • Difficult or controlling (care-seeking or refusing) • Language barrier • Transgender • Confronting own limitations • Stereotypes (PSA, IVDU, dwarfism) • Disclosing difficult diagnosis (HIV, cancer) Learman LA, Autry AM, Pliska L, O’Sullivan PS. WGEA Meeting 2006.

15. Reflection Scenarios: Surgical • Surgical Skills • 9 Routine Cesarean with complication • 3 Emergency Cesarean delivery • 6 Surgical decision-making (including conflict with attending) • 6 Gyn surgery – complications • 4 Gyn surgery – technical challenges • 2 Other Learman LA, Autry AM, Pliska L, O’Sullivan PS. WGEA Meeting 2006.

16. ACGME Portfolio Management Tool “An interactive web-based development tool that residents can use throughout their residencies to record and organize their learning and to reflect and receive feedback on their skills as physicians, building evidence that allows them to chart their own progress over time.” http://www.acgme.org/acWebsite/portfolio/cbpac_faq.pdf

17. ACGME Portfolio Management Tool • “First and foremost . . .a learning tool for residents that enables them to”: • track their experiences • self-reflect on those experiences • share their insights with mentors, • receive real-time formal feedback • A repository for resident work products and professional documents meeting the needs of many groups including licensing and certification boards http://www.acgme.org/acWebsite/portfolio/cbpac_faq.pdf

18. Portfolio Functions - ACGME • Growth Model (formative) – tracks learner development over time • Showcase Model (summative) – snapshot demonstrating achievement of identified outcomes as for a grade, promotion, or graduation • Hybrid – supports both purposes http://www.acgme.org/acWebsite/portfolio/cbpac_memo.pdf

19. Portfolio Approach at CCLCM Dannefer EF, Henson LC. Acad Med 2007;82:403.

20. Evidence-based Essentials • Time for reflection and mentorship • Separation of formative and summative • Students select evidence of learning • Essays required to aid reflection on integration of competencies • Rigorous measurement standards for summative assessments (fair, valid, reliable) Dannefer EF, Henson LC. Acad Med 2007;82:403.

21. UCSF Scoring Rubric more superficial 1. Describes encounter only. 2. Unsupported opinions about lessons learned. 3. Superficial justification of lessons learned. 4. Discussion well-supported with examples of challenges, techniques and lessons learned. 5. Analyzes factors from experience that contribute to progress. 6. Justifies strategies used and evidence for effectiveness. deeper Learman LA, O’Sullivan PS. AAMC-RIME 2006.

22. Reflection Growth by PGY * Effect size = (difference in means)/pooled standard deviation ^Skill scores range from1-6 and include all exercises completed by 32 residents. #Overall scores include only the 25 residents who completed all six reflections. Learman LA, O’Sullivan PA. AERA Annual Meeting 2007.

23. Resident Feedback Focus groups after 1st year of reflection 05/06: • Valued reflection, but had incomplete and variable understanding • Preferred reflection-in-action (in real time) over reflection-on-action (delayed) • Discounted value of written (versus oral) reflections • Felt that specific assignments were constraining and artificial • Wanted feedback on and discussion of reflections Foster-Barber, Chittenden, Learman, O’Sullivan. UCSF Education Day 2007

24. Improvements in 06/07 • Better explanation of role of reflection in medical education • Choice among 6 options, 3 for each semi-annual meeting • Clearer instructions • Exercises divided into discrete tasks • More lead time for sharing with peers • More lead time for review by program directors prior to feedback session Foster-Barber, Chittenden, Learman, O’Sullivan. UCSF Education Day 2007

25. Lessons Learned on Reflection • Optimal design of exercises unclear • Need to: • Hone residents’ understanding of reflection • Give more freedom in content/timing of reflections • Provide mechanisms to ensure timely feedback • Introduce reflection exercises earlier in medical education to improve their acceptability to residents Foster-Barber, Chittenden, Learman, O’Sullivan. UCSF Education Day 2007

26. Reflecting About Portfolios

27. What They Are • Purposeful collections of evidence used by students to document and reflect on learning outcomes over time. • A learning tool enabling residents to: • track their experiences • self-reflect on those experiences • share their insights with mentors, • receive real-time formal feedback

28. Consider • What do you think? • Fad or Formidable Innovation • Which aspects seem more/less valuable? • Tracking progress • Refection and self-assessment • Feedback and mentorship

29. Their Purposes or Functions • To store evidence of learning ≠ a true portfolio • To promote feedback, reflection, growth = formative • To determine advancement, graduation = summative

30. Consider • Which functions will be easiest to implement in your residency program? • Data warehouse • Formative review of growth, development • Summative decisions • Why?

31. The Essentials for Success • Time for reflection and mentorship • Separation of formative and summative • Learners own access, grant permission • Essays assigned to aid reflection • Summative decisions are based on fair, valid, and reliable assessments » Not even the best technology can make-up for an absence of any of these essentials!

32. Consider One • Mentorship and Feedback • How is time currently set aside for mentorship? Are mentors different than faculty responsible for advancement and graduation? Do they review evidence of learning and provide formative feedback? How might this be improved? • Owning the Evidence • Currently, are any assessments or evidence of learning “owned” by your residents and shared with you at their discretion? What kinds of evidence could be? • Reflecting on Learning • What opportunities (or time) do your residents have to do structured reflections about their progress? How could this be created or expanded?

33. ACGME Learning Portfolio Timeline • Now: Alpha test, cultivate early adopters, specialty user-groups • Early 2008: Create beta phase prototype • Mid 2008 – Late 2009: Beta testing phase • Early 2010: Finalize initial roll-out prototype • 2010: Initial roll-out available, voluntary • 2016: Full implementation, available for all • Ongoing: Consider linkages to UME and MOC http://www.acgme.org/acWebsite/portfolio/cbpac_memo.pdf

34. Coming Sooner Than 2016 • PIF Transition Document • PBLI: “Describe one learning activity in which residents engage to identify strengths, deficiencies, and limits in their knowledge and expertise (self-reflection and self-assessment); set learning and improvement goals; identify and perform appropriate learning activities to achieve self-identified goals (life-long learning)” • New PIF likely to elaborate . . .

35. Opportunities • Studies across GME programs to establish fairness, validity and reliability of measures • Faculty development for giving high-quality, behaviorally-anchored feedback • Faculty development for advisors and summative assessors • Optimize how self-reflection happens • Participate in beta-testing of ALP

36. Dannefer EF, Henson LC. The portfolio approach to competency-based assessment at the Cleveland Clinic Lerner College of Medicine [of Case Western Reserve University]. Academic Medicine 2007;82:493-502. Challis M. AMEE Medical Education Guide No. 11 (revised): Portfolio-based learning and assessment in medical education. Medical Teacher 1999;21(4):370-86. O’Sullivan PS, Cogbill KK, McLain T, Reckase MD, Clardy JA. Portfolios as a novel approach for residency evaluation. Academic Psychiatry 2002;26(3):173-79. To Learn More . . .

37. Acknowledgments • Meg Autry, Laura Pliska, and Patricia O’Sullivan for development and implementation of the reflection exercises • Patricia O’Sullivan for feedback regarding earlier versions of this presentation