Feedback & Evaluation

1. Feedback & Evaluation Eva Aagaard, MD Vice Chair for Education Department of Medicine

2. Overview • Feedback • Why should you care about feedback? • What is feedback? • What’s the evidence that feedback works? • How do you give feedback effectively? • Case Discussion

3. Feedback

4. Why Should You Care? • Trainees WANT feedback • In a study of over 1500 residents, 96% believed feedback was important for learning • Schultz, BMC Central 2004 • Trainees feel they NEVER get enough feedback • Gil, J Med Educ 1984 • Isaacson, J Gen Int Med 1995

5. Feedback is ESSENTIAL to Teaching • Giving high-quality feedback is strongly associated with teaching ratings • Torre, Acad Med 2003 • Constructive and specific feedback can improve learner knowledge and skills (More on this later)… • Boehler, Med Ed 2006 • Clay, Critical Care Med 2007

6. Definitions Or What is Feedback?

7. Evaluation VS. Feedback • Evaluation: • Summative • Higher stakes • Generally standardized • Goal is to grade relative to peers or a gold standard • Feedback: • Formative • Goal is to help people improve • Can be formal or informal

8. Categories of Feedback • Brief feedback • Brief, specific notation of a positive or negative behavior • Generally provided for directive teaching • Ex: “I think that what you noted to be distension is actually ascites. Let me show you some techniques to distinguish these things.” Branch, Acad Med 2002

9. Categories of Feedback • Formal Feedback • Set aside a period of time (5-20 min) to discuss performance on a specific issue • Can be event or encounter specific • Exs: • Mistakes • Handling of a specific patient case • Feedback following a case presentation or a teaching encounter • Still very specific and generally limited in scope Branch, Acad Med 2002

10. Categories of Feedback • Major Feedback: • Scheduled sessions to review overall performance to date (15-30 min) • Must be given in a timely enough fashion that trainee has the opportunity to improve/ grow/ respond • Ex: • Midpoint feedback on a rotation Branch, Acad Med 2002

11. The Evidence

12. Procedures TrainingBoehler, Med ED 2006 Does Provision of Specific Procedural Skill Feedback Improve Performance in that Skill Compared to General Comments?

13. Design • 33 2nd and 3rd year Med Students at Southern Illinois University • All given same instruction on 2-handed surgical knot-tying • Randomized to specific feedback or general compliments • Videotaped performing the skill before and after feedback • Videotapes scored by 3 blinded observers • Student satisfaction with instruction assessed via 7-point global rating scale

14. Mean Performance Scores P<0.001 P=0.2 Total Possible Ratings 0-32; IRR >0.8

15. Mean Satisfaction Scores P=0.005 7-Point Likert Scale (1=Very Poor; 7= Truly Exceptional)

16. Take Home Points • Specific feedback can improve performance on procedural skills • FEEDBACK  LEARNING • Satisfaction with feedback is a poor marker of the QUALITY of feedback • SATISFACTION ≠ QUALITY • Combination of specific feedback and compliments may be best option

17. Nontechnical SkillsSavoldelli, Anesthesiology 2006 Does Feedback Improve Resident Performance in Nontechnical Simulator Skills? Is Video-Assisted Oral Feedback Better than Oral Feedback Alone?

18. Design • Forty-two 1st, 2nd and 4th year anesthesia residents at the University of Toronto • Videotaped managing an intraoperative cardiac arrest on a simulator with scripted operating room personnel • Randomized to 3 arms • No feedback • Oral feedback • Video-assisted oral feedback • Videotaped on a second (different) intraoperative cardiac arrest scenario

19. Design Cont… • Videos scored by 2 blinded reviewers using a validated behavioral marker system (ANTS) • Task management • Team working • Situational awareness • Decision making • Scenario order randomized • Reviewers blinded to chronological order of scenarios • OUTCOME=Mean change in ANTS score

20. Key Feedback Points • Focused on nontechnical (cognitive and behavioral) skills • Resident reflection on encounter elicited • Specific and constructive comments provided • Both positive and negative • Feedback limited to 4-6 main points

21. % of ANTS Score Change P<0.05 P<0.01

22. Take-Home Points • Oral feedback improves nontechnical skills in a simulated crisis environment • Specific, limited feedback along with resident self-reflection were used in this study • Videotape review was not better than oral feedback alone

23. How Do I Do It Well? Giving Feedback 101 Adapted from Ende, JAMA 1983

24. Key Principles • The Groundwork • The Meeting • Things to Remember

25. The Groundwork • Set clear objectives and goals upfront • What does the trainee hope to get out of your time together? • What specific behaviors do you expect? • When will you give the trainee feedback? • SET AN APPOINTMENT NOW • When will you reassess their performance and reset goals?

26. Preparation • Organize your thoughts and observations ahead of time • Setting: • Positive brief feedback can be given almost anywhere • Negative or major feedback should always be given in private and without interruptions • Timing: • As close to event as possible • Enough time to make a change • Make an appointment • Make sure learner is ready to hear it • Not being paged • Not distracted • Not distraught

27. Opening the Encounter • Describe the Purpose • Label it FEEDBACK Sostock, Acad Med 2002

28. Conducting the Meeting • ASK • Elicit self-reflection • TELL • Give both positive and constructive feedback ALWAYS • Be SPECIFIC and use non-judgmental language • Behaviors not personality • Objective, observable and modifiable • Provide suggestions for how to improve • ASK • Elicit trainee understanding of feedback • Allow trainee to develop own suggestions for improvement plan

29. Closing the Meeting • SUMMARIZE • Positives • Areas for improvement • Plan for improvement • Plan for when to reassess/ meet again

30. REMEMBER…. • Limit constructive feedback to NO MORE than 4 (and probably 2) areas

31. REMEMBER… • Listen to the persons perspective and feelings

32. PRACTICE

33. BRIEF FEEDBACK

34. The Consult Call You ask the MS4 to call a consult on a patient with brisk UGI bleeding. You listen as she speaks to the fellow on call. The presentation includes all pertinent material and is concise and organized. She asks specifically if the fellow would be willing to perform an emergent EGD, and states that this is the reason for her call.

35. Your Feedback? When? Where? What?

36. BRIEF FEEDBACK • Timely- in the moment • Generally can be given anywhere (unless of sensitive nature) • Specific • Objective

37. An Example “I wanted to give you some quick feedback on your consult call. You provided the patient information clearly and concisely. More importantly, you clearly stated the reason for the consult. Nice work.”

38. FORMAL FEEDBACK

39. The Medical Student Case Presentation “Mr. Jones is a 50 year old man with cirrhosis related to hep C and alcohol abuse who is admitted with altered mental status. He was brought in by his friend who found him confused and lethargic after not showing up to work as expected. The patient is unable to give a history of recent events due to marked lethargy and confusion. Past medical is as already noted. Meds are unknown. He was discharged on spironolactone and atenolol at his last admission.

40. Presentation Cont… He works construction and lives alone. The friend believes that he has quit drinking and drug use, although he is not sure. On PE he has normal vital signs. Neuro exam is nonfocal. Abdomen is distended with tense ascites, diffusely tender without rebound. Rectal is hemoccult negative.”

41. Your Feedback? When? Where? What?

42. FORMAL FEEDBACK • Label It- “Let me give you some feedback on your presentation on Mr. Jones” • Set an appropriate time and place • Timely • Private place • Include positive and negative • Be specific • Provide suggestions for improvement

43. An Example + Your identification of Mr. Jones as a cirrhotic due to alcohol and Hep C helped give me an idea of what the most likely causes of his altered mental status are - It would have been very helpful to have a better understanding of the severity of his underlying illness. It sounds as if that information may have been available in the most recent discharge summary

44. Suggest Please make sure to include anything you think might be important information from the medical records in your oral presentation next time.

45. MAJOR FEEDBACK

46. Mid-Point Feedback You have been working with your medical student for 2 weeks. His presentations are excellent and his clinical acumen is strong. Today one of your nurses pulled you aside. Per your instructions, she had been calling him with the results of labs and x-rays on the patients that he has seen in clinic. She is upset because yesterday he took a long time to return pages and when he finally did call her back, he was abrupt. Today is the day you are scheduled to give midpoint feedback.

47. Your Feedback? When? Where? What?

48. MAJOR FEEDBACK • Set the Stage • Private place • Enough time free of interruptions • ASK—TELL---ASK • Give both positive and constructive feedback • Be SPECIFIC and use non-judgmental language • Behaviors not personality • Objective, observable and modifiable • Provide suggestions for how to improve, and elicit a self-improvement plan from the learner • Summarize

49. An Example • Set the Stage: • It’s time for our feedback session. • Let’s go to my office • Is now a good time? • ASK: • How do you think you are doing so far? • What things have gone particularly well? • Are there any areas that you think you need to improve on?

50. Example Cont… • Tell: + Your oral presentations and written notes are clear, concise, and do an excellent job of depicting our assessment and plan. + Your clinical decision-making has been consistently right on such that I feel I can completely count on your recommendations. This has been true even in complicated cases such as….