Karen Broquet , M.D. Eugene Beresin , M.D. Nyapati Rau , M.D.

1. Clinical Skills Verification Rater TrainingMODULE 3Setting the Clinical Skills Verification Examination Climate and Giving Feedback Karen Broquet , M.D. Eugene Beresin, M.D. Nyapati Rau , M.D.

2. Module 3Pre-test • Which of the following is assessed in the Clinical Skills Verification Evaluation? a. Patient Doctor Relationship b. Clinical Formulation c. Differential Diagnosis d. Treatment Planning

3. Module 3Pre-test • In assessment of clinical skills evaluations, which of the following methods is most likely to demonstrate what a resident actually “does” according to Miller’s pyramid? a. Arrange Mock Oral Board-style evaluations b. Use Standardized Patients c. Perform the exercise during routine work d. Make use of the Hawthorne Effect e. Add a multiple choice test following the interview

4. Module 3Pre-test • Which of the following rating methods is recommended for a documented Clinical Skills Verification Evaluation? a. Any rating form can be used b. Process notes provided by the resident c. Faculty should use a rating form he/she most prefers d. Faculty should all use the same rating form

5. Module 3Pre-test • Which of the following is considered formative assessment? a. Year-end reviews of performance b. Pass/fail examinations c. Specific feedback of behavior in real time d. Anonymous 360 degree evaluations sent to the resident e. Periodic summative evaluations

6. CSV Exam MUST Assess… Ability to establish doctor-patient relationship Conduct a clinical interview Oral presentation of patient history and clinical findings

7. But can (and should) ALSO be…. • Valuable educational experience • Opportunity for • Direct observation of clinical skills by faculty • Formal evaluation of clinical skills • Focused feedback

8. Feedback is Formative Process by which the teacher provides the learner with information about his/her performance for the purpose of improving performance (Ende 1983) • Without specific and focused feedback, learners: • Rely on self-assessment (unreliable!) • May establish undesired behaviors and/or extinguish desired behaviors

9. Evaluation is Summative “Did I Pass the exam!?!”

10. Feedback = Coach Timely, immediate, ongoing Direct Verbal Face to face Guiding future performance

11. Evaluation = Judge Summative Not ongoing, occurs at the end of time together Assesses whether or not learner met performance standards

12. Quality of both feedback and evaluation depends on the observer’s detection, perception and recall of the resident’s performance How to maximize this: • Incorporate direct observation into clinical teaching routine • Brief (daily rounds) and not so brief (CSV exam) • Normalizes observation/assessment experience • Hawthorne effect: the more we know we are assessed, the better we perform • Write it down! • Specific, accurate feedback is better feedback • Also helps with accurate evaluation • Prepare for the encounter

13. Setting Expectations • How long will the encounter be? • Who will be observing? • What are the goals of the encounter? • i.e., is this a formal CSV evaluation? • Assessment parameters? • What rating instrument/form will be used? • When will “results” be given? • What are the consequences of “passing” or “failing” a CSV exam?

14. What Rating Form/Instrument to Use? Multiple forms accepted by the ABPN Forms vary by level of detail All faculty/CSV raters in a program should be trained on and use the same form

15. How Long Should the CSV Exam Be? • ABPN: “about” 30 minutes for interview and 15 minutes for presentation • AADPRT CS Assessment Task Force: 30-45 minutes for interview, 60-75 minutes total • Programs should choose one format and use consistently

16. Guidelines for giving effective feedback – whatever the setting/encounter

17. Effective/Meaningful Feedback is… • approached with teacher and learner working as allies with common goals • given immediately in a private environment • well timed and expected • based on first-hand (observed) data • focused on specific behaviors, not generalizations • limited to behaviors that are remediable

18. Effective/Meaningful Feedback is… • regulated in quantity (no more than 2-3 items for improvement) • limited to actions and decisions, not assumed intentions or interpretations • phrased in descriptive, nonjudgmental language • closed with suggestions for improvement and a mutually agreed upon action plan

19. Barriers to Effective Feedback Faculty Resident Time Lack of feedback skills Feels like being critical Want learners to like us Gestalt vs specific observations Impact of CSV ‘failure” Time Lack of feedback experience Feels like being criticized Don’t want to expose weaknesses

20. Residents CRAVE feedback

21. To minimize barriers, faculty can… Understand the profound and beneficial educational value of feedback Provide the proper climate and preparation for the experience Elicit reciprocal feedback from trainees Gain the skills for giving both constructive and reinforcing feedback Use a consistent, common rating instrument

22. Cultural Phenomena – International Medical Grads Lack of feedback Experience Diverse cultural backgrounds Power dynamics between teacher and learner Accepted communication patterns Attitudes towards critique Vulnerability in evaluation Stress and difficult social situations Experience with discrimination

23. Important for all learners – especially IMGs Learner centered approach Explicitly prepare the resident for the expectations of the exam and the provision of feedback The importance of the teacher and the IMG working together cannot be over emphasized. Create a safe learning environment and clear expectations and educational objectives

24. Important for all learners – especially IMGs Invite learner to indicate his/her learning needs and provide a self-assessment. Normalize feedback as regular educational activity that includes areas to be reinforced and areas requiring improvement. Be clear when feedback is formative or summative (i.e evaluation)

25. Module 3Post-test • Which of the following is assessed in the Clinical Skills Verification Evaluation? a. Patient Doctor Relationship b. Clinical Formulation c. Differential Diagnosis d. Treatment Planning

26. Module 5 Post-test • Which of the following is assessed in the Clinical Skills Verification Evaluation? a. Patient Doctor Relationship Diagnosis, biopsychosocial formulation and treatment planning are not assessed

27. Module 5Post-test • In assessment of clinical skills evaluations, which of the following methods is most likely to demonstrate what a resident actually “does” according to Miller’s pyramid? a. Arrange Mock Oral Board-style evaluations b. Use Standardized Patients c. Perform the exercise during routine work d. Make use of the Hawthorne Effect e. Add a multiple choice test following the interview

28. Module 5Post-test • In assessment of clinical skills evaluations, which of the following methods is most likely to demonstrate what a resident actually “does” according to Miller’s pyramid? c. Perform the exercise during routine work

29. Module 5Post-test • Which of the following rating methods is recommended for a documented Clinical Skills Verification Evaluation? a. Any rating form can be used b. Process notes provided by the resident c. Faculty should use a rating form he/she most prefers d. Faculty should all use the same rating form

30. Module 5Post-test • Which of the following rating methods is recommended for a documented Clinical Skills Verification Evaluation? d. Faculty should all use the same rating form Although the ABPN has approved a variety of different rating forms for the CSV exam, consistent use of and feedback regarding the same form by all raters in a program is associated with improved inter-rater reliability

31. Module 5Post-test • Which of the following is considered formative assessment? a. Year-end reviews of performance b. Pass/fail examinations c. Specific feedback of behavior in real time d. Anonymous 360 degree evaluations sent to the resident e. Periodic summative evaluations

32. Module 5Post-test • Which of the following is considered formative assessment? c. Specific feedback of behavior in real time Annual performance reviews, 360 degree evaluations and pass-fail examinations are all types of summative assessment.