RESIDENT EVALUATION PROCESS

1. RESIDENT EVALUATION PROCESS Wesley Naritoku, M.D., Ph.D. LAC+USC Medical Center/VAGLAHS

2. Resident Evaluation Process In past, directed at medical knowledge Evaluation process has often been very nebulous, no specific guidelines July 1, 2002 - ACGME Outcome Project was enforced in all residency training Evaluate more than medical knowledge; Six General Competencies How do we document competency-based education?

3. Why Do We Evaluate? Residents expect it - educational role ACGME requires it - “must” = mandatory, you must do it, and you must document that you do it Legal aspects - protecting the public “Cardiothoracic surgeon” applying for privileges

4. What Do We Evaluate? Traditionally, attending staff evaluates resident

5. What Do We Evaluate Now? ACGME Outcome Project Attending evaluates resident Resident evaluates attending Resident evaluates rotation GME Committee evaluates resident GME Committee evaluates program Program Director evaluates resident semi-annually and at end of training ASCP RISE American Board of Pathology 360º Global Evaluations Post-residency evaluation to assess program

6. Building an Evaluation Tool Monthly or rotation-wise evaluation, not specified by ACGME; semi-annual Most programs do a monthly or rotation-wise evaluation (See VII.A.1) Gather Ideas

7. Building an Evaluation Tool: Gather Ideas - Review: ACGME/ABMS Toolbox Other Program’s Evaluation Forms ACGME Booklet (The Nuts and Bolts of Developing Competency Objectives and Assessment Measures) Portfolio for Internal Medicine Residency Programs What are you currently doing? How can this be modified to reflect competency-based education?

8. Building an Evaluation Tool: Designing the PE Form: IMPORTANT! Involve stakeholders (faculty and residents) Be prepared for input (positive and negative) Be willing to make modifications, multiple drafts Consider reiteration of learning objectives within the PE form Do you have a curriculum with learning objectives? Faculty should write them. Make them “robust” Use active language (The resident will demonstrate…) Avoid passive language (The resident will learn…, the resident should be able to recite…)

9. Building an Evaluation Tool: Designing the PE Form: PE forms should be clear-cut so evaluator can complete it with ease. PE forms should be clear for the Program Director to easily identify problems. Forms can easily be adapted for special use, such as “Need Improvement” periods.

13. Designing Monthly Evaluations ACGME Field Representative described a program that listed learning objectives on the back of the evaluation form to prompt memory Why not integrate into form? Assign faculty to write “robust” learning objectives Classify them into the Six General Competencies

15. Designing Monthly Evaluations Monthly PE forms specific for rotation Some PE forms are specific for competency level (graded responsibility) Transfusion Medicine has specific forms for month 1, month 2, month 3 & 4, and month 5 (examples in handout)

16. Distribution Keep track of who gets what form and when; bookkeeping is very important Make certain that the forms are available to the evaluator (beginning vs. end of rotation) The above are both done by the online evaluation system

18. Implementation of Evaluation Process In-Service lecture to faculty and residents: Explain ACGME Outcome Project Explain the need for resident evaluation Explain the due process in evaluations Send out mid-month reminders by e-mail to review objectives If not meeting goals and objectives, inform resident what they need to do to improve; opportunity to improve before giving an adverse evaluation

19. Implementation of Evaluation Process If resident falls short of goals and does not improve, place on “Improvement Needed” status Special evaluation form used daily or weekly during period of remediation Form not an ACGME requirement, but it documents due process (opportunity to improve)

22. Implementation of Evaluation Process Send out end of month reminders to meet with resident for exit interview to discuss evaluation If evaluations are delinquent, send out reminders Resident and faculty must sign and date form, file in resident’s folder Limited access to folder (resident, Chair, Program Director, Program Coordinator)

23. Designing Semi-Annual Evaluations ACGME requires it (VII.A.1.b.) Include evaluation by faculty, patients, peers, self and other professional staff to achieve progressive improvements in residents’ competence and performance for past six months Must meet with Program Director or Associate Program Director (designee) File with other evaluations, keep orderly Limit access

24. USC Semi-Annual Performance Evaluation Design Six General Competencies defined Include period of time covered by semi-annual evaluation List overall evaluations and the evaluations of the Six General Competencies With summer evaluation, include ASCP RISE score Questions about rotation, stress, fatigue and Duty Hours violation

26. USC Semi-Annual Performance Evaluation Design Record Conference attendance (Professionalism) ACGME - Self evaluation, others, case logs Question on future career, fellowship Final comment on how resident performed over past six months Signature and date by resident and Program Director (or designee) to document semi-annual meeting took place

28. Designing Final Evaluations ACGME requires it (VII.A.2) Must include a review of the resident’s performance during the final period of education Must verify that the resident has demonstrated sufficient professional ability to practice competently and independently

29. Designing Final Evaluations An ACGME Field Representative stated that the final evaluation does not need to be signed, but I exit interview residents that are available, and have them sign the form. Residents that have left, I mail the form, have them review and sign and return the form. File with other PE forms Limit access

30. USC Final (Summation) Performance Evaluation Design In spite of only two requirements, I model the Final Evaluation after Dean’s letter - simplicity with one form A senior ACGME Field Representative stated that he used Semi-Annual Evaluation template, re-named as Final Evaluation, and added competency comment

31. USC Final (Summation) Performance Evaluation Design Final Evaluation includes: Resident’s name Begin and end date of training Faculty advisor’s name Any special notation (discontinuation of training, transfer from another program, etc.) List rotation evaluations in chronological order Include any free text comments made by faculty Include any free text comments from 360º Global Evaluations, particularly if significant

38. Designing 360º Global Evaluations Keep it simple, use templates The easier the form is to complete, the better the compliance Also, simple design makes the Program Director’s job easier to assess resident’s performance USC Design of 360º Global Evaluations

43. Faculty Evaluation ACGME requires it (VII.B.) Must be no less frequent than at midpoint between site visits Must be anonymous USC: Faculty evaluated rotation-wise To maintain anonymity, evaluations tabulated on Excel spreadsheet, batched quarterly or semi-annually Online system does this automatically

44. Faculty Evaluation Annual resident retreat, faculty teaching is evaluated At USC, the residents are informed that their evaluations are considered for annual departmental faculty merit evaluation, mandated by the Dean Two-sided form rated 1 to 5; now done on internet Place evaluation reports in document review binder

47. Rotation (Program) Evaluation ACGME requires it (VII.C) Rotation evaluations done rotation-wise At USC, residents discuss ways of improving the rotations at the annual resident retreat Both monthly and annual resident retreat evaluations are discussed and plan of action decided and documented in committee minutes Two-sided form rated 1 to 5; now done on internet File in document review binder

50. Designing Alumni Evaluations Outcomes ACGME implies it (VII.C.3) This is at the heart of the Outcomes project: how effective is the program in training pathologists ABP Report Card on first time pass rate ASCP RISE Both discussed at GME committee, documented in minutes

51. Designing Alumni Evaluations Outcomes Not specified in Pathology Program Requirements; however, other residency and fellowship programs require surveys of graduates at 1 and 5 years Many medical schools are sending evaluations on their graduates to Program Directors When asking for supervisor/senior partner to evaluate your graduate, must consider confidentiality - signatures, anonymous

52. Designing Alumni Evaluations Outcomes Keep form simple I don’t ask for signature of evaluator Alumni survey done simultaneously; separate form that alumni fills out Mail out with self-addressed stamped envelops Tabulate returns, discuss at GME Committee, document in minutes USC survey based upon design by Dr. Ralph Green

55. Alumni Performance Evaluation If the alumni agrees to be evaluated, he/she signs “consent” He/she forwards evaluation to supervisor/senior partner to have them complete form Both evaluatee and evaluator are anonymous

58. Summary Use templates Keep it simple Keep on top of evaluations

59. SUMMARY:Keep your residents’ files orderly

60. SUMMARY: Keep document review binders (ongoing basis)

61. Summary Resident and Faculty must sign and date forms Naturalize the process Empower the faculty to accurately evaluate residents in a timely manner

62. QUESTIONS? naritoku@usc.edu