Competency Based Assessment

1. Competency Based Assessment Baraa Alghalyini SAPIC Dec 1st 2012

2. Today.. • Define Competency Based Assessment • Discuss CBA • goals, • tools, • limitations and, • unintended effects

3. What is Competence in Medicine? • Epstien RM, JAMA 2002 “The habitual and Judicious use of communication, knowledge, technical skills, clinical reasoning, emotions, values, and reflection in daily practice for the benefit of the individuals and communities being served”

4. What is Competence in Medicine? • Frank et al 2010 • Competency Based Education is • “Approach to preparing physicians for practice that is fundamentally oriented to graduate outcome abilities and organized around competencies derived from an analysis of societal and patient needs. It deemphasizes time-based training and promises greater accountability, flexibility, and learnercentredness”

5. Goals of Assessment • 1) Optimize Learners capabilities • 2) Protect the Public • 3) Determine bases of applicants’ selection for advanced training

6. Summative Assessment • Provide overall impression of learners’ • performance • Competence • Qualification for advancement to higher education • Assessment of learning

7. Formative Assessment • Reinforce internal motivation • Guide future learning • Promote reflection • Assessment for learning

8. Good Assessment Criteria • Validity or Coherence • does the assessment measure the real world? • Reproducibility or Consistency • assessment results the same if repeated under similar circumstances

9. Good Assessment Criteria • Equivalence • Different repetitions of same assessment yield same results • Educational effect • motivates those who takes it to prepare for it in a way that promotes learning

10. Good Assessment Criteria • Catalytic effect • provides feedback that enhances education • Acceptability • assessors and learners found it credible • Feasibility • practical and realistic

12. Assessment Tools

13. Commonly used assessment tools Bandiera G, Sherbino J, Frank J. The CanMEDS Assessment Tools Handbook: An introductory guide to assessment methods for the CanMEDS Competencies. Ottawa (ON): Royal College of Physicians and Surgeons of Canada; 2006.

14. Commonly used assessment tools Bandiera G, Sherbino J, Frank J. The CanMEDS Assessment Tools Handbook: An introductory guide to assessment methods for the CanMEDS Competencies. Ottawa (ON): Royal College of Physicians and Surgeons of Canada; 2006.

15. Commonly used assessment tools Bandiera G, Sherbino J, Frank J. The CanMEDS Assessment Tools Handbook: An introductory guide to assessment methods for the CanMEDS Competencies. Ottawa (ON): Royal College of Physicians and Surgeons of Canada; 2006.

16. Assessment Tools • Multisource Feedback (360° Evaluation) • Supervisor report of direct observation • Peer assessment: Residents, Medical students • Allied Health Professionals • Patients, their family members • Self assessment • Summative and Formative

17. Assessment Tools • Multisource Feedback: • Advantages: • Encourage self reflection • Evaluate certain behaviors: professionalism, communication skills • Disadvantages: • Time consuming • Survey to be meticulously designed to ensure validity

18. Assessment Tools • ITER/FITER In Training Evaluation Report/Final In Training Evaluation Report • Report completed based on several observation opportunities in different clinical settings

19. Assessment Tools • ITER/FITER • Advantages • Immediate feedback is possible • High face validity if structured and standardized • Disadvantages: • Different observers might observe different things • Documentation not timely or not done

20. Assessment Tools • Encounter Cards: • Daily Evaluation Cards, • Daily Encounter Cards, • Daily Operative Cards, • Daily Teaching Evaluation Cards • Capture clinical competence from brief encounters • Increasingly popular in ICU, Emergency Medicine, and Surgical rotations

21. Assessment Tools • Encounter Cards: • Advantages: • Frequent, timely, formative and grounded feedback • Method of tracking procedures performed/ clinical exposure • May contribute to ITER/FITER • Disadvantages: • Frequent feedback meetings • Assessment fatigue • Faculty development is required to train assessors use cards effectively

22. Assessment Tools • Portfolio • Collection of assessment tools: log book, performance diaries, multisource feedback, others?

23. Assessment Tools • Portfolio • Advantages: • Learner centered • Longitudinal • Authentic • Flexible • Disadvantages • Require on going monitoring by reviewers • Learners’ poor compliance • Difficult to be standardized

24. Assessment Tools • Simulation: • Artificial recreation of clinical environment or circumstances

25. Assessment Tools • Simulation: • Advantages: • High risk procedures in safe environment • Measure performance in emergency situations • Standardized • Disadvantages: • Cost • Resources

28. CanMEDS Assessment Matrix

29. CanMEDS implementation 2001-2009 Faculty Adoption of a Nationwide Competency Framework 2001-2009: The CANMEDS Roles in Residency EducationJason Frank, C. Abbott, G. Bourgeois, S. Hyde, A.C. Lee

30. CanMEDS Roles Implementation 2001 VS 2009 A little Somewhat Mostly Completely Not at all Faculty Adoption of a Nationwide Competency Framework 2001-2009: The CANMEDS Roles in Residency EducationJason Frank, C. Abbott, G. Bourgeois, S. Hyde, A.C. Lee 30

31. Which of the following methods do you use to teach the CanMEDS competencies of your residents? All CanMEDS Roles Faculty Adoption of a Nationwide Competency Framework 2001-2009: The CANMEDS Roles in Residency EducationJason Frank, C. Abbott, G. Bourgeois, S. Hyde, A.C. Lee

32. Which of the following methods do you use to assess the CanMEDS competencies of your residents? All CanMEDS Roles Faculty Adoption of a Nationwide Competency Framework 2001-2009: The CANMEDS Roles in Residency EducationJason Frank, C. Abbott, G. Bourgeois, S. Hyde, A.C. Lee

33. Barriers in ensuring teaching and assessment of CanMEDS (N = 374) % of Program Directors indicating barrier Faculty Adoption of a Nationwide Competency Framework 2001-2009: The CANMEDS Roles in Residency EducationJasonFrank, C. Abbott, G. Bourgeois, S. Hyde, A.C. Lee

34. Assessment Pitfalls, Challenges&Unintended Outcomes

35. Pitfalls of Assessment • (Ginsbur, 2010) • More emphasize on objectivity of assessment method • Discount subjective measures thinking they are unreliable • Misconception that eliminating human inference from judgment process minimize error

36. Pitfalls of Assessment • ( Kogan, 2010) • Faculty’s own clinical skills may affect their evaluation of trainees

37. Challenges with Current Assessment Methods What are your challenges?

38. Challenges with Current Assessment Methods Assessors: • (Laeeq et al, 2010) • Poorly motivated • Lack of resources :time, financial, personnel • Inadequate training • Solutions.. • Embrace CBE curriculum and concepts • Internalize it, teach it, assess it • Self reflection!

39. Challenges with Current Assessment Methods Trainees • Not involved in the process of curriculum development and its assessment tools • Not being well oriented to the assessment tools or entrustable professional activity(EPA) • Not providing feedback to program directors and faculties on the assessment process

40. Challenges with Current Assessment Methods Curriculum • Doesn’t meet specific “competencies” standards • Evaluated with different assessment methods

41. Challenges with Current Assessment Methods Tools • Newly introduced without proper training to faculty and trainees • Emphasis on what learners have done instead to what they demonstrated • Impractical • Invalid

42. Unintended Effects of Assessment • What are the negative effects of assessment? • Learners focus on the test, not on improvement • De-emphasizes things that are not ‘testable’

43. Unintended Effects of Assessment • Episodic anxiety and stress • May not assess what it is meant to assess • Unwanted bias to affect assessment of progress (afraid to give negative feedback) • May take time away from effective learning

44. What is Entrustable Professional Activity ( EPAs)? • What the public trust clinicians are capable doing • Represent what trainees are expected to master doing based on their specialty and subspecialty

45. What is Entrustable Professional Activity ( EPAs)? • Cate, Scheele 2007 • “ a practitioner has demonstrated the necessary knowledge, skills, and attitudes to be trusted to independently perform this activity”

46. Conditions of Entrustable Professional ActivitiesConditions of Entrustable Professional Activities (Cate, Scheele 2007) • 1. Part of essential professional work in a given context. • 2. Must require adequate knowledge, skill, and attitude. • 3. Must lead to recognized output of professional labor. • 4. Should be confined to qualified personnel. • 5. Should be independently executable. • 6. Should be executable within a time frame. • 7. Should be observable and measurable in its process and outcome (well done or not well done). • 8. Should reflect one or more competencies

47. Could you suggest an EPA for your Learners?

48. Home take message • Every key competency MUST be ASSESSED somewhere in the program • Not every competency needs to be assessed in every activity/rotation • Choose the best methods for the competency • Be careful to match expectations to level

49. Home-take message • Try to combine competencies into assessment strategies (such as an OSCE station that reviews multiple competencies at once) • Choice of assessment tool should depend on educational context rather than a blind desire to use objective or standardized assessment tools • EPAs, EPAs, and EPAs!!

50. Questions?