The Irkutsk State Medical Academy for Postgraduate Education (Tempus IV 159328)

1. The Irkutsk State Medical Academyfor Postgraduate Education(Tempus IV 159328) STEERING COMMITTEE MEETING & INTERNATIONAL FINAL CONFERENCE Evaluation of Medical Student: Knowledge, know-how and life skills Pr Sylvain Meuris Faculté de Médecine UniversitéLibre de Bruxelles

2. Objective :To Reach Professional Competence ! Is the habitual and judicious use of: • Communication • Knowledge • Evidence-based decision-making • Technical Skills • Emotions • Values and reflection to improve the health of the individual patient and the community

3. Who Assess and What Should Be Assessed ? • Transmission of knowledge, know-how & life skills between teacher and student • Who : • Teacher - During the training process • Academic steeplechase (Education Minister)! • Peers - End of the training process • Accreditation Council for Graduate Medical Education (Health Minister) • What : • Numerous and Cumulative Competencies : Knowledge + Practice-based learning and improvement +Interpersonal and communication skills + Ethics & Professionalism + System-based practice + Patient care,…

4. Is Assessment Necessary ? • No formal assessment before the Renaissance in Europe • Chinese invention (VI century) to • Select Imperial officers • Imported by Portuguese Jesuits to • Select Representatives for the Portuguese Colonial Empire • Widely adopted in • European administrations and • Universities !!! • Probably useful

5. Is Assessment Necessary ? • A useful tool to : • Train students and • Assume graduation responsibility • Two questions: • WHY do we need to assess ? • WHAT do we want to measure ?

6. Is Assessment Necessary ? • Answers depends on the position • Teacher verify if the minimum level of “knowledge” is reached • Learning strategy • Student prepare examination to reach the minimum level to be authorized to continue… • Strategy to pass

7. WHY do we assess ? • To ensure safety of patients our responsibility to the public • To achieve of a minimum standard responsibility to the candidate and to the University ! In principle…

8. WHY do we assess ? • To ensure competence • As a means of academic competition In practice: the scope…

9. WHY do we assess ? • Formative: to give feedback and advice • When the cook tastes the soup, that's formative • Summative: to grade • When the guests taste the soup, that's summative • Qualificative: to graduate In practice: the purpose When should we assess ?

10. WHEN should we assess ? Professional authenticity Performance or hands on assessment Does Shows how Written, Oral or Computer based assessment Knows how Knows Miller GE. The assessment of clinical skills/competence/performance. Academic Medicine (Supplement) 1990; 65: S63-S7.

11. Pass/Fail F GRADES 85% 93% WHEN should we assess ? Certified 7/10 A C+ B S 4/5 14/20 A- 67% D C Qualified Unsatisfactory Assessment is continuous (Life-Long-Learning)

12. HOW do we assess ? • Assessment • The process of measuring something with the purpose of assigning a numerical value. • Scoring • The procedure of assigning a numerical value to assessment task. • Evaluation • The process of determining the worth of something in relation to established benchmarks using assessment information.

13. Assessment Concerns • Validity -- Is the test assessing what’s intended? • Are test items based on stated objectives? • Are test items properly constructed? • Difficulty -- Are questions too easy or too hard? (e.g., 30% to 70% of students should answer a given item correctly) • Discriminability -- Are the performance on individual test items positively correlated with overall student performances? (e.g., only best students do well on most difficult questions)

14. Evaluation Types • Criterion-referenced evaluation -- student performance is assessed against a set of predetermined standards • Norm-referenced evaluation -- student performance is assessed relative to the other students

15. Criterion-Referenced Eval’s • Based on a predetermined (minimum) set of criteria. • For instance, • 90% and up = A • 80% to 89.99% = B • 70% to 79.99% = C • 60% to 69.99% = D • 59.99% and below = F

16. Criterion-Referenced Eval’s • Pros: - Sets minimum performance expectations. - Demonstrate what students can and cannot do in relation to important content-area standards • Cons: - Some times it’s hard to know just where to set boundary conditions - Lack of comparison data with other students and/or schools.

17. Norm-referenced Evaluation • Based upon the assumption of a standard normal (Gaussian) distribution with n > 30. • Employs the z score: • A = top 10% (z > +1.28) • B = next 20% (+0.53 < z < +1.28) • C = central 40% (-0.53 < z < +0.53) • D = next 20% (-1.28 < z < -0.53) • F = bottom 10% (z < -1.28)

18. ECTS = European Credit Transfer System

19. Norm-referenced Evaluation • Pros: - Ensures a “spread” between top and bottom of the class for clear grade setting - Shows student performance relative to group • Cons: Top and bottom performances can sometimes be very close Dispenses with absolute criteria for performance Being above average does not necessarily imply “A”performance In a group with great performance, some will be ensured an “F.”

20. Criterion and Norm Evaluation • Norm-Referenced Ensures a competitive classroom atmosphere Assumes a standard normal distribution Small-group statistics a problem Assumes “this” class like all others • Criterion-Referenced Allows for a cooperative classroom atmosphere No assumptions about form of distribution Small-group statistics not a problem Difficult to know just where to set criteria

21. Academic Grading ! 1 Country = 1 Grading Scale Need to be harmonized In Sweden

22. WHAT do we measure ? To test not only presence of knowledge …but also the application of knowledge In principle…

23. Model of clinical assessment • Certification of competence - pass / fail a state (and legal) requirement • Grading in rank order for employment / placement purposes • A competition for the award of a prize In principle…a three-fold aim

24. Model of clinical assessment Measurement of: • adequacy of basic clinical skills • ability to interpret clinical findings • facility of communication in practical settings • ability to think analytically about diagnosis • ability to discuss management logically In practice…

25. Practical steps for assessment • At least 6 encounters with different clinical situations (Internal Medicine, Surgery, Pediatrics, Ob-Gyn,…) • At least 2 examiners at every encounter, each examiner giving an individual assessment • Highly structured examination and detailed assessment of skills • Examiners from other Universities for process evaluation and quality control

26. Set-points for Testing • Attitude to patient • Actual examination skills • Presentation of findings • Clinical judgment Clinical examination of patients

27. Methodology-related Issues • Reliability • Validity • Practicality • Transparency

28. Organisational Issues • Resources available • Student numbers • Patient availability • Invasive procedures • Scheduling

29. Role of Grading • Is there need for grading ? • Selection process for employment • Selection for postgraduate training • Quality of assessment method and performance of candidates

30. Final question :What happens to candidates who fail ? • Review of performance – a formative exercise • Counselling at a personal level • Specific attention and individual training • Repeat assessment after a period of time

31. In conclusion • Assessment of student (a future peer) is a continuous and complex process • Assessment of clinical skills is Teacher, Medical School, National and Culture-Dependent • There is place for widespread application of agreed standard methods to assess - clinical competences in core curricula - additional competences essential to individual practice • Teachers must give example of what are Knowledge, Know-How and Life-Skills