Standard setting Determining the pass mark - OSCEs

1. Standard setting Determining the pass mark - OSCEs

2. The Old Way (1) …..I think that the pass mark in this is exam is probably about here….. there is a natural break I can see dividing the bottom from the rest

3. The Old Way (2) …..The pass mark is 60%.

4. EMQ Reports OSCE MEQ 360 Orals Log book Assessments

5. EMQ Reports OSCE MEQ 360 Orals Log book Assessments Match Outcome Objectives

6. EMQ Reports OSCE MEQ 360 Orals Log book Assessments Match Outcome Objectives Integrated

7. EMQ Reports OSCE MEQ 360 Orals Log book Assessments Match Outcome Objectives Integrated Clinical Competencies tested throughout

8. EMQ Reports OSCE MEQ 360 Orals Log book Assessments Progressive testing Match Outcome Objectives Integrated Clinical Competencies tested throughout

9. EMQ Reports OSCE MEQ 360 Orals Log book Assessments House style Progressive testing Match Outcome Objectives Integrated Clinical Competencies tested throughout

10. EMQ Reports OSCE MEQ 360 Orals Log book Assessments From approved list House style Progressive testing Match Outcome Objectives Integrated Clinical Competencies tested throughout

11. EMQ Reports OSCE MEQ 360 Orals Log book Assessments From approved list House style Student information Progressive testing Match Outcome Objectives Integrated Clinical Competencies tested throughout

12. EMQ Reports OSCE MEQ 360 Orals Log book Assessments External Examiners From approved list House style Student information Progressive testing Match Outcome Objectives Integrated Clinical Competencies tested throughout

13. Standard setting Knowledge Skills

14. Standard setting Angoff Hofstee Knowledge Borderline method Skills

15. Borderline procedure Used for clinical (OSCE) examinations

17. Borderline procedure Used for clinical (OSCE) examinations Examiners score the students’s performance at the station e.g 17/20 Examiners judge the overall performance clear pass / borderline / clear fail Mark sheets rated borderline identified and the scores of borderline students averaged Process repeated for each station Calculate the median borderline score for all stations

18. Borderline Method

19. Station 15 (static) Oral Lesions Pass/Fail A A 1 50 A 2 A 3 40 A 30 Count A A 20 A A 10 A A A A A A A A A A A A A A A A 0 0.00 1.00 2.00 3.00 4.00 5.00 6.00 7.00 8.00 9.00 10.00 Score

20. pass Median Borderline score 58.41 Borderline group fail

21. pass Median Borderline score 58.41 Borderline group How wide should this band be? fail

22. pass Median Borderline score 58.41 Borderline group How wide should this band be? fail +/- 1 - standard deviation - standard error - or what?

23. The Standard Error of Measurement • depends on the reliability of the test (R) • depends on the standard deviation of the • test (SD) • SEM = SD 1 - R • acts as a confidence interval in high • stakes situations

24. The Standard Error of Measurement • depends on the reliability of the test (R) • depends on the standard deviation of the • test (SD) • SEM = SD 1 - R • acts as a confidence interval in high • stakes situations R = Variance Variance x Error

25. Pass Score 60.93 Pass } + 1 SEM Median Borderline score 58.41 Borderline group }- 1 SEM Fail Fail score 55.88

27. Internal Reliability of the ExamCronbach’s Alpha Item / Stations  Population (students)  Split half Correlations (SH) Cronbach’s alpha Shows whether randomly split halves of the exam by item vary together

28. Grading System Top Mark Distinction Good Pass Pass Pass Pass Mark Borderline Borderline Mark Fail Borderline Fail

29. Reducing measurement error 1. Increase reliability of exam 2. Compromise with feasibility/cost * Content specificity * More items in the exam * Blueprinting * Quality assurance of item writing * Training examiners/standardised patients * Feedback from exam performance