Implementation of a Licensing Examination

1. Implementation of a Licensing Examination Steven Lillis

2. Why assess? Find those who are not yet competent Demonstrate to others that adequate systems are in place that promote public safety Reflection on appropriate standards for practice Feedback on teaching and curriculum

3. Why assess? Find those who are not yet competent Demonstrate to others that adequate systems are in place that promote public safety Reflection on appropriate standards for practice Feedback on teaching and curriculum

4. Purpose The purpose anchors the examination. Mission statement of a company “The mission statement should be a clear and succinct representation of the enterprise's purpose for existence”

5. Purpose Examples: NZREX: “…candidates are competent to enter a period of provisional registration in New Zealand, during which time they will be further assessed.” RNZCGP: “…to be able to safely practice in advanced vocational education”

6. Purpose PLAB: “Passing the PLAB test demonstrates that you have reached the minimum standard required to practise safely in the UK”

7. Purpose (MCCQE)...assesses the competence of candidates who have obtained the medical degree, for entry into supervised clinical practice in postgraduate training programs, with respect to their knowledge, clinical skills and attitudes as outlined by the MCC objectives.

8. Purpose Needs to reflect: Who is being examined (IMG with WHO recognised qualification) Why they are being examined (safety, competence) What standard they are being examined at (further supervision in a training scheme)

9. Purpose

10. Find those who are not yet competent What is the material to be tested? How are you going to test it? How do you know which candidates are acceptable?

11. Find those who are not yet competent What is the material to be tested? How are you going to test it? How do you know which candidates are acceptable?

12. What is the material to be tested? The crucial importance of blueprinting…… “An accurate image of a product”

13. What is the material to be tested? The crucial importance of blueprinting…… “An accurate image of a product” “A sufficiently detailed plan that allows construction”

16. Domains Cardiovascular Neurology Women's health Child health Respiratory Genitourinary etc etc etc Competencies Communication Problem Prioritisation Acute serious deterioration History Physical Examination Tests Procedures AXES

17. The 3rd axis There will be a minimum of • 2 cases / 10% of the questions on elderly care • 3 cases / 17 % of the questions on women's health issues • 1 case / 5 % of the questions on preventative health issues

18. Sample from blueprint... “The competencies that will be tested are: • Communication skills (2 cases) • Prioritisation of management (4 cases) • Recognition of acute serious deterioration and its management (2 cases), • History taking (4 cases) • Physical examination (3 cases) and • Interpretation of tests and procedures (1 case). There will be at least one case on child health, one case on mental health and one case on women’s health.”

19. Sample from blueprint • All domains are covered at least once • All competencies are covered at least once

20. Purpose Blueprint

21. Find those who are not yet competent What is the material to be tested? How are you going to test it? How do you know which candidates are acceptable?

22. Miller’s triangle

23. Practicality for licensure examinations... • A written test…for material that is reasonable for this format • MCQ • EMQ • KFP • An OSCE…for material that is reasonable for this format

24. Two different MCQs The first line treatment of staphylococcus skin infections is: • Penicillin • Co-trimoxazole • Flucloxacillin • Cefaclor • Nitrofurantoin

25. Two different MCQs A 25 year old man presents 3 days after a puncture wound to his leg. Examination shows his temperature at 38.2 C and the appearance of his leg is:

26. Two different MCQs Ass well as surgical drainage, the most appropriate antibiotic to prescribe is: • Penicillin • Co-trimoxazole • Flucloxacillin • Cefaclor • Nitrofurantoin

29. Purpose Blueprint Implement

30. Find those who are not yet competent What is the material to be tested? How are you going to test it? How do you know which candidates are acceptable?

31. Which candidates are acceptable? The importance of the ‘cut’ score Implies the ability to rank candidates

32. Which candidates are acceptable? Growing acceptance of combination of two methods

33. Which candidates are acceptable? Growing acceptance of combination of two methods Angoff process: a group of experts who are familiar with the expected standard making a judgment about how a borderline candidate would score in each question Used in written or static OSCE stations

34. Which candidates are acceptable? Growing acceptance of combination of two methods Contrasting groups: each candidate gets a score on a station as well as an assessment of “pass”…”borderline”…..”Fail” P B F Score

35. Which candidates are acceptable? Growing acceptance of combination of two methods Contrasting groups: each candidate gets a score on a station as well as an assessment of “pass”…”borderline”…..”Fail” B Score

36. Which candidates are acceptable? Growing acceptance of combination of two methods Contrasting groups: each candidate gets a score on a station as well as an assessment of “pass”…”borderline”…..”Fail” The mean of the borderline cases is taken…… Score

37. Which candidates are acceptable? Growing acceptance of combination of two methods Contrasting groups: each candidate gets a score on a station as well as an assessment of “pass”…”borderline”…..”Fail” The mean of the borderline cases is taken…… …and readjusted for the error in the examination Score

38. Why assess? Find those who are not yet competent Demonstrate to others that adequate systems are in place that promote public safety Reflection on appropriate standards for practice Feedback on teaching and curriculum

39. External confidence Transparency and communication Release the blueprint Openness around expectations of candidates when in the examination Give pass rates Give methods of devising the ‘cut score’ Follow up of successful candidates Responsibility to failing candidates

41. Why assess? Find those who are not yet competent Demonstrate to others that adequate systems are in place that promote public safety Reflection on appropriate standards for practice Feedback on teaching and curriculum

42. Reflection on appropriate standards for practice Method of engaging many experienced clinicians in reflecting on appropriate standards for the profession “What counts as competence and by whom is something that the community negotiates over time; indeed, it is this negotiation that defines the community.” Eckert P, Wenger, E. What is the role of power in sociolinguistic variation? Journal of Sociolinguistics 9/4 2005 582-589

43. Purpose Blueprint Implement Competence

44. Why assess? Find those who are not yet competent Demonstrate to others that adequate systems are in place that promote public safety Reflection on appropriate standards for practice Feedback on teaching and curriculum

45. Feedback on teaching and curriculum Assessment drives learning What information is given to potential candidates and why is this information given? Default curriculum May assist in courses aimed at exam preparation

46. Purpose Blueprint Implement Safe medical practice Default Curriculum Competence

47. Purpose Blueprint Implement Safe medical practice Default Curriculum Competence

48. Purpose Blueprint Implement Safe medical practice Default Curriculum Competence An iterative process where the examination both tests and informs the profession about standards of competence