Specialty Selection Current Practice

1. Specialty SelectionCurrent Practice Jane Dacre Luci Etheridge Alison Sturrock Hilary Spencer Fiona Patterson

2. Objectives • Summarise current activity regarding use of MMTs (CPS & SJTs) in specialty selection • Share evidence and learning from pilots conducted to date in individual specialties • Explore areas of commonality and divergence between specialties with regard to MMTs

3. Intended Outcomes • A shared understanding of current thinking on the use of MMTs in postgraduate specialty selection (a.m.) • Shared ideas about the way forward in the use of MMTs in specialty selection (p.m.)

4. Background • Pre 2002: candidates applied to each deanery & specialty individually • 2003: reforms recommended by Liam Donaldson in ‘Unfinished Business’ led to MMC programme • 2005: MMC introduced • 2007: MTAS launched, failed & was abandoned • 2008: Specialties & deaneries reverted to using mixed methods

5. Discussion What was wrong with the old system?

6. What was wrong with the old system? • Varied geographically and by specialty • Required major effort from all concerned • Unfair to many candidates, ineffective for Trusts • Generally failed to produce good results

7. In an ideal world …. • How would you select? • What criteria would you use? • Why would you choose this method?

8. Current practices (national) • GP: well established NRO • Public health: well established NRO • O&G: Developing national system • CMT: new NRO • Histopathology: new NRO

9. Current practices (local) • Anaesthesia & ACCS: applications to deaneries, longstanding SJT pilot • Surgery: no national application system, looking into using SJTs in selection • Radiology: currently local, national next year

10. Possible new selection stages

11. Components • Application form • Longlisting • Shortlisting • MMT • Interview/selection centre • Offers made • Accept / hold / reject cycle • Round 2 : repeat for unsuccessful candidates and IMGs

12. Discussion What are the unresolved issues for each of the possible selection components ?

13. Unresolved issues • White space not popular • Time & effort required for shortlisting • Correlation with interview scores • Geography v. specialty • Competition ratios • Standardisation of selection centres • PMETB assessment guidance Can MMTs help?

14. Machine Markable Tests • Cost effective • Strong validity evidence, including incremental validity • Fairness and positive applicant reactions • Familiar to most ST1 applicants • Flexible: computer or paper based, managed or self run, invigilated or anywhere tests

15. Machine Markable Tests • Clinical Problem Solving tests: • assess doctor’s ability to apply knowledge • usually single best answer or extended matching format • Situational Judgement tests: • assess doctor’s ability to deal with complex situations • multiple answers, either ranked by importance or select three most appropriate

16. Recent Specialty Pilots inc MMTs • Acute specialties Tom Gale • CMT Liz Berkin • GP Bill Irish • Paediatrics Colin Campbell • Public health Nora Pashayan • O&G Maggie Blott • Surgery David Rowley • Multispecialty Neil Douglas

17. Clinical Problem Solving Tests • Tests of ability to use acquired knowledge in a clinical scenario • Not just factual recall • Assesses problem solving ability

18. CPS - Example A 23 year old woman with SLE is mildly jaundiced. Her haemoglobin is 8.2 g/dL, MCV 115fL, and her blood film shows spherocytes and polychromasia Which single test is most likely to confirm the diagnosis? A. antibody screen B. anti-DNA antibody titre C. direct Coombs test D. platelet antibody test E. rhesus blood group

19. Situational Judgement Tests • Assesses judgement in work relevant situations • Presents challenging situations likely to be encountered at work • Makes judgements about possible responses • Scored against predetermined key • Used often in high volume selection • Validity (predictive & incremental) is well established

20. Public Health: Professional integrity Coping with pressure Managing others & team Organisation & planning ACCS: Professional integrity Vigilance & situational awareness Empathy & sensitivity Coping with pressure Surgery: Coping with pressure Decision making Professional integrity Managing others & team Paediatrics: Empathy & sensitivity Professional integrity Team involvement Coping with pressure Job analysis: priorities

21. SJTs – Example (FBI) You are shopping when you notice a man robbing the store. What would you do? (choose most likely & least likely response)   A.   Leave the store as quickly as possible and call the police   B.   Try to apprehend the robber yourself   C.   Follow the man and call the police as soon as he appears settled somewhere   D.   Nothing, as you do not wish to get involved in the matter.

22. Predictive validity of shortlisting ** p<.001CPS – Clinical Problem-Solving test. SJT – Situational Judgement test • SJT is the best single predictor for shortlisting • Both tests are strong predictors & show incremental validity over the other

23. Research evidence Medical Education, 2009

24. Current consensus? • The old methods are no longer fit for purpose • Selection methods vary depending on specialty & deanery • Most stakeholders want ST1 selection methods to change • Application forms, portfolios, MMTs & SCs can all contribute

25. Discussion • Do we still favour our chosen methods? • What might we want to change? • Where should the sector be heading?

26. Selection….testing higher order cognitive skills