Workplace-based Assessment and Feedback

1. Workplace-based Assessment and Feedback Dr. Kevin West Director of Examinations and Assessment

2. Why change? • “Something must be done!” • Training more geared to “Good Medical Practice” • “Bad” doctors must be weeded out before let loose on public • Assessments other than exams necessary

3. PMETB/GMC and WBA • PMETB established by Parliament 2003 • Started operation 30th September 2005 but groundwork already underway • Established standards for assessment • Under GMC from April 2010

4. PMETB strategy on assessment • PMETB principles for assessment 2004 • Nine principles for assessment systems • Now rewritten as 17 standards for curricula and assessment systems • Principles based on “overarching strategy” • Workplace-based assessment • Examinations of knowledge • Examinations of clinical skills • Specialty curricula and assessment systems now approved by GMC

5. Value of multiple assessments • Key is several modes of assessment – triangulation • A single exam system does not provide adequate coverage of clinical practice • WBA is not enough on it own (Assessment System Guidance – PMETB)

6. The College Assessment system: Miller’s Pyramid WBA – MSF, CbD Does Action WBA e.g. Mini-CEX, DOPS Shows FRCPath Part 2 Practical Performance Knows how Competence Knows FRCPath Part 1 Knowledge

7. Workplace-based Assessment • Assessment of performance in the workplace and measured there • Assessed locally by supervisors as well as staff of all grades • Assessment occurs during normal work immediately followed by feedback so trainee receives continuous feedback on their routine performance • Results recorded in trainee portfolio for annual review - important for trainee development • Starts the process of CPD and revalidation

8. Workplace-based Assessment • The College uses workplace-based assessments similar to other specialties with some adaptations • Only Chemical Pathology and Medical Microbiology use Mini-CEX • DOPs and CBD are relevant to all specialties • Histopathology pioneered Evaluation of Clinical Events for assessment of trainee interactions with other medical teams • The College uses a specialty-specific MSF

9. Essential elements of WBPA • Requires assessment by as many observers as possible for reliability (at least 7 for MSF and 6-8 for other tools) • Aim is to show progression so repeating a failed assessment is perfectly acceptable but best if both recorded and available (doesn’t happen in practice) • Aim is to show satisfactory progression through stages of training to full professional outcome • Requires professional judgement

10. Problems with WBA • “I just wanted to ensure (the trainee) had sufficient satisfactory assessments to ensure I didn’t need to do any more” • “I don’t want to be responsible for their failing this year and repeating it” (here) • “I don’t feel I can judge whether it was satisfactory or not” • “I’ll reserve my comments for the educational supervisor’s report”

11. College Assessment System • Workplace-based assessment • Year 1 assessment • FRCPath Part 1 (test of knowledge) • FRCPath Part 2 (test of skills and applied knowledge) • Feedback on regular basis (WBA) • Educational supervisor’s report • ARCP (Annual Review of Competency Progression)

12. Examinations and Assessment • Examinations are one form of assessment • Assessments are: • formative (involve feedback and intended to guide training) or • summative (assess whether reach given standard) e.g. driving test, FRCPath • All assessments should be “blueprinted” to the curriculum to demonstrate that the most appropriate method is being used

13. How Workplace-based Assessment fits with FRCPath • Workplace-based assessment depends on assessment by local staff, examinations rely on assessment by those who have had no contact with the trainee • Examinations provide an independent assessment of knowledge and practical skills to national standards • Workplace-based assessment provides a local insight into trainees’ performance in routine tasks and interaction with staff (and patients)

14. How Workplace-based Assessment fits with FRCPath • WBA should provide insight into attitudes and functioning day to day – but does it succeed? • Workplace-based assessment is a local process governed by deaneries to GMC and College standards • Comments, complaints and appeals about WBA go to deanery • WBA is a continuum with F1 and F2 assessment and is being used in medical schools

15. Workplace-based Assessment – Advantages • Measurement occurs in more authentic environment • Assessment may be retrospective review of actual routine day to day performance • Trainees can take assessment in their own time • The assessment tools should provide extensive feedback to the trainee (formative) • Use of computerised forms should make them easy to record

16. Workplace-based Assessment – Disadvantages • Computerised forms can over-simplify • Do we recognise excellence? • A robust rating can only occur if there are a number of independent observers – which can be difficult in small specialties • Assessors may be unwilling to be critical of trainees they interact with on a day to day basis - are we just too nice? • Summative v formative – more later

17. Annual progression during training

18. The Training Environment • Workplace-based assessment utilises the skills of a wide variety of professional groups as assessors: medical, nursing, BMS, clinical scientist, ancillary staff • The College cannot train or quality assure all assessors • Training centres are recommended to develop a training environment where those with the skills needed for assessment are recognised

19. Undertaking assessments Opportunities and scenarios Assessment process Feedback Guidance notes for assessors and trainees Curriculum Schedules of scenario complexity Assessment form & feedback guidance Personal Development Plan Portfolio Assessment form Standards for Assessment

20. “Overview of WBA” document • Customised for each specialty • Similar content • Introduction to WBA • Outline of WBA tools • Who can be an assessor? • Standards for assessment • Record keeping • Examples of scenarios • Hyperlinks to relevant guidance notes and forms

21. Guidance note for each WBA tool • Aimed at assessors and trainees • Customised for each specialty and each tool • Content: • Explanation of the WBA tool • Suitable cases • Who can be an assessor • How the assessment works • Standards for assessment • Outcome of assessment • Feedback • Record keeping

22. Opportunities & scenarios Opportunities and scenarios Assessment process Feedback Guidance notes for assessors and trainees Curriculum Schedules of scenario complexity Assessment form & feedback guidance Personal Development Plan Portfolio Assessment form Standards for Assessment

23. How to use opportunities for workplace-based assessment • Numerous opportunities/scenarios available - refer to guidance notes • Be realistic about assessment • duration • complexity • appropriateness • Sampling

24. Undertaking assessments Opportunities and scenarios Assessment process Feedback Guidance notes for assessors and trainees Curriculum Schedules of scenario complexity Assessment form & feedback guidance Personal Development Plan Portfolio Assessment form Standards for Assessment

25. Conducting the assessment • Ensure both you and the trainee have read and understood the guidance • Check appropriateness of event • Adequate time to undertake assessment and provide feedback • Appropriate place for feedback • Have the assessment form with you and make notes on it

26. The assessment form

27. Information about the assessment …

28. Benefits of using the assessment form • Gives structure to the encounter • Encourages a systematic approach • Generic forms, so some items will be not applicable (“unable to comment”) • Reduces bias • Helpful in providing feedback • Helpful in identifying developmental points • Helpful in commenting on behaviour

29. The future of WBA • GMC has just approved new Foundation Curriculum • WBA will be Supervised Learning Events • Formative - no pass/fail • Feedback will be unavoidable • No hiding behind a set of ticks

30. Feedback Opportunities and scenarios Assessment process Feedback Guidance notes for assessors and trainees Curriculum Schedules of scenario complexity Assessment form & feedback guidance Personal Development Plan Portfolio Assessment form Standards for Assessment

31. What is Feedback? Information given in response to a product, performance etc., used as a basis for improvement. Modification or control of a process or system by its results or effects.

32. What is Feedback? Information given in response to a product, performance etc., used as a basis for improvement. Modification or control of a process or system by its results or effects. A loud whining sound.

33. Psychologists’ view of feedback

34. Giving feedback – why? • Feedback • one of the most important ways in which we learn • one of the most important aspects of teaching and relatively neglected in medical education • an essential component of workplace based assessment which is intended to be formative • Emphasised by GMC

35. Giving feedback • Why is feedback so powerful? • It connects with feelings and helps us reflect • When did you last get some good feedback • how did you feel afterwards? • Do you remember how you felt when someone was really critical of your performance?

36. Giving feedback - skills • Traditionally medical education has used destructive feedback • Times are changing • Greater emphasis on • Training the trainers • Teaching the Teachers • Educational Supervisor courses

37. Giving feedback - skills • Praising is easy, makes people feel better but needs to be constructive – “That was OK but…” is not positive! • Skill requires: • detailed evaluation • active listening • constructive criticism

38. Feedback skills - active listening • Non-verbal behaviour • Focus on speaker’s experience of activity • Acceptance (don’t dismiss ideas) • Empathy – relate to their feelings • Probing – need to get to what is behind what is said • Check and summarise • Advise – e.g. alternatives

39. Feedback skills - Constructive criticism • Let the trainee speak first • Begin with the good points • Plan a solution for each problem • Be sensitive to person • Show interest – use your experience to coach and counsel not score points • Be constructive and encourage trainee to participate in finding solution

40. Feedback skills - Constructive criticism • Deal with one point at a time • Criticise the act (deal with facts) not the individual • Do not hyperbolise • Don’t joke – it can be taken the wrong way • Don’t compare with other people • Be constructive, the aim is to improve performance

41. A simple framework for feedback • Acknowledge the role of feelings • Start with description before moving on to evaluation and action planning • Allow trainee to speak first at each stage • Consider positive aspects before negative • Plan for future action and offer support

42. Summary • Understand the assessment tools and process • Use your existing people, places and opportunities • Train the trainers/assessors and trainees • Allay their anxieties • Use it as a formative process

43. Summary • Encourage trainees to plan WBA • Schedule WBA throughout attachments • Avoid last minute panic • Look out for hawks and doves • Trainees will know who they are! • Give proper feedback • Can be difficult with weak trainees

44. Clear vision required