Clinical Research Centre For Health Professions

1. University of Brighton Dr Vinette Cross Senior Research Fellow Clinical Research Centre For Health Professions

2. Key concepts in practice-based assessment

3. What is assessment? Assessment is the process of measuring progress/achievement against agreed criteria (Chambers et al 2003) ‘Appraisal’ is an official or formal recognition of the strengths and weaknesses of someone or something (Collins COBUILD English Dictionary 1999)

4. What should assessment be? • An enabling process that: • provides personal and professional insights; • influences and improves learning and performance; • enhances quality; • stimulates change.

5. What is practice-based assessment? Practice-based assessment is asocial interactionbetween individuals, in the workplace, with all that that implies. Stakeholders in practice-based assessment havediffering expectationsof the process, which assessors are required to meet. Practice-based assessors are important gate-keepers of professional standards.

6. Important questions about practice-based assessment…is it Legal -able to withstand legal challenge? Decent -acceptable andappropriate? Honest -open and transparent? Truthful -an accurate measure? Does it protect the consumer?

7. Important questions about assessment instruments Does our assessment instrument measure something that actually exists and that reflects our priorities?(construct validity) Is the content of the assessment instrument sufficiently representative of elements that define our priorities? (content validity) Is our assessment instrument something that we can use (practicability) and relate to personally as practitioners (credibility)? (face validity)

8. Is our assessment reliable? Does the assessment instrument discriminate between levels of achievement? Number of scale points Range of marks • Do the assessment outcomes demonstrate • intra-rater reliability? • inter-rater reliability?

9. Does our approach to assessment make an effective contribution to learning for individual students? ourselves as individual educators and assessors? our intra-professional teams? our inter-professional teams? Does it protect the users of our service?

10. As assessors, what questions should we ask ourselves?

11. Professional competence - what are our priorities? (Clinicians: n = 85 Academics: n = 98) Safe Good communicator Aware of limitations Applies theory to practice Eager to learn Good practical skills Empathetic Critical thinker Conscientious Good knowledge base Independent learner Effective/efficient

12. Academics Clinicians and academics - comparing priorities 1 0.9 0.8 Clinicians 0.7 0.6 0.5 Order of importance 0.4 0.3 0.2 0.1 0 Safe Effective Empathetic Conscientious Eager to learn Knowledgeable Critical thinker Aware of limitations Good communicator Able to apply theory Good practical skills Independent learner

13. A B C D E F G H I J K L • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • Professional competence - assessment criteria (183 respondents) 89 assessable & accessible behaviours (98 respondents)

14. Content validity How important are the behaviours (criteria) to effective practice in different specialties (and professions)? Are there areas of commonality that provide a consensus for assessment across specialties (and professions)?

15. FACTOR 1 (Factor loadings 0.7-0.5  = 0.9) 25 Organising workload efficiently 47 Managing time efficiently 59 Predicting clinical outcomes based on experience 26 Completing tasks fully and properly 65 Relating clinical signs to pathology 12 Working safely and effectively without supervision 44 Identifying salient points of assessment 81 Recognising typical patterns of clinical presentation 24 Setting appropriate treatment priorities 03 Positioning optimally for treatment and assessment 33 Performing treatment techniques accurately

16. FACTOR 2 (Factor loadings 0.7-0.6  = 0.8) 80 Always giving standard warnings 89 Always checking for contraindications 73 Always checking for adverse effects 87 Always checking relevant information before treatment 83 Maintaining a portfolio as evidence of learning FACTOR 4 (Factor loadings 0.7-0.5  = 0.7) 31 Adhering to local health and safety policies 40 Checking functioning and safety of equipment 05 Identifying and clearing hazards in environment 07 Asking for advice or help when needed

17. FACTOR 3 (Factor loadings 0.7-0.6  = 0.8) 61 Being flexible and adaptable 46 Keeping accurate, concise and thorough records 52 Speaking clearly and audibly 06 Writing reports legibly, usefully and articulately 32 Showing interest in patients’ lives and activities 04 Listening attentively with appropriate eye contact 62 Dressing professionally

18. FACTOR 5 (Factor loadings 0.7-0.5  = 0.8) 15 Justifying decisions on basis of research evidence where posible 16 Accessing databases to inform practice 35 Practising new techniques when opportunity arises 02 Organising time for reading and reflection 21 Referring to own reading in discussions about practice 28 Explaining why interventions might be inappropriate

19. FACTOR 6 (Factor loadings 0.6-0.5  = 0.9) 71 Giving talks/presentations to professional colleagues 54 Demonstrating techniques to professional colleagues 30 Participating in professional dialogue within MDT 85 Reflecting with peers on learning and practice 50 Trying to measure clinical outcomes for patients 51 Testing professional knowledge through discussion 72 Selecting outcome measures related to own practice 88 Challenging status quo in informed and constructive way 36 Understanding interface between PT and role of others 82 Asking others’ opinions of own work as appropriate

20. How can we collect evidence of competence? •   Direct observation •   Questioning •   Oral/viva voce examination •   Structured practical assessment •   Portfolio assessment • Clinical audit

21. Using a rating scale to measure performance 5 = far exceeds expectations 4 = exceeds expectations 3 = meets expectations standard 2 = falls below expectations 1 = falls well below expectations

22. Facilitation Model Patrimony Model Bureaucratic Model Mode of teaching Apprenticeship Formal training organised bureaucratically Learning communities Mode of learning From master to novice Rule-bound & standard procedures Learning partnership & collaboration Source of professional identity Social/professional status Bureaucratic machine Organisational mission & core values Learning outcomes Reproduction of traditional knowledge & skills Prescribed skills & competencies Empowerment & collective agency Nature of professional identity Expert practitioner Competent, ‘fit for purpose’ practitioner Capable lifelong learner Models of workplace learning (Cross, Hicks, Parle & Field 2006)

23. Communities of practice Situated learning Negotiated meanings inside and outside Legitimate peripheral participation Full participation Power relations and inequalities

24. Competence is about…. • possession of specialist knowledge & skills relevant to a specific context • control of standards • delivering the present based on past performance • fitness for (usually someone else’s) purpose

25. Capability is aboutjustified confidence based on…. • specialist knowledge & skills • ability to manage own learning & learn from experience • power to perform under stress • capacity for dealing with values issues • judging the fitness of the purpose itself • taking effective action in unfamiliar and changing circumstances • imagining the future and bringing it about • courage, risk-taking, intuition

26. Learning for assessment Assessment for learning Models of learning Accountability Evidence Learners Educators Organisation Competence Public Learners Collaborative learning Capability Practice-based educators Peers Inter-professional Safety & Justice Self-Actualisation

27. COLLABORATIVE ASSESSMENT Increasing learner responsibility for assessment Educator decides criteria, makes judgements, gives feedback Educator & learner negotiate criteria and methods Formative feedback from peers affects learner’s actions Learner decides criteria, makes judgements and decides on action SELF ASSESSMENT UNILATERAL ASSESSMENT

28. Assessment: a socially constructed act SOCIAL FIELD Observed relations between positions Struggle for power and resources • Positions in the field determined on the basis of CAPITAL • (Economic, cultural, social, intellectual) • Most valued capital defines • Institutional and individual positions in the field • Common professional vision • Legitimation – ‘feel for the game’ (Bourdieu 1990)

29. Assessment: a socially constructed act ASSESSORS’ INTERPRETIVE ACTS Are influenced by: EXPERIENCE Professional socialisation to the common vision – Particular kinds of experience are privileged Sustain or challenge: Belief systems Identities and interpersonal relationships (Butler Shay 2004)

30. How do we construct our identity as assessors? • What are our values and beliefs about learning? • What is our vision for the future of practice?