Self- Assessment

1. Self- Assessment Karen V. Mann PhD ‘Fall Together’ Department of Family Medicine University of Calgary October 22nd, 2013

2. Objectives: Participants will: • Explore the concept of self- assessment: how do people know “how they are doing”? • Discuss an evidence-based model for understanding informed self-assessment • Discuss conditions that can facilitate informed self-assessment and use of external feedback. • Consider the implications and apply these conditions and approaches to practical examples. • Identify and develop practical educational strategies to enhance informed self-assessment and feedback adoption, both for learners and themselves.

3. How do we know “how we are doing?” • Consider how you make judgments about your performance as a physician, or teacher: What sources of information do you use? • What challenges do you experience in assessing your own performance ?

4. The self-assessment literature • “in a majority of the relevant studies, physicians do not appear to accurately self-assess …” (Davis, et al., 2006) • “…the most important educational activity related to self-assessment … should entail helping people overcome their personal belief that they can rely on it.” (Eva & Regehr, 2008) • What’s worse, those with the least proficiency in a domain are the least accurate self-assessors (Kruger, Dunning 1999; Violato, Lockyer 2006)

5. Background • Self-assessing on one’s own is a little like…

6. Understanding informed self-assessment: Social psychological theory • “The person and the situation; perspectives of social psychology” (Ross, Nisbett1991): • Situational factors are powerful influences of behaviour. • It is actually the person’s perceptions of situational factors which are influential.” • The social system and individual cognition/ perceptions are in a dynamic state of flux. Therefore making inferences about individual behaviouris complex/ difficult.

7. Understanding informed self-assessment: Constructivist learning theory • Learning takes place in contexts • Learners form or construct much of what they understand as a function of their • Experiences in situations • Social interactions

8. Learning is personal growth, self-actualization Includes affective and cognitive domains Focuses on personal goals and accomplishments Learning is self-directed (which assumes some degree of self-assessment) Understanding informed self-assessment: Humanist learning theory

9. Honest and accurate external feedback is crucial to facilitate improvement and to accurate self-assessment (Larsen, et al., 2008; Boehler, et al., 2007; Rees,Shepherd, 2005)

10. Background: The terms we use • Self-assessment: “a global judgement of one’s ability in a particular domain”(Eva & Regehr, 2010) • Informed Self-Assessment: “a set of processes through which individuals use external and internal data to generate an appraisal of their own performance”. (Sargeant et al, 2011)

11. Informed self-assessment: Summary of research • Funding: • Research grants: Medical Council of Canada • Additional funding: ABIM, Dalhousie University • Research team: • Joan Sargeant PhD, Dalhousie University (CA) • Kevin Eva PhD, McMaster University (CA) • Karen Mann PhD, Dalhousie University (CA) • Heather Armson MD, University of Calgary (CA) • Jocelyn Lockyer PhD, University of Calgary (CA) • Elaine Loney, MSc, Qual Research Consultant (CA) • Cees van der Vleuten, PhD, Universiteit Maastricht (NL) • Timothy Dornan MD, PhD, Manchester University (UK) • Eric Holmboe MD, Am Board Internal Medicine (USA) • Benjamin Chesluk PhD, Am Board Internal Medicine (USA)

12. Informed self-assessment: Research objectives: To understand - • how students, residents and physicians perceive and use self-assessment in clinical learning and practice • their views of activities aimed at guiding self-assessment and factors that influence their use • how external feedback might be better tailored to the needs of health professionals

13. Research design and methods • International qualitative study using focus groups drawing upon principles of grounded theory • Purposive recruitment from identified programs using specific approaches to enhance self-assessment

14. Participating programs

15. Results: Participants • 17 focus groups • 2 for each of 8 programs except 3 for Canadian MSF program • 134 participants • 53 undergraduate/ student learners (3 programs) • 32 postgraduate/ resident learners (2 programs) • 49 practicing doctors/ consultants (3 programs)

16. Results – what we learned: • Self-assessment and “informing” self-assessment are complex activities • They are: • multi-faceted • contextualized • dynamic • non-linear

17. Dimensions of Informed Self Assessment External & Internal Conditions –e.g. External - learning climate, relationships Internal - emotions, confidence, perceived credibility of feedback Sources of Information - External – feedback, people, processes Internal – “self” Responses to Information- Ignore Reject Seek Accept Interpretation of Information - Reflect, compare, calibrate, filter, assimilate Tensions - Within people Between people Within the learning environment

18. Dimensions of Informed Self Assessment External & Internal Conditions –e.g. External - learning climate, relationships Internal - emotions, confidence, perceived credibility of feedback Sources of Information - External – feedback, people, processes Internal – “self” Responses to Information- Ignore Reject Seek Accept Interpretation of Information - Reflect, compare, calibrate, filter, assimilate Tensions - Within people Between people Within the learning environment

19. 1. Sources of information: external (FEEDBACK) I feel like I’m … kind of working under my own steam, and thinking, “I hope I’m doing enough, doing the right thing”. I got a ‘satisfactory’ but it might be nice for someone to give a bit more feedback about how we’re actually doing, because sometimes you feel very alone. (Medical Student B3)

20. Sources of information:external Anytime that you have to interact with a colleague, be it at a consultant level, personal, group practice level or as a teacher, that always provides feedback. It is a general amalgamation of all those things that gives you an assessment of how you are doing. (Physician L8)

21. Sources of information:internal … you are sometimes aware when you are talking to patients that you are vague or are filling in gaps that shouldn’t be gaps. I think we all do that to come across as being coherent, but one should be able to recognize when one is doing that, and fill in the gaps... (Physician K3)

22. Dimensions of Informed Self Assessment External & Internal Conditions –e.g. External - learning climate, relationships Internal - emotions, confidence, perceived credibility of feedback Sources of Information - External – feedback, people, processes Internal – “self” Responses to Information- Ignore Reject Seek Accept Interpretation of Information - Reflect, compare, calibrate, filter, assimilate Tensions - Within people Between people Within the learning environment

23. 2. Interpretation of information (REFLECTION) If at the end of the day, if you haven’t asked yourself some questions, then that’s a problem. If you go through the whole day and think you know all about all the problems that faced you, then you don’t know. You have to ask yourself why, because you can’t know everything. (Physician M5)

24. Dimensions of Informed Self Assessment External & Internal Conditions –e.g. External - learning climate, relationships Internal - emotions, confidence, perceived credibility of feedback Sources of Information - External – feedback, people, processes Internal – “self” Responses to Information- Ignore Reject Seek Accept Interpretation of Information - Reflect, compare, calibrate, filter, assimilate Tensions - Within people Between people Within the learning environment

25. 3. Responses to information • . . . somebody gave me a ‘below-standard’ score for something, and at first you’re like, ‘Oh, my life, I can’t believe it! I’m not fit to be a doctor!!” …then it actually makes you think, ‘yeah, okay, I am weaker in that area, I need to work on it’. And it does kind of give you a scare - ‘Okay, in future that’s something that I’m going to have to do.’ (Postgraduate E6)

26. 3. Responses to information “But for some patients… I thought I’d been keeping up with things, and I found (from the audit) that actually, no… I am missing things.” (Physician P1)

27. Dimensions of Informed Self Assessment External & Internal Conditions –e.g. External - learning climate, relationships Internal - emotions, confidence, perceived credibility of feedback Sources of Information - External – feedback, people, processes Internal – “self” Responses to Information- Ignore Reject Seek Accept Interpretation of Information - Reflect, compare, calibrate, filter, assimilate Tensions - Within people Between people Within the learning environment

28. 4. External and internal conditions/ influences: Climate Because there is that level of trust within the group, I don't mind my peers knowing that I might not know the answer to something. I don't feel they would judge me … (Physician M2)

29. External and internal conditions/ influences: Internal “. . . When you are more confident, you can ask for more detailed feedback.” (Medical student H9)

30. Dimensions of Informed Self Assessment External & Internal Conditions –e.g. External - learning climate, relationships Internal - emotions, confidence, perceived credibility of feedback Sources of Information - External – feedback, people, processes Internal – “self” Responses to Information- Ignore Reject Seek Accept Interpretation of Information - Reflect, compare, calibrate, filter, assimilate Tensions - Within people Between people Within the learning environment

31. 5. Tensions • Overarching tension –”Between people”: • Wishing to learn and improve vs. wishing to appear knowledgeable and confident among peers and superiors • “Tensions in learning environment”: • Providing genuine assessment feedback vs “playing the evaluation game” • “Tensions within self” • Wanting feedback yet fearing disconfirming information

32. 5. Tensions: Within people “… there is this sense of intimidation that you don’t want to speak up in case you might just be deficient in your medical knowledge.” (Physician M3)

33. 5. Tensions: Within learning environment “I know you’ve got to play the game, but that’s how I always feel when I’m doing it. I’m playing the game. I don’t feel like I’m learning and evolving and becoming a better person because I’ve done it.” (Postgraduate E2)

34. Dimensions of Informed Self Assessment External & Internal Conditions –e.g. External - learning climate, relationships Internal - emotions, confidence, perceived credibility of feedback Sources of Information - External – feedback, people, processes Internal – “self” Responses to Information- Ignore Reject Seek Accept Interpretation of Information - Reflect, compare, calibrate, filter, assimilate Tensions - Within people Between people Within the learning environment

35. Summary • Overarching finding: • No one activity was generally effective in informing self-assessment • Effectiveness was moderated by external factors, internal perceptions, tensions • The value of a specific activity was in how it was used, not in the activity itself

36. Question for discussion • What is your experience in helping learners to use self-assessment effectively? • What are the challenges? What have you found helpful?

37. Discussion and activity Considering our discussion of self- assessment, today, and the factors that influence it… • What strategies might you use with your learners? • How would you incorporate it?

38. Helping learners (1) • Providing structured opportunities e.g., MSF, reflection on portfolios • Modeling self-assessment • Asking reflective questions – helping learners to reflect on performance • Providing effective feedback

39. Effective Feedback • Information, provided by an agent (e.g., teacher, peer, book, parent, self, experience) regarding aspects of one’s performance or understanding. • Its goal: to identify the gap between current and desired performance and to identify a route to get there. (Hattie & Timperley, 2007)

40. Effective Feedback: What We Have Learned • Focus feedback on the task not the learner • Provide elaborated feedback to enhance learning • Give feedback in manageable units • Be specific and clear • Make feedback as simple as possible and no simpler (Shute, 2008)

41. Effective feedback: two helpful concepts • Zone of proximal development (Vygotsky) The distance from where a learner is currently in their development to the level they can perform at with assistance. • The importance of scaffolding Scaffolding provides a framework for the learner to achieve the task.

42. Helping learners (2) • Assisting with identifying sources of information • Promoting the seeking of feedback • Facilitating the incorporation and use of feedback • Recognizing the emotional content of feedback • Addressing the tensions ( within people, between people and in the environment)

43. Tips: Educational strategies to enhance informed self-assessment Selecting assessment & feedback strategies: • Feedback (formal, informal) is required to inform self-assessment • To work, feedback needs to be timely, specific, credible, authentic, provided by informed & engaged supervisors, guided by standards • Useof objective feedback data as possible; e.g., observed, chart audit, practice review • Assessment processes need to be credible, authentic, meaningful (not simply “jumping through another hoop”)

44. Tips: Educational strategies to enhance informed self-assessment • Faculty development: • Increase awareness that feedback (formal, informal) is required to inform self-assessment • Provide timely, relevant, specific, feedback • Engage learners in their own self-assessments • Practice providing constructive feedback that will help learners see how they can improve • Create a safe learning environment where it’s OK to say “I don’t know” or ask for help

45. Tips: Educational strategies to enhance informed self-assessment • Learner/ physician development: • Help them to identify valid sources of external performance feedback • Help them to develop skills in critically interpreting/ analyzing the feedback data, and judging their performance • Help them be aware of and manage their emotional response to corrective feedback • Teach skills for seeking specific feedback from supervisors that will help them improve • Empower learners to seek feedback • Encourage physicians to seek feedback from colleagues

46. Summary and next steps • What will you take away? • Thank you!

47. Informed self-assessment: Publications • Sargeant J, Armson H, Chesluk B, Dornan T, Holmboe E, Eva K, Mann K, Lockyer J, van der Vleuten C, Loney, E. Processes and dimensions of informed self-assessment: A conceptual model. Acad Med. 2010; 85(7):1212-20. • Lockyer J, Armson H, Chesluk B, Dornan T, Holmboe E; Loney E, Mann K, Sargeant J. Feedback Data Sources that Inform Physician Self Assessment, Med Teach, 2011; 33(2): Pages e113-e120 (DOI: 10.3109/0142159X.2011.542519). • Sargeant J, Eva KW, Armson H, Chesluk B, Dornan T, Holmboe E, Lockyer J, Loney E, Mann K,. Features of assessment learners use for informed self-assessments of clinical performance. Med Ed 2011; 45(6): 636-647. (DOI: 10.1111/j.1365-2923.2010.03888.x) • Mann K, van der Vleuten C, Armson H, Chesluk B, Dornan T, Eva K, Holmboe E, Lockyer J, Loney E, Sargeant J.Tensions in informed self-assessment: How the desire for feedback and reticence to collect/use it create conflict . Acad Med 2011, 86 (9):1120-1127 • Eva KW, Holmboe E, Lockyer J, Loney E, Mann K, Sargeant J. Factors influencing responsiveness to feedback: On the interplay between fear, confidence, and reasoning processes. Adv Health Sci Ed 2012 Mar;17(1):15-26