Dr. Jose Pereira Alberta Cancer Foundation Professor of Palliative Medicine University of Calgary

1. Rural Palliative Care (PC) Education: Results of a Hybrid Course with Face-to-Face and Online Learning. Dr. Jose Pereira Alberta Cancer Foundation Professor of Palliative Medicine University of Calgary

2. Goals of the Project • Instil residents with competencies required to care for terminally ill patients. • Develop evaluation tools • knowledge, attitudes and skills. • Explore residents’ responses to the inclusion in the curriculum of: • Spirituality, hope, suffering, self-awareness, use of narrative, humanities to engage affective domain of learning

4. Course Design Movies & arts Course intro Technology intro. Communication Pain/Symptoms Interactive, schemes & case-based • Hybrid model: F2F & online OSCEs (x4) &F2F workshops 1½ days Online 10 weeks Communication Decision-making Ethics Pain/Symptoms F2F workshops & OSCEs (x4): 1½ days

5. Course Design Online Case Discussions Thematic Discussions Ask the Expert Assignments (E-mail) 4 modules Each 2-weeks long 10 weeks Small group asynchronous discussions

6. Evaluation Framework • Knowledge • Pre vs Post-course knowledge test • 20-items, multiple choice questionnaire • Based on blueprint; face validity • Varying Bloom’s hierarchies • Attitudes • Surveys • Self-perceived changes in clinical comfort levels • 22 items, 5-point Likert-like scale (1=not at all comfortable, 5=very comfortable) • Inclusion of topics in learning • Focus Groups • Skills • 4 OSCEs • Course participation • Course itself • Surveys & Focus Groups

7. Results: Knowledge • N=15 • Internal Reliability: Cronbach’s : 0.5 • (Need to increase # of items to 30 to increase reliability to 0.67) • Significant improvement in knowledge • Repeated measures test: F=19.8, p=0.001 • Cohen’s effect size: 0.77 • Pre-course mean (SD): 12 (2.6) • Post-course mean (SD): 16 (1.9)

8. Self-perceived clinical comfort levels: Pre vs Post course • N=15 • Significant improvement in comfort levels • Repeated measures test: F=75.3, p<0.001 • Cohen’s effect size: 0.92 • Pre-course mean (SD): 59.7 (10.9) • Post-course mean (SD): 82.8 (4.7)

9. Self-perceived comfort levels • Pre versus Post Course • Little change in communication • Large change in pain & symptom management But • At post course when asked “compared to when you first started…” • Large change in communication as well • Role of OSCEs for self-assessment

10. Focus Groups Results • Ambivalence to including psychosocial care in case studies. • “..talk about one topic at a time; not mix; separate the psychosocial from the clinical” • “Would have liked to see more clinical stuff” [online] • “I don’t agree; the patient is a whole person, you cannot separate”

11. Focus Groups Results • Ambivalence to spirituality in care • “Physicians should address spirituality when treating palliative patients…one cannot separate the physical and the spirit.”..but no-one has taught us how to do this • “For now, we want to learn more about fundamentals of medicine rather than spirituality”

12. How should we introduce spirituality? • Perhaps in disguise

13. Possible Roles of OSCEs Needs Assessment Education tool Formative evaluation Summative evaluation

14. 4 OSCEs in this Course • Developed from real cases • 3 domains in each OSCE: physical issues, psychosocial issues & communication. (clinical decision-making & communication) • Reflect major competencies • 58 y/o university professor with breast cancer. Presents with cancer pain. • Cancer pain management. Address fears of opioids, explore illness experience. • Young 32 y/o with advanced gastric cancer, nausea & vomiting from upper GI obstruction.Young children. • Manage psychological distress, being in presence of suffering, managing nausea & vomiting. • 60 Y/o man with severe shortness of breath from advanced ALS. Accompanied by wife. • Explore fears, advanced planning & discuss code status, home care needs, manage dyspnea. • Office visit by home care nurse • Interdisciplinary collaboration, manage delirium, inability to swallow & hypercalcemia in home setting

15. Steps in developing OSCEs • Identify competencies & blueprint • Develop OSCEs (as a team) • Review OSCEs with content experts & potential learners (sample from target group of learners.) • Prepare score sheets • Checklist & Global Rating Scale. • Train actors & actresses • Prepare logistics for implementation. • Test OSCEs with actors/actresses • Do OSCEs (videotape) • Rehearse scoring with scorers • Preliminary reliability testing • Scoring • Modifying OSCEs.

16. Checklist vs Global Rating Scale? • Opted for checklist & global rating • Literature • Global rating scales scored by experts showed higher inter-station reliability, better construct & concurrent validity than did checklists. • The use of checklists prior to using a global ratings scale did not improve the reliability or validity of the global rating. Regehr G, et al. Acad Med 1998;73:993-997

17. Scale Design 1. Separate score sheet for each OSCE 2. Scale consists of two subscales: • Performance of Skill • Degree to which skill performed • Items rated on a 3-point scale • Criterion-based scoring [Doig et al; Thompson et al] • Omitted, performed but not competently, performed competently

18. Results • Inter-rater Reliability based on 4 raters • Cronbach’s Alpha • Performance of Skill: .77 - .88 • Degree of Skill: .72 - .88 • Overall Scale: .87 to .92 • Further inter-rater and intra-rater reliability and generalizability being assessed.

19. What residents thought of OSCEs • Very useful learning tools. • Helped them identify their learning needs and provide them with practice. • Would recommend it to other residents

20. Overall Course Evaluation • Would recommend it to future residents • Want practical approaches, not theoretical discussions • Want more mentoring • Some ambivalence about: • Online learning component • Thematic discussions • Psychospiritual issues

21. Strengths & limitations • Limitations • Small numbers limit generalisability

22. Conclusions • There is a culture that does not value integrated care- need to address this in the undergraduate curriculum • Evaluation methods require careful thought and expertise