Michelle Lefevre University of Sussex OSWE Symposium 15.5.09

1. Evaluating the teaching and learning of communication skills with children and young people Michelle Lefevre University of Sussex OSWE Symposium 15.5.09

2. Focus of this presentation • This presentation will discuss a 2 year evaluation of the approach at Sussex University to teaching MA qualifying social work students how to communicate effectively with children and young people. • Qualitative and quantitative data collected at four points during the programme will be presented • Implications for curriculum structure, and further data analysis and data collection will be discussed. • The usefulness of vignettes as a research tool will be considered Michelle Lefevre, University of Sussex, M.Lefevre@sussex.ac.uk, presentation 15.5.09

3. Why measure the effectiveness of teaching and assessing communication skills with children? • SCIE Knowledge Review (Luckock, Lefevre & Orr et al, 2006) identified what counts as effective communication in practice (‘Knowing-Being-Doing taxonomy) – so what to teach in a whole programme approach • But - concerning findings … • Communication with children is obscured and marginalized in the taught curriculum • Dearth of evidence on what counts as effective methods of teaching and assessment • Pedagogical methods are posited anecdotally or theoretically, lacking hard evidence on their effectiveness • (See Luckock, Lefevre & Tanner, 2007Lefevre et al, 2008). Michelle Lefevre, University of Sussex, M.Lefevre@sussex.ac.uk, presentation 15.5.09

4. What kinds of pedagogical evidence might be needed? • The norm is post-hoc studies of satisfaction with teaching – but they are subjective and do not measure the skills/knowledge which have been developed • More robust/objective approaches needed/possible? • Collection of data regarding effectiveness of the teaching approaches across the whole curriculum which enhance communication skills with children • Measures of student knowledge and skills pre- and post- teaching to evaluate whether the input (i.e. teaching) produces the desired outcomes (i.e. enhanced knowledge and skills in practice). • Evaluation tools which can be disseminated so building the evidence base for different approaches Michelle Lefevre, University of Sussex, M.Lefevre@sussex.ac.uk, presentation 15.5.09

5. Methodological position – critical realist approach (Pawson and Tilley, 1997) • Important to look at what is learned, and after what interventions, not just what students think they have learned or ‘liked’ • ‘Effectiveness’ is subjective and constructed, being personally and contextually-dependent. But shared understandings can be developed. • The taxonomy developed through the Knowledge Review viewed not as a straightforward ‘checklist’ of requirements but as a source of possibilities which would be drawn upon depending on the given needs of the situation. • Intention to find a more nuanced way of including how students can apply contexualised knowledge in situ Michelle Lefevre, University of Sussex, M.Lefevre@sussex.ac.uk, presentation 15.5.09

6. Evaluating this aspect of the Sussex MA curriculum Evaluation Question: To what extent, and in what ways, does the MA in Social Work Programme at Sussex contribute to social work students’ development of confidence and competence in communicating with children and young people? • Aims to uncover which aspects of the programme have had what effects • Doesn’t assume no skill at the beginning but looks at individual and collective journeys to skilled and confident practice Michelle Lefevre, University of Sussex, M.Lefevre@sussex.ac.uk, presentation 15.5.09

7. Mixed methods approach • Initial Questionnaire: Collection of students’ personal characteristics which would allow for analysis on the basis of their profile, i.e. have particular kinds of student learned most/least? • Ratings: Confidence in communication with children rated by students at each stage and mapped to profile. • Post-hoc subjective student feedback on the aspects of the programme they perceived to facilitate their confidence and skills with children • Case vignette - more ‘objective’ measure of students’ increase in ability to apply knowledge to situations (though analysis is subjective….) • Interviews – looking back, what was useful? How prepared were you? (in planning) Michelle Lefevre, University of Sussex, M.Lefevre@sussex.ac.uk, presentation 15.5.09

8. Vignettes • Questions divorced from context make it difficult for students to draw upon knowledge available to them in a real life scenario. • Hypothetical persons or situations provide a shared ‘reality’ from which students could contextualise their responses and make decisions appropriate to the circumstances - more likely to test students’ capacity to apply knowledge in situ (Hughes,1998). • They set out and hold constant the contextual framework so increasing standardisability and reliability, & provide internal validity (Finch, 1987; Schoenberg & Ravdal, 2000). • Provides a more ‘objective’ measure of students’ increase in knowledge about constituents of effective communication • Analysis is subjective but could be improved by co-rating. Michelle Lefevre, University of Sussex, M.Lefevre@sussex.ac.uk, presentation 15.5.09

9. Collecting data at 4 points • T1 – at the beginning of the programme • T2 – just prior to the students undertaking the sub-module of 3 focused sessions on communicating with children and young people • T3 – just after the students completed the sub-module of 3 focused sessions on communicating with children and young people • T4 – just prior to the students completing the programme • Proposal now for T5 – interviews at end of NQSW year Michelle Lefevre, University of Sussex, M.Lefevre@sussex.ac.uk, presentation 15.5.09

10. Methodological challenges • Student time – justifiable? A learning tool too… • Dilemmas of questionnaire fatigue – does modifying vignette each time affect validity? • Standardising questions would have simplified analysis but previous pilot indicated it suggested answers. • Open questions provide rich data but huge resource implications for analysis – how realistic for time-strapped programmes? • Possibility of mapping the development of knowledge and skills of some individuals over time – although not all have completed all 4 Michelle Lefevre, University of Sussex, M.Lefevre@sussex.ac.uk, presentation 15.5.09

11. SCALE 0 1 2 3 4 5 6 7 8 9 10 Key findings % SCORING T1 4 12 12 4 16 28 20 4 T2 4 19 15 8 42 11.5 T3 17 6 17 33 22 17 T4 14 27 36 23 Key findings:Subjective rating scale regarding confidence at 4 points 4 measures indicate students’ participation in this programme has led to increased confidence in communicating with children and young people Michelle Lefevre, University of Sussex, M.Lefevre@sussex.ac.uk, presentation 15.5.09

12. Exercise/teaching strategy % citing this % citing this in top 5 Mean for importance Key findings Contributions from student colleagues in the group 100% 92% 3.75 Direct teaching input 85% 54% 2.9 Discussion on relationships 85% 54% 2.7 Case study exercise 77% 69% 2.2 Tutor style (e.g. Modelling, managing group process and discussions) 77% 54% 3.3 Experiential exercise reflecting on what was learned in professional experience 62% 38% 2.6 Experiential exercise role playing being a child 54% 38% 3.2 Role-play exercise regarding a child client 54% 46% 2.8 Research findings presentation 46% 8% 2 Subjective measure: the focused teaching on skills with children was felt to be useful Michelle Lefevre, University of Sussex, M.Lefevre@sussex.ac.uk, presentation 15.5.09

13. Most important elements which enhanced the students’ learning % citing this n= 22 % citing this in top 5 N=17 Mean score for importance Key findings Direct practice with children and young people in placement 91% 94% 1.7 Course teaching on child development 82% 53% 3.1 Sub-module on communication skills with children and young people 68% 59% 3.9 Pre-course professional experience with children 68% 59% 2.5 Other personal experience with children 64% 53% 3.2 Input from practice assessor/on-site supervisor 55% 47% 3.1 Perceived most important elements which enhanced the students’ learning Michelle Lefevre, University of Sussex, M.Lefevre@sussex.ac.uk, presentation 15.5.09

14. Posi-tion Those aspects about which students demonstrated most knowledge at T4 % OF STUDENTS DEMONSTRATING KNOWLEDGE Key findings 1* Knowledge about child development 95% 1* Understanding of the purpose of communication in context 95% 1* Knowledge of appropriate models, approaches, methods and skills in work with children 95% 1* Child-centred communication 95% 1* Use of play, symbolic, creative, non-verbal and expressive techniques 95% 6* Interviewing techniques 91% 6* Ability to recognise and use one’s own feelings (and counter-transference) in the work 91% 6* Being able to work with depth processes in the work not just surface ones 91% 9* Knowledge of how (adverse) experiences affect children 86% 9* Actions to promote children feeling trust and safety 86% 11 Eliciting and taking into account children’s views & concerns 81% Vignette: key qualities, knowledge & skills about which students demonstrated MOST knowledge at T4 Michelle Lefevre, University of Sussex, M.Lefevre@sussex.ac.uk, presentation 15.5.09

15. Interviewing techniques 71% Key findings Being able to work with depth processes in the work not just surface ones 63% Feeling comfortable to work with children’s strong feelings 48% How inherited traits, capabilities or impairments affect child’s communication 41% Being able to use a variety of tools (e.g. ecomaps) 39% Providing information and explanations 25% Child-centred communication 19% Being playful 18% Vignette: Aspects of Knowledge which increased the most during the programme go across all 3 domains (Knowing-Being-Doing) Michelle Lefevre, University of Sussex, M.Lefevre@sussex.ac.uk, presentation 15.5.09

16. AOP – non-judgemental attitude 45% Key findings Respectful, 27% Considering issues of confidentiality 26% Open and honest 17% Listening 16% Understanding of how the social work role and task impacts upon communication 15% Eliciting and taking into account children’s views & concerns 15% Understanding of the purpose of communication in context 5% Going at the child’s pace 5% Actions to promote children feeling trust and safety 2% Knowledge of appropriate models, approaches, methods and skills in work with children 1% Why was less knowledge shown at T4 about some aspects? Michelle Lefevre, University of Sussex, M.Lefevre@sussex.ac.uk, presentation 15.5.09

17. Some implications for the curriculum and further research • Focused skills teaching important but practice learning opportunities with children essential • Pre-course, paid work experience and personal contact with children to be encouraged (prior to programme?) • Not all students will respond well to every learning method so variety important • Importance of more evidence on effectiveness • Enhancing validity of vignette tool through co-rating • Much more data to analyse • Follow-up interviews to look at what happens once in practice • Engaging others in evaluating their own curriculum Michelle Lefevre, University of Sussex, M.Lefevre@sussex.ac.uk, presentation 15.5.09

18. References • Lefevre, M., Tanner, K. & Luckock, B. (2008) Developing social work students’ communication skills with children and young people: a model for the qualifying level curriculum, Child and Family Social Work 2008, 13, pp 166–176 • Luckock, B., Lefevre, M., Orr, D., Tanner, K., Jones, M. & Marchant, R. (2006) Knowledge Review:Teaching Learning and Assessing Communication Skills with Children in Social Work Education. Social Care Institute for Excellence, London. • Luckock, B., Lefevre, M. & Tanner, K. (2007) Teaching and learning communication with children and young people: developing the qualifying social work curriculum in a changing policy context. Child and Family Social Work, 12, 192–201. Michelle Lefevre, University of Sussex, M.Lefevre@sussex.ac.uk, presentation 15.5.09