Evaluation of the efficacy of the Tweenees lessons Ian Barron Teaching Fellow

1. Evaluation of the efficacy of the Tweenees lessons Ian Barron Teaching Fellow Child Care and Protection School of Education, Social Work and Community Education, University of Dundee

2. School-Based Abuse Prevention Programme EvaluationTraditional Literature Review: 1990 - 2007 • 22 Studies (3 secondary) evaluated school-based abuse prevention programmes over a 17 year period. The following outcomes were identified: • Children displayed high levels of ‘prior knowledge’ • On average small gains in knowledge and behavioural intention (telling) were achieved • Cognitively able middle class girls made the most gains • There were small increases in the number of disclosures (8 studies) • At least 4 lessons, boosted annually were necessary to sustain knowledge gains • There was no evidence that school-based abuse prevention programmes cause any harm School of Education, Social Work and Community Education, University of Dundee

3. Meta-analyses (1990 – 2007) Outcomes • 4 identified meta-analyses (most recent Davis and Gidycz, 2000) with effect sizes ranging from d = 0.47 to 1.07 moderate to high effect size) Moderator variables • Duration (4 lessons and booster) and content of the programme (modelling and skill rehearsal) • Age (older pupils achieved greater gains) • SES (pupils from middle class families achieved greater gains than working class) School of Education, Social Work and Community Education, University of Dundee

4. Gaps in the research to date • Participants (survivors, upper primary and secondary aged pupils’ response to programmes) • Lack of systematic recording (demographic, attrition rates, disclosures, effect sizes) • Assessment measures (limited standardised measures, over reliance on questionnaires, children’s views not sought, children’s negative experiences not asked for, no cost-effectiveness measures, virtually no programme integrity measures) School of Education, Social Work and Community Education, University of Dundee

5. Current study • Evaluation of a local school-based abuse prevention programme – the Tweenees (VIP package) • Design: Pre and post-test measures with waiting list control group, matched for numbers • School and year group sampling – purposeful • Sampling for interviews - stratified sampling for age and gender and random sampling for SES • Bias towards urban, working class, Caucasian pupils School of Education, Social Work and Community Education, University of Dundee

6. School-Based Abuse Prevention Evaluation: The Local Study • 4 lessons were delivered from the Tweenees programme over a 4 week period to a small group of survivors, as well as primary and secondary aged pupils. All schools had volunteered to take part of the evaluation. • Survivors’ Group: Intervention group – 18u delivered lessons (10 pupils) and waiting list control group (10 pupils), aged 6 to 13. • Primary Group (Primary 7): 18u delivered lessons (68 pupils), class teacher delivered lessons (20 pupils) & waiting list control group (59 pupils) • Secondary Group: S1 (118) and S2 (114) including waiting list control groups for each year. Equivalent numbers in each condition. Guidance teachers delivered the lessons. School of Education, Social Work and Community Education, University of Dundee

7. School-Based Abuse Prevention Evaluation: Local Study Outcome Measures • Knowledge & behavioural intentions (CSKS-Q adapted, psychometric properties, used in 3 previous studies) • Perception of risk assessment (perception of harm and capacity to keep self safe; exploratory – signs at home, in school and in the community) • Emotional impact (explicit v’s implicit, positive/negative) • Disclosure rates (within and beyond lessons - generalisation) • Customer satisfaction from teachers and pupils • Implementation integrity of class lessons (exploratory) • Tools: Questionnaires, interviews, disclosure forms and video observation School of Education, Social Work and Community Education, University of Dundee

8. OutcomesSurvivors’ Group: Knowledge and Skills Questionnaire Pre and post-test ‘Total Scores’ (114 total score) • Intervention group - significant difference on ‘post-test total scores’ on knowledge/skills questionnaire found (t = -4.035, p = 0.003) compared to the control group (t = 0.079, p = .939), • 6.7 points v’s 0.1 points, SD = 5.25 • Survivors who received the Tweenees lessons demonstrated on average, significant small gains in knowledge and behavioural intention to tell compared to survivors who had not received the lessons. • Effect size d = 1.00 (high) School of Education, Social Work and Community Education, University of Dundee

9. OutcomesSurvivors’ Group: Knowledge and Skills Questionnaire - ‘Specific Questions’ • Significant difference found on question 10 (t = -2.862, p < .05) and 11 (t = -4.000, p < .01) post-test scores compared to control group (t = 1.861, p = .096 & t = .557, p = .591) respectively • Question 10: What if your uncle was giving you a big tight hug and you don’t like it. What would you do? • Question 11: What if someone touches you in a way that makes you feel uncomfortable inside and you ask them to stop but they don’t. What would you do? • Survivors who received the Tweenees lessons were more likely on average to suggest more effective strategies to these questions compared to survivors who didn’t receive the lessons, e.g. ‘saying no, getting away and telling’. School of Education, Social Work and Community Education, University of Dundee

10. OutcomesSurvivors’ Group: Knowledge and Skills Questionnaire Pre and post-test ‘Moderating Factors’ • When comparing the scores of male and female survivors, who had received the Tweenees lessons, a significant difference was found for Gender on question 21 [F (1,8) = 7.273, p < .05] and question 24 [F(1,8) = 6.316, p < .05]. Given the control group was all female there was no control group comparison. • Female survivors who received the Tweenees lessons were more likely on average to report higher overall ‘sense of safety’ and ‘feeling good about themselves’ scores compared to the male survivors who received the lessons. School of Education, Social Work and Community Education, University of Dundee

11. OutcomesPrimary Evaluation (P7): Knowledge and Skills Questionnaire - Pre and post-test ‘Total Scores’ • 18u worker delivered lessons - significant difference found (t = -5.433, p < .01), 3.4 points, SD = 5.13, effect size d = 0.60 • Class Teacher delivered lessons – significant difference found (t = -3.107, p < .01), 4.1 points, SD = 5.90, effect size d = 0.65 • Waiting list control – no significant difference (t = 0.692, p = .492), -0.5 points, SD = 5.45 • Primary 7 pupils who received the Tweenees lessons, whether delivered by 18u workers or their own class teacher, achieved significantly more knowledge and behavioural intention to tell gains on average than pupils who had not received the lessons. School of Education, Social Work and Community Education, University of Dundee

12. OutcomesPrimary Evaluation (P7): Knowledge and Skills Questionnaire - Pre and post-test ‘Specific Questions’ • 18u worker delivered lessons - significant results found for: Question 10: Intervention group (t = -2.065, p < .05) compared to control group (t = 0.435, p = 0.665). Question 14 ‘What if a friend of your parent/carer is hugging and kissing you and then asks you to keep it a secret. What do you do?’ Intervention group (t = -2.031, p < .05) compared to control group (t = -0.195, p = 0.845). • Class Teacher delivered lessons - significant result found: Question 13 ‘What if a 17yr old is asking you to do something you know you shouldn’t do. You say No! I’m going to tell. Then he says if you do I’m going to beat you up. What would you do?’ Intervention group (t = -2.486, p < .05) compared to control group (t = -0.286, p = 0.776). • On average primary 7 pupils who received the Tweenees lessons were more able to suggest more effective solutions (telling and not keeping the secret) to these specific scenarios than pupils who did not receive the lessons. School of Education, Social Work and Community Education, University of Dundee

13. OutcomesPrimary Evaluation (P7): Knowledge and Skills Questionnaire - Pre and post-test ‘Moderating Factors’ • On pre and post-test scores, no apparent significant differences were found between the pupils who received the Tweenees lessons and those who did not on the moderating factors of age, ethnicity, socio-economic status, and gender • Age (within the P7 year group), gender, socio-economic status and ethnicity (too small a sample) did not appear to be factors which effected the children’s response to the Tweenees lessons. School of Education, Social Work and Community Education, University of Dundee

14. OutcomesSecondary Evaluation: Knowledge and Skills Questionnaire - Pre and post-test ‘Total Scores’ • S1: Intervention group - significant difference found (t = - 2.248, p < .05) 1.6 pts, compared to control group (t = 1.547, p = 0.127) 0.1pts • S2: Intervention group - significant difference found (t = - 2.703, p < .01) 2.2pts, compared to control group (t = - 1.594, p = .117) 1.3pts • S1 and S2: Intervention group - significant difference found (t = - 3.523, p < .01) 1.8pts, compared to control group (t = - 0.393, p = .695) 0.2pts. • Effect size d = 0.33 • In conclusion secondary S1 and S2 pupils who received the Tweenees lessons made small average knowledge and behavioural intention gains to tell, compared to same age pupils who did not receive the Tweenees lessons. • The S2 group received an abuse prevention lesson a month prior to the 4 lessons. It is unclear whether this made any impact on the S2 results. School of Education, Social Work and Community Education, University of Dundee

15. Secondary Evaluation: Knowledge and Skills Questionnaire - Pre and post-test ‘Specific Questions’ • No significant results were found for specific questions in either S1 or S2 for pupils who received the Tweenees lessons and those in the waiting list control School of Education, Social Work and Community Education, University of Dundee

16. OutcomesSecondary Evaluation (S1): Knowledge and Skills Questionnaire Pre and post-test ‘Moderating Factors’ • A significant result was found for the moderating factor of ‘Presenter’ [F(2,57) = 3.182, p < .05] from the post-test Total Scores. • Significant results were found for three specific questions for Presenter • Q 3 ‘When you are on your own who is taking care of you?’; • Q15 ‘Do you have to do everything your baby sitter tell you to do?’ • Q17 ‘What if a 17yr old is asking you to do something you shouldn’t do. You say No, I’m going to tell my parent/carer. Then he says if you do I’m going to beat you up. What do you do?’). • Findings suggested that the extent of knowledge and skill gains for S1 pupils may be partly dependant upon the experience of the teacher presenting the lessons. School of Education, Social Work and Community Education, University of Dundee

17. OutcomesSecondary Evaluation (S2): Knowledge and Skills Questionnaire - Pre and post-test ‘Moderating Factors’ • No apparent significant results were found for the moderating factors of age, gender, socio-economic status, ethnicity and presenter for S2 pupils. • Age ‘within the S2 year group’, gender, socio-economic status and ethnicity did not appear to be factors which impacted on pupils response to the Tweenees lessons. School of Education, Social Work and Community Education, University of Dundee

18. OutcomesSecondary Evaluation (S1 & S2): Knowledge and Skills Questionnaire - Pre and post-test ‘Moderating Factors’ • No apparent significant results were found in the whole S1/S2 group for the moderating factors of: gender, socio-economic status, ethnicity, presenter and year group. • The above factors did not appear to significantly effect pupils responses in S1 and S2 (when taken as a whole group) to the Tweenees lessons School of Education, Social Work and Community Education, University of Dundee

19. OutcomesSurvivor Group Disclosures over 4 lessons • Intervention Group: 12 Disclosures* 2 Child Sexual Abuse (previously known) 8 Bullying 2 Physical Assault • Waiting List Control: 0 *All disclosures in the lessons were verified from video analysis of the sessions and from adults/peers within the situations in which the harm occurred. School of Education, Social Work and Community Education, University of Dundee

20. OutcomesPrimary Group Disclosures during the 4 lessons • 18u worker delivered lessons (68 pupils): 65 Disclosures 15 - physical abuse 2 - child sexual abuse 1 – sexual assault 14 – bullying 2 – grooming 1 - erotica 11 - physical assault 1 - attempted stealing 1 - abduction 10 - domestic violence 1 - emotional abuse 5 - attempted abduction 1 – rape • Class Teacher delivered lessons (20 pupils): 4 Disclosures 2 attempted assault 1 domestic violence 1 domestic violence/attempted assault • Waiting List Control (59 pupils): 0 Disclosures • Taking into account the numbers of pupils in each group, there appears to be a significant difference between the number of disclosures received by 18u workers compared to the P7 class teacher. A number of disclosures could refer to the same incident. School of Education, Social Work and Community Education, University of Dundee

21. OutcomesSecondary Group: S1 and S2 - Disclosures during the 4 lessons • S1 Intervention Group: 5 Disclosures* 1 domestic violence 1 bullying 1 physical assault 1 physical & emotional abuse 1 abduction • S2 Intervention Group: 0 • S1: Waiting List Control: 0 • S2 Waiting List Control: 0 *S1 disclosures occurred in the 1st session of one class, the only secondary lesson led by an 18u worker. All disclosures were verified as having been said via video analysis. School of Education, Social Work and Community Education, University of Dundee

22. Outcomes: GeneralisationPrimary and Secondary Evaluations: Telephone calls to survivors helpline (Oct – April) by school area* School area 2004/52005/6 Primary 7: 18u delivered schools 0 44 Primary 7: Class Teacher delivered school 0 0 Primary 7: Waiting List Control schools 0 0 Primary 7: Other area schools 0 28 Secondary S1/2 : Guidance Teacher delivered school 2 5 Secondary S1/2: Waiting List Control school 0 0 Of the 77 calls: 36 to check the number out 4 reported female friend raped (all female) 16 bullying (14 female, 2 male) 2 sexual assault (all female) 7 domestic violence (all female) 1 rape (female) 5 physical abuse from parents 1 fear of stranger “who hurts children” (boy) (3 male, 2 female) 5 calls – no data * Available information analysed from Survivors Service helpline records School of Education, Social Work and Community Education, University of Dundee

23. OutcomesPrimary and Secondary Evaluations: Telephone calls to survivors helpline – some themes • Records for 2004/5 showed that children did not use the helpline. Phone calls were from adults seeking support for children. • Some primary 7 children who received the Tweenees lessons subsequently used the helpline number (applying skills in real life). Some simply checked out the helpline, others disclosed abuse. • A significant number of children in Primary 7 phoned the helpline from other school areas not part of the evaluation. Given the specificity of the age group, this may be evidence of children talking to each other across schools leading some children, not involved in the Tweenees lessons seeking help (generalising knowledge and skills from child to child?). • Five pupils from secondary (S1) phoned the helpline – 4 reported the same incident of rape. School of Education, Social Work and Community Education, University of Dundee

24. OutcomesSurvivor, Primary and Secondary Interviews (children’s learning and experiences) 20 interviews (6 survivors, 6 primary pupils and 8 secondary pupils – matched for gender) • Nearly all children reported new learning’s, e.g. what to do in a risky situation, discovering some people having a tough time and hearing different perspectives • Little to no negative experiences or outcomes were reported despite being asked this directly • Some felt they could talk about their feelings more openly because they heard others doing so. • After experiencing the lessons most reported feeling more confident in keeping themselves safe • Some judged themselves to be more assertive in ‘telling’ because of their experience of adults listening, believing them and giving them more choices within the lessons. School of Education, Social Work and Community Education, University of Dundee

25. OutcomesSurvivor, Primary and Secondary Interviews (children’s learning and experiences) • Nearly all children affirmed ‘telling’ as the best strategy • Most felt that following the lessons, they were more able to keep themselves safe & less likely to be harmed • On hearing others stories of harm most described their reaction as feeling ‘sad and wanted to do something about it’ • All thought ‘none’ of the ideas within the lessons were difficult to understand School of Education, Social Work and Community Education, University of Dundee

26. Presenters: Teachers (6) and 18u Workers (3) – main themes from the interviews • 18u and teachers held different goals, the former emphasised disclosure, the latter improving children’s behaviour towards others. • 18u workers and teachers held different definitions of abuse. Teachers tended to operate what could be described as a ‘narrow’ definition of abuse, in terms of pupils disclosing to the teacher and requesting that the disclosure be passed on. 18u workers operated a ‘wide’ definition of abuse incorporating any stories of harm that were shared. • Such differences in definition may have led to the significant differences in the recognition and reporting of disclosures. • Teachers thought half the pupils benefited whereas 18u workers thought all the pupils benefited. This was underpinned by a difference in the definition of what ‘benefit’ meant. • Most teachers reported that they were confident to run the lessons next year. • Because teachers reported to experiencing few disclosures during the lessons, none of the teachers felt the lessons had better prepared them to receive disclosures from children in the future. • All thought parents should be more actively involved. School of Education, Social Work and Community Education, University of Dundee

27. Presenters: Teachers and 18u Workers – main themes from the interviews • The only detrimental effects on pupils mentioned was that some boys were showing off during the lessons. • The concept seen as easiest for children to learn was to “say no, go and tell”. • The concept seen as most difficult to learn was that “violence is preventable”. • There were differing views on what limited the effectiveness of the lessons. Teachers reported on children’s troublesome behaviour and 18u workers reported on children getting in trouble at times for their behaviour. • There were shared views on making the Tweenees more effective for the future. These included more lessons, smaller groups and more preparation and training for teachers. • Another suggestion was to use multi-agency teams to resource the delivery of the lessons. This was seen as an opportunity to facilitate closer partnership working between schools and child protection services. School of Education, Social Work and Community Education, University of Dundee

28. Survivor, Primary and Secondary Groups: Customer satisfaction – pupils’ views • All survivors (100%) and most primary aged pupils liked the Tweenees lessons ‘a lot’ (74%). • A third of secondary pupils either ‘liked’ or liked the lessons ‘a lot’, with the rest finding the lessons ‘ok’. A small number of S2 pupils (7) felt bored at times • The favourite parts of the Tweenees lessons were the ‘discussions’ (15%) and the games (13%), especially -Truth Dare Scare , Iffy Jiffy and Traffic lights. • Pupils liked the writing worksheets the least (19%) • Despite being specifically asked, only 2 primary pupils felt frightened and 3 embarrassed during the lessons. No pupils reported experiencing upset. School of Education, Social Work and Community Education, University of Dundee

29. Survivor, Primary and Secondary Groups: Customer satisfaction – pupils’ views • Almost half (40%) the pupils did not talk to their parents about the Tweenees lessons. Those that did, spoke about it being ‘fun’ and ‘learning about safety’. • ‘How to keep safe’ were the most frequently reported ‘useful’ ideas (14%) in the lessons. • Most pupils found the Tweenees lessons ‘understandable to very understandable’ (over 90%) • Most pupils felt they had been given enough information to decide whether to tell if they were being harmed in some way (over 90%). • Most reported to feeling good, comfortable and relaxed during the lessons (80%). A small number felt sad (2%) on hearing stories of harm. School of Education, Social Work and Community Education, University of Dundee

30. Presenters: Teachers and 18u Workers - Customer satisfaction Most presenters reported • Most pupils looked forward to the lessons • ‘Truth Dare Scare’ was the favourite part of the lessons for pupils • The lessons were understandable • No parts of the lessons were upsetting or anxiety provoking for pupils • Pupils did not become unduly afraid of strangers nor overly assertive • Children of all ages should be taught such lessons as part of the school curriculum • They would like to teach these lessons next year, although one teacher responded ‘no’ in the evaluation form and ‘yes’ within the later interview. School of Education, Social Work and Community Education, University of Dundee

31. Presenters: Teachers and 18u Workers - Customer satisfaction There was a difference between 18u workers and teacher responses in that 18u workers also reported that children: • became more assertive through the ‘telling of abuse’ • increased in their frequency in approaches to them to tell their stories of harm • applied the strategies they learned through ‘telling’ within and out-with the lessons • applied their strategies to other contexts, e.g. at home or in a small survivors group. School of Education, Social Work and Community Education, University of Dundee

32. Implementation Integrity:Video Analysis Summary From video analysis 18u worker led delivery of lessons included: • Open body language and orientation and movement around the class towards pupils • High frequency of interactions from the ‘support’ adult with individual pupils within the class • Support adult giving individual pupils an opportunity to tell their story to an affirming adult before giving the pupil the choice to tell the class. • Facilitation of ‘peer to peer’ talk with a high frequency of spontaneous contributions (more akin to the playground setting) • Space and time given for longer turn-taking • Specific and explicit questions about harm • All pupil responses explicitly received without judgement • Validation of all viewpoints as right for that child • Different pupil responses summarised to enable pupils to come to their own conclusions • Disclosures received without judgement or blocking • Frequent recognition and reporting of actual disclosures • Space and time given for spontaneous class discipline to occur among pupils rather than active adult intervention School of Education, Social Work and Community Education, University of Dundee

33. Implementation Integrity:the ‘Prevention Mind’ 18u worker delivery (from interview) was underpinned by: • A high level of motivation and unity of purpose for the safety and protection of children • An explicit goal of disclosures as an outcome • An expectation that disclosures would occur • The motivation and lack of fear to notice and record disclosures • A thorough understanding of the materials and how they work, i.e. the ownership of an active process to facilitate disclosure • An absence of ‘adult myths’ about abuse • A high level of knowledge about child development and child protection, e.g. definition, signs and symptoms and what they mean • A high knowledge and skill level in child-centred communication School of Education, Social Work and Community Education, University of Dundee

34. Measure of Cost-effectiveness In terms of additional costs the cost of delivering the Tweenees lessons in the intervention group was worked out per pupil, per school and per disclosure. This worked out as follows: • Per pupil = £7.73 (There were 205 pupils in the primary and secondary intervention group. This figure was divided into the additional total costs to get the cost per pupil for the intervention group). • Per disclosure = £21.71 (The figure selected for disclosures was the 73 disclosures made within the four lessons in the primary and secondary intervention groups. The additional total cost was divided by this figure to get a cost per disclosure). • Per school = £396.25 (There were four schools in total. This figure was divided into the additional total costs to get the cost per school). School of Education, Social Work and Community Education, University of Dundee

35. Implications and Recommendations • The Tweenees lessons appear to be one of the most effective abuse prevention programmes to date when delivered with skill (no other programme has evidenced such levels of disclosure) • Teachers effectively delivered the Tweenees lessons to a ‘primary’ prevention level (increased knowledge and behavioural intention) • 18u workers effectively delivered the lessons to both a primary and ‘secondary’ prevention level (disclosure of abuse) • Teachers will need focused training in the ‘Prevention Mind’ in order to deliver the Tweenees materials to a secondary prevention level. School of Education, Social Work and Community Education, University of Dundee

36. Implications and Recommendations • A video should be developed demonstrating the effective delivery of the Tweenees lessons, incorporating how to recognise and respond appropriately to disclosure. • Abuse prevention lessons should be delivered annually. • Methods for active parental involvement should be built into delivery of the lessons. • A multi-agency model of lesson delivery should be explored. • Longer running groups are required for survivors of abuse. School of Education, Social Work and Community Education, University of Dundee

37. Conclusions • The Tweenees programme matches other abuse prevention programmes in achieving on average small knowledge and behavioural intention gains. • The Tweenees lessons go substantially beyond these other programmes by further evidencing the transfer of these skills into daily life, i.e. children’s use of ‘telling’ behaviour. • Such developments appear to be achieved by a highly knowledgeable and experienced prevention mindset coupled with a child-centred interactive class process. • Teachers need focused training and support to develop the knowledge, mindset and child-centred communication skills that facilitate such a class context and process. • The most effective and pragmatic way forward is to draw on the collaborative and joint-expertise of teachers and child protection workers (survivor services and statutory professionals) in the delivery of the Tweenees lessons. School of Education, Social Work and Community Education, University of Dundee