School-wide Evaluation Tool (SET)

1. School-wide Evaluation Tool (SET) MN SW-PBIS Training Kevin Filter 1

2. Virtual Training Materials Please have the following materials printed out and available during the training session. Set Evaluation Tool - http://rainbowresearch.org/clients/pbis/docs/manual10_evaltool.pdf Case Study 1 – http://rainbowresearch.org/clients/pbis/docs/manual05_casestudy1.pdf

3. SET Overview Purpose • Assess the extent to which the critical features of SW-PBIS are in place in a school • Results can serve as a basis for determining annual goals in school 3

4. SET Overview Purpose (continued) 2. Determine whether technical support has resulted in changes in school adoption of SW-PBIS 3. Serves as a partial basis for determining whether adoption of SW-PBIS leads to valued changes in school • Context for SWIS data 4

5. SET Overview Scoring • Each item is scored as a 0, 1, or 2 based on information gathered as described below • Seven sections with two to eight items each 5

6. SET Overview Procedures and Data Sources • Interview administrator (20 minutes) • Interview teachers and team members (2-3 minutes each) • Interview students (1-2 minutes each) • Observe posted products in school • Review permanent products 6

7. SET Scoring Guide Sections 7

8. Scoring Sections • Expectations Defined • Behavioral Expectations Taught • On-going System for Rewarding Behavioral Expectations • System for Responding to Behavioral Violations 8

10. Scoring Sections (cont’d) • Monitoring and Decision-Making • Management G. District-level Support 9

12. CollectingData 10

13. Administrator Interview Guide (20-30 minutes) (Page 4 of SET) 3 sections • Make this a fluent conversation and record responses as they come up • Try to ask open-ended questions and narrow questions if necessary • This may also be a chance for the administrator to brag… that’s a good thing! • Don’t ask for details if some basic features are not in place (“skip to” rules) • Tip: ask who’s on PBS team 11

15. Administrator Interview Guide Let’s talk about your discipline system 1) Do you collect and summarize office discipline referral information? Yes No If no, skip to #4. 2) What system do you use for collecting and summarizing office discipline referrals? (E2) a) What data do you collect b) Who collects and enters the data? 3) What do you do with the office discipline referral information? (E3) a) Who looks at the data? b) How often do you share it with other staff? 4) What type of problems do you expect teachers to refer to the office rather than handling in the classroom/ specific setting? (D2) 5) What is the procedure for handling extreme emergencies in the building (i.e. stranger with a gun)? (D4)

16. Administrator Interview Guide Let’s talk about your school rules or motto 6) Do you have school rules or a motto? Yes No If no, skip to # 10. 7) How many are there? 8) What are the rules/motto? (B4, B5) 9) What are they called? (B4, B5) 10) Do you acknowledge students for doing well socially? Yes No If no, skip to # 12. 11) What are the social acknowledgements/ activities/ routines called (student of month, positive referral, letter home, stickers, high 5's)? (C2, C3)

17. Administrator Interview Guide Do you have a team that addresses school-wide discipline? If no, skip to # 19 12) Has the team taught/reviewed the school-wide program with staff this year? (B3) Yes No 13) Is your school-wide team representative of your school staff? (F3) Yes No 14) Are you on the team? (F5) Yes No 15) How often does the team meet? (F6 16) Do you attend team meetings consistently? (F5) Yes No 17) Who is your team leader/facilitator? (F4) 18) Does the team provide updates to faculty on activities & data summaries? (E3, F7) Yes No If yes, how often?

18. Administrator Interview Guide Do you have a team that addresses school-wide discipline? If no, skip to # 19 19) Do you have an out-of-school liaison in the state or district to support you on positive behavior support systems development? (G2) Yes No If yes, who? 20) What are your top 3 school improvement goals? (F1) 21) Does the school budget contain an allocated amount of money for building and maintaining school-wide behavioral support? (G1) Yes No

19. Transition to “wandering” stage • Gather permanent products • Find a place to finish the SET (a small private room) • Transfer some information onto the Interview and Observation Form (page 6)

20. Interview and Observation Form (page 5 of SET) • 15 students and 10 staff (3 team members) • Catch teachers and students in hall or other non-disruptive area • Select times when it will be easy to talk to students and teachers (i.e., lunch) • Introduce yourself and thank everyone! • Look for posted materials during or after interviews 12

22. Interview and Observation Form (cont’d) For all of the following, compare responses to what administrator said • What are school rules? • What type of problem would you refer to office? • Categories vs. examples • Procedure for a stranger with a gun • Who is team leader? 13

23. Interview and Observation Form (Observation Items) Rules & expectations posted What do they look like? What does posted mean? • Clearly visible Documented crisis plan posted What do they look like? • Contents What does posted mean? • Clearly visible 15

24. 16

25. Permanent Products • Ask administrator for these materials during interview • List of materials on page 0 of SET • Hold on to them and review them when scoring the SET (after all interviews) • Best to review them at school in a staff lounge or other semi-private space 18

27. Case Study #1

28. Case Study Exercise(30 minutes) • Use Case Example #1 to independently review a case study • After reading the case, complete a scoring guide (the blank scoring guide is in the complete SET download) • Discuss the case study exercise as a whole group when we come back together 20

29. Materials for the Case Study Exercise Set Evaluation Tool - http://rainbowresearch.org/clients/pbis/docs/manual10_evaltool.pdf Case Study 1 – http://rainbowresearch.org/clients/pbis/docs/manual05_casestudy1.pdf

31. Case Study Exercise Review

34. Next Steps in Training Reliability check • Conduct SET with me or another reliable evaluator • Once reliable (85%), you are eligible to conduct SETs 24

35. What’s Next Establish reliability Conduct SETs Contact Pat Seppanen for co-observation opportunities in your area: pseppanen@rainbowresearch.org

36. Independently Completing a SET Evaluation

37. Logistics of Completing a SET SET Implementation Guide (Page 1 of SET) • Make initial contact • Give school the SET overview (page 1) • Confirm date to conduct SET • Administrator on site • Answer questions related to SET 21

39. Site Visit Logistics and Tips • Send a friendly reminder a few days before visit • Encourage administrator to announce the SET visit and logistics to staff on day of visit (avoid surprises!) • When scheduling, make sure that you can begin with 20-30 minute administrator interview • Bring a calculator • Please avoid evaluating your own school 22

40. After the SET • Log on to www.pbssurveys.org and complete the SET on-line • Copy and paste graph into report template • Complete the report and give it to (a) the school and (b) Rainbow Research within one week

42. Report Template

43. Report Template

44. Report Template

45. What’s Next Establish reliability Conduct SETs Contact Pat Seppanen for co-observation opportunities in your area: pseppanen@rainbowresearch.org

46. Contact Pat Seppanen, Rainbow Research pseppanen@rainbowresearch.org 612-824-0724, ext 204 Kevin Filter MN State Univ, Mankato kevin.filter@mnsu.edu (507) 389-5828 23