Teacher Development and Support System

1. Teacher Development and Support System Feedback from SIG schools July, 2011

2. Context for feedback • As part of developing the teacher evaluation and support system for SIG schools, the district has been soliciting feedback from participants on various aspects of the system • Approximately 650 teachers and staff participated in this work • 75% completion rate as of June 24 • The feedback is still in the process of being gathered and analyzed; those areas with the most feedback to date have been analyzed • This feedback will inform the further development of the measures and the piloting of the evaluation system in school year 2011-12

3. The Multiple Measuresof Teacher Effectiveness Observation of Practice Observing teaching & review artifacts of practice (e.g. lesson and unit plans, student work) By Administrators & By Teachers Development & Support Self-Reviews Individual Growth Plans Multiple Measures Multiple Measures Stakeholder Feedback Parent Surveys Student Surveys Contributions to School Community TBD Contributions to Student Learning Outcomes Academic Growth over Time Analysis of Supplemental Close-Ended and Open-Ended Assessments Differentiated Compensation & Recognition Teaching & Learning Framework A common foundation for effective teaching

4. Key questions for teachers’ feedback How is effective teaching best measured? Observation of Practice Observing teaching & review artifacts of practice (e.g. lesson and unit plans, student work) By Administrators & By Teachers How is effective teaching best supported? Development & Support Self-Reviews Individual Growth Plans Multiple Measures Multiple Measures Stakeholder Feedback Parent Surveys Student Surveys Contributions to School Community TBD Contributions to Student Learning Outcomes Academic Growth over Time Analysis of Supplemental Close-Ended and Open-Ended Assessments How are decisions related to promotion, compensation, and intervention aligned to support teacher effectiveness? Differentiated Compensation & Recognition Teaching & Learning Framework A common foundation for effective teaching How are changes to teacher support and development best implemented?

5. Activities that informedeach component of the system Observation of Practice Teaching & Learning Framework (TLF): Workshop I & II Teaching & Learning Framework: Book Study Group Total Effectiveness Results: Small Group Total Effectiveness Results: Individual Feedback Listening Sessions Development & Support TLF: Workshop II Listening Sessions Multiple Measures Multiple Measures Stakeholder Feedback Contributions to School Community Contributions to Student Learning Outcomes Differentiated Compensation & Recognition Stakeholder Input: Survey Feedback Total Effectiveness Results: Small Group & Individual Listening Sessions Total Effectiveness Results: Small Group & Individual AGT: School Level Results TER: Small Group & Individual Listening Sessions Diff Comp: Small Group Teaching & Learning Framework A common foundation for effective teaching Implementation Listening Sessions

6. Feedback on Observation of Practice

7. What are the benefits of the Teaching and Learning Framework? Listening Sessions: • Virtually all teachers seemed to like the TLF and praised it for providing “common language” for all. Comments included: • “really good guide on how to be a good teacher” • “not McStandards” • “good foundation” • “job description” that includes “all aspects” of being a teacher. • In two separate schools, teachers expressed their pleasure that the new framework seemed to be a move away from evaluation based on “red flags” such as “standards on the board.” Book Study: • Predominantly, participants felt that the TLF will benefit teachers as a roadmap and guide for reflection and self review. • Participants also felt that the TLF will promote collaboration between colleagues by providing a common language and clear expectations.

8. What are the concerns with the Teaching and Learning Framework? Listening Sessions: • The most prevalent concern was principals’ capacity to use the framework to evaluate teachers due to the time, knowledge and skill required by such a complex and rigorous tool. They acknowledged the demands that the TLF placed on evaluators • Several other concerns were mentioned, though less frequently: • Concerns about favoritism (real or perceived) • Potential for distorted reports or poor interpretation of the framework’s intention • Time required to complete might lead to falsifying the document (as has been known to happen with the STULL evaluation). Book Study: • Given the complexity and size of the TLF, participants were concerned about the implementation and whether the district will provide enough time and support to teachers and administrators.

9. Perceptions about the Teaching and Learning Framework Workshops Workshop 1: Over 90% of participants agreed or strongly agreed that • They learned useful information about the TLF • Colleagues were interested in the workshop content • They were interested in the workshop content Workshop 2: • 87% of participants agreed or strongly agreed that they learned useful information about the TLF self review process and lesson design template. • 88% of participants agreed or strongly agreed that they were interested in the workshop content. • 90% of participants agreed or strongly agreed that their colleagues were interested in the workshop content

10. Feedback on the TLF lesson design process and template Workshop 2: • Benefits of the lesson design template (link): • considers all components of teaching, very detailed • can be used as a guide to organize lesson • pushes evaluator to look at various aspects • lets teacher know what's being observed • Concerns about the lesson design template (link): • the design template is overwhelming and will take too much time to complete • the observers, especially the administrators, will not understand the design template and adhere to the observation protocol • teachers will have difficult meeting the expectations of the design template

11. How would you weight Observation of Practice towards a Total Effectiveness Result? Total Effectiveness Result: Teacher Results: • On average, participants felt that Observation of Practice should count towards 55% of the Total Effectiveness Result, with 35% coming from Administrator Observation and 20% coming from Teacher Leader Observation (link). • 80% and 62% of responses wanted Administrator Observation and Teacher Leader Observation, respectively, to count towards teacher evaluations (link).

12. Benefits of and concerns with using Observation of Practice as part of teacher evaluations Total Effectiveness Result: Teacher Results:

13. Feedback on STAKEHOLDER SURVEYS

14. Feedback on Stakeholder Surveys Stakeholder Input Small Group Activity: Participants felt the pilot student survey was clear and organized, had a good coverage of topics, and would be useful for informing teaching practice Many also felt the survey was too lengthy and could be subject to student bias Listening Session: Few participants mentioned the role of surveys. Of those who did, most thought they would be useful and had the potential to engage others (students and parents) in the work of school improvement. One teacher thought that disgruntled students would use them as a means to get back at strict teachers.

15. How would you weight Stakeholder Input towards a Total Effectiveness Result? Total Effectiveness Result: Teacher Results: • On average, participants felt that Stakeholder Input: Student Surveys should count towards 5% of the Total Effectiveness Result (link), with 51% wanting to use it as feedback only, and 14% not count all (link). • The benefit of Stakeholder Surveys: provides opportunity for learners to weigh-in (link). • The concerns: subjective, could conflict with discipline and grading efforts (link).

16. Feedback on CONTRIBUTIONS TO SCHOOL COMMUNITY

17. How would you weight Contributions to School Community towards a Total Effectiveness Result? Contributions to School Community Activity Unfortunately, due to time constraints, we were not able to conduct the activity for this component. Total Effectiveness Result Small Group Activity: On average, participants felt that Contributions to School Community would count towards 8% of the Total Effectiveness Result, with 33% wanting to use it as feedback only, and 21% not count all. The benefit of using this measure: supports collaboration (link) The concern: depends on definition of measure (link)

18. Feedback on CONTRIBUTIONS TO STUDENT OUTCOMES

19. What are the benefits of using Academic Growth over Time (AGT) as a measure of teacher effectiveness? AGT Small Group Activity: Gives teachers motivation and direction to improve Can be used as benchmark and comparison to other schools Focus on improvement and not just achievement Total Effectiveness Result Small Group Activity: Provides objective measure of student learning Supports educators to seek best practice to improve growth Supports collaboration within school Listening Sessions: Some teachers appeared to look forward to the opportunity to demonstrate their effectiveness via test scores

20. What concerns do you have about AGT? AGT Small Group Activity: External factors/variables not factored into score Incomplete/limited results by subject, grade level, subgroups Accuracy, Validity, Reliability of AGT Total Effectiveness Result Small Group Activity: Could focus efforts too narrowly on test prep Test scores are due to many variables Can be outside of individual teacher’s control Listening Sessions: “…participants named as a fear the possibility that test scores would come to dominate the evaluation process…”

21. How would you weight AGT towards aTotal Effectiveness Result? 66% and 53% of participants supported the inclusion of individual and school AGT, respectively, in the TER score (link). On average, student outcomes (individual and school AGT) weighted 28% of Total Effectiveness, second to observations (link). Variance shown in how much AGT should count towards TER score (link). Source: SIG Total Effectiveness Results: Teacher Results small group activity 2011; N= 133 small groups; blank answers excluded and answers adjusted to total 100%.

22. Feedback on DEVELOPMENT AND SUPPORT

23. How should we support effective teaching? Support teachers through FEEDBACK • While overall teachers liked the framework as a document, how the document is used was equally important: • Use “as a tool, not a weapon” • Use as an “appraisal,” as a step in developing teachers not, “punitively.” • They wanted to know that the process would be “sufficient” and “fair,” providing multiple rounds of constructive feedback and allowing for opportunities to improve. Support teachers through CULTURE OF COLLABORATION • The dominant theme among all groups was the importance of collaboration. Collaboration was indicated as a way to benefit both individual teachers and their school overall. • A few teachers noted that collaboration is not necessarily intrinsic and that it must be developed. Several more expressed concerns that the EE work has the potential to be divisive, if it becomes competitive rather than collaborative. Source: SIG Listening Small Group Sessions 2011; 17 sessions; 76 participants.

24. How should we support effective teaching? Support teachers through PROFESSIONAL DEVELOPMENT Not surprisingly, teachers repeatedly noted the importance of high quality professional development. Several specified that it should be specifically linked to an element of the framework, including opportunities to learn further about the framework itself and to strategize with peers about putting it into action Several teachers noted the importance of “real” time – not just the minutes left over during their weekly meeting after the school logistics and announcements are completed. Support teachers through FLEXIBILITY In numerous sessions, teachers spoke about the flexibility inherent in the TLF. They appreciated that the framework appears to focus on the quality of instruction rather than adherence to a particular way of teaching. Some fear that the TLF will be misinterpreted as “one and only one way to implement.” Source: SIG Listening Small Group Sessions 2011; 17 sessions; 76 participants.

25. How should we support effective teaching? Support teachers through SELF-ACCOUNTABILITY In nearly every listening session, teachers talked about their hopes for their own growth as teachers, many very eloquently. Many appreciated the role that the TLF could play in encouraging “self-reflection,” “accountability for self” and “personal growth.” Interestingly, most of these comments were made prior to participating in the second TLF workshop that included the self-review process. Source: SIG Listening Small Group Sessions 2011; 17 sessions; 76 participants.

26. Feedback on DIFFERENTIATED COMPENSATION AND RECOGNITION

27. What kinds of incentives should be used and on what basis? Monetary incentives were, by far, the most frequently proposed form of compensation, followed by career pathways (link). Academic Growth was the most frequently mentioned standard of measurement for receiving incentives (link). Most respondents preferred at least 1 incentive (link).

28. How should the differentiated compensation system be structured? Majority of respondents preferred fixed incentives, such as bonuses (link). Respondents preferred individual and school-wide incentives over group (team/department) incentives (link). Variance shown in the weighting of individual, versus group versus school-wide incentives (link). Respondents tended to weight individual incentives more than group and school incentives (link and reasons for weighting: individual, group, school).

29. How should the differentiated compensation system be structured? Majority of respondents say teachers who are more effective should be compensated more. However, there was concern about divisiveness and less collaboration (link). Respondents also felt that multiple indicators, and not just CSTs and graduation rates, should be used as a measure of achievement (link).

30. Feedback on IMPLEMENTATION

31. What additional factors should we consider in implementation? Student Achievement In every single listening session, teachers named improved student learning and achievement as a hoped for outcome from the Educator Effectiveness work. Communal Responsibility Teachers spoke about the role that other factors play into successful teaching and learning. One asserted that “academic success of a student is not solely the responsibility of one teacher [but rather] a collection of things.” Several teachers believed that peer observations and collaborative work focused on the framework could be used to hold each other accountable and to encourage everyone to “do a little extra” thereby raising expectations for all. Some teachers stressed the role of collaboration among school staff in the form of “collective accountability for students with each other.” Others talked about seeing parents and students as “pieces of the puzzle” and discussed how to engage parents as partners in their children’s success. Source: SIG Listening Small Group Sessions 2011; 17 sessions; 76 participants.

32. What additional factors should we consider in implementation? Student Accountability Several groups of teachers spoke about the need for student accountability as well as teacher accountability. “Teacher Bashing” In most sessions, teachers expressed concerns about the trend toward “teacher bashing” and fears that the “entire burden” for student and school success would be placed on them. One teacher (who incidentally likes the TLF and expressed strong support for its use) noted that there is an “underlying attitude that teachers are not working hard enough.” Turnover In about half of the sessions, teachers talked about the disruptive and demoralizing effects of turnover. While concerns about turnover appeared repeatedly, some thought this work would increase it, while others thought it would lead to a more stable teaching force. Source: SIG Listening Small Group Sessions 2011; 17 sessions; 76 participants.

33. What additional factors should we consider in implementation? Alignment Aligning TLF with Beginning Teacher Support and Assessment (BTSA) standards, or finding a way to use TLF in its place, would simplify the work for beginning teachers and those who work with them. Organization and Communication The nature of the pilot phase (i.e. compressed timeline to complete 50-hours of professional development, school calendars that were established prior to pilot phase, and inconsistent information pathways) led to teachers’ concerns about organization and communication. This created an “air of anxiety” in many schools. Teachers stressed repeatedly that “transparency,” “good communication” and “consistency” would significantly improve the implementation of the EE work. “Stick With It” In all schools, there were teachers who expressed the need to persist with the Educator Effectiveness work beyond the 50-hour commitment. Many noted that that it requires substantial learning and will take time to succeed. They expressed hopes that their colleagues will “step up.” Source: SIG Listening Small Group Sessions 2011; 17 sessions; 76 participants.

34. Appendix: analyses of feedback by activity

35. Year 1 SIG Activities:Teacher Effectiveness Click the hyperlinks below to jump to any of the activities: Teaching & Learning Framework: Workshop I—Introduction to the Framework Teaching & Learning Framework: Workshop II—Application of the Framework Teaching & Learning Framework: Book Study Group Activity Academic Growth over Time: School Level Results Overview Total Effectiveness Results: Small Group—Teacher Results Total Effectiveness Results: Individual Feedback Form Differentiated Compensation: Small Group Stakeholder Input: Survey Feedback—Small Group Activity or Focus Group Listening Sessions

36. Teaching & Learning Framework: Workshop 1 Click for last viewed slide The first Teaching & Learning Workshop introduced participants to the Teaching and Learning Framework and rubric. Participants filled out feedback forms at the end of their session, 495 responses in total. Click for Activities List

37. Feedback on Workshop 1 Structure Click for last viewed slide • Overall, over 90% of the participants Agreed or Strongly Agreed that • The sessions were well organized • Activities were well-paced and appropriately varied • Presenters had strong knowledge • They learned useful information about the TLF • Colleagues were interested in the content • They were interested in the content • Overall, 75-77% of participants  Agreed or Strongly Agreed that • They have a clear understanding of how the session relates to their SIG work back at school • They understand the next steps related to the SIG work Click for Activities List Source: SIG TLF Workshop 1 feedback forms 2011; N= 239; Blank answers excluded.

38. Feedback on Workshop 1 Content Click for last viewed slide Overall, Wisdom of Practice and Memorable Moment were the most engaging for the teachers (79-80% selecting highly engaging or engaging) Give One-Get One, Evidence for Standards 4 & 5, and Reflective Writing were the least engaging (60%, 54%, and 55% selecting highly engaging or engaging, respectively) As a whole, participants had a clearer understanding of “Establishing a Culture for Learning” (Q18), the difference between Developing and Effective (Q19), and Cognitive Engagement (Q21). Less than 50% of participants selected the correct answer for “Ineffective Practice” (Q20) and 21st Century Skills (Q22). Click for Activities List Source: SIG TLF Workshop 1 feedback forms 2011; N= 239; Blank answers excluded.

39. Feedback from Participants Click for last viewed slide About 20% of the surveys had comments Participants were most confused about applying the framework to their teaching, understanding the purpose of the framework, implementing the framework at the classroom, schools, and district level given the complexity, and the next steps in the SIG process Participants thought it would be most helpful to review and study the content on their own or with peers, see real-life examples of the framework being used in the classroom, and receiving more PD like this one. Click for Activities List Source: SIG TLF Workshop 1 feedback forms 2011; N= 239; Blank answers excluded.

40. Workshop 1 Participant Make-Up Click for last viewed slide Over half were middle school teachers, 15% were elementary teachers, and 19% were high school teachers. 2% were principals Compared to the High School teachers, Elementary teachers had a more favorable opinion about the workshop structure (on average +8% response rate for agree and strongly agree) and content (on average +13% response rate for engaging and highly engaging). High school teachers and middle school teachers had similar ratings of the workshop Compared to the Elementary school teachers, principals had a more favorable opinion about the structure but a less favorable opinion about the content. Click for Activities List Source: SIG TLF Workshop 1 feedback forms 2011; N= 239; Blank answers excluded.

41. Teaching & Learning Framework: Workshop 2 Click for last viewed slide The second Teaching & Learning Workshop showed participants how to apply the Teaching and Learning Framework and rubric to self review and lesson study. 239 feedback forms inputted to date with approximately 400 remaining Click for Activities List

42. Feedback on Workshop 2 Structure Click for last viewed slide • Overall, over 90% of the participants Agreed or Strongly Agreed that • The sessions were well organized • Activities were well-paced and appropriately varied • Presenters had strong knowledge • Colleagues were interested in the content • Overall, 82-88% of participants  Agreed or Strongly Agreed that • They learned useful information about the Self-Review process • They have a clear understanding of how the session relates to their SIG work back at school • They learned useful information about the Lesson Design Template • They were interested in the content that was discussed in the workshop Click for Activities List Source: SIG TLF Workshop 2 feedback forms 2011; N= 239; Blank answers excluded.

43. Feedback on Workshop 2 Content Click for last viewed slide Reflecting on My Instruction and Completing the Self Review were most engaging (77% selecting highly engaging or engaging) Reviewing the TLF, Examining a Sample Lesson Plan Using the Lesson Design Template, and Applying the Lesson Design Template to My Own Content were the least engaging (71%, 71%, and 68% selecting highly engaging or engaging, respectively) Click for Activities List Source: SIG TLF Workshop 2 feedback forms 2011; N= 239; Blank answers excluded.

44. Feedback on using the Lesson Design Template when preparing for a formal classroom observation by administrator or another teacher Click for last viewed slide Click for Activities List Source: SIG TLF Workshop 2 feedback forms 2011; N= 239; Blank answers excluded.

45. Feedback on using the Lesson Design Template when preparing for a formal classroom observation by administrator or another teacher Click for last viewed slide Click for Activities List Source: SIG TLF Workshop 2 feedback forms 2011; N= 239; Blank answers excluded.

46. Feedback on using the Lesson Design Template when preparing for a formal classroom observation by administrator or another teacher Click for last viewed slide Click for Activities List Source: SIG TLF Workshop 2 feedback forms 2011; N= 239; Blank answers excluded.

47. Teaching & Learning Framework: Book Study Click for last viewed slide Participants formed book study groups to read and discuss Charlotte Danielson’s Enhancing Professional Practice, culminating in written reflections about the Teaching and Learning Framework and the SIG educator effectiveness work in general. 631 people participated of which 100 reflections were sampled Click for Activities List

48. Teaching & Learning Framework: Book Study Click for last viewed slide Predominantly, participants felt that the Teaching and Learning Framework (TLF) will benefit teachers as a roadmap and guide for reflection and self review. Participants also felt that the TLF will promote collaboration between colleagues by providing a common language and clear expectations. Given the complexity and size of the TLF, participants were concerned about the implementation and whether the district will provide enough time and support to teachers and administrators. Click for Activities List

49. Teaching & Learning Framework: Book Study Click for last viewed slide Click for Activities List

50. General comments about the TLF, evaluations, and educator effectiveness Click for last viewed slide Click for Activities List