Mathematics Performance Tasks Applying a Program Logic Model to a Professional Development Series

1. Mathematics Performance TasksApplying a Program Logic Model to a Professional Development Series California Educational Research Association December 6, 2013 Sally J. Bennett-Schmidt, Ed.D. Assessment Director San Diego County Office of Education

2. Overview of the Session • Context of the Professional Learning Experience • Summary of the Professional Development Series • Application of the Program Logic Model • Vision & Current Situation • Resources & Activities • Outputs • Outcomes: Short-, Intermediate-, Long-Term • Learnings and Next Steps

3. Context of the Project • San Diego County Office of Education • Support 42 school districts + county-run programs • Curriculum, instruction, and assessment staff working collaboratively to support districts in implementation of the common core state standards • San Diego County School Districts • Along a continuum in implementation of CCSS • Substantial needs in mathematics, both content knowledge and instructional approaches

4. The Professional Development Series:Mathematics Performance Tasks & Formative Assessment • Purpose: Support district efforts in the use of performance tasks to maximize student achievement • Key Activities • Examine assessment in the light of the new Common Core State Standards and work being explored by the Smarter Balanced Assessment Consortium • Experience performance tasks along with scoring protocols • Explore the formative use of performance task results including student re-engagement • Incorporate formative assessment practices in districts’ ongoing work

5. The Professional Development Series • Days 1-2: August 2012 • Audience: District teams of 5-7 members (district and site administrators, mathematics teacher leaders) • Facilitated by David Foster from the Silicon Valley Mathematics Initiative • Focus: • Understand the new assessment context under the common core standards • Experience several performance tasks with scoring and analysis protocols • Begin discussion of the formative use of performance tasks and student re-engagement

6. The Professional Development Series • Day 3: September – October 2012 • Audience: Expanded district teams with additional classroom teachers • Facilitated by SDCOE Mathematics & Assessment Staff • Focus: • Select an extended response mathematics task to use with students • Anticipate student successes and challenges • Plan for embedding the task in instruction

7. The Professional Development Series • Day 4: December 2012 – January 2013 • Audience: Expanded district teams with additional classroom teachers • Facilitated by SDCOE Mathematics & Assessment Staff • Focus: • Score student work from the performance task • Analyze student performance for mathematical understanding and misunderstanding • Plan a re-engagement lesson

8. The Program Logic Model • Context for using a Logic Model • Learning and Leadership Services Division began exploring tools to help us better determine the impact of our work on district systems, instructional leadership, teacher practice, and student learning. • Logic model introduced as a tool to help us frame our services within an outcome orientation • For planning • For program evaluation • Logic model completed via a series of guiding questions

9. The Program Logic Model • What is our vision and our current situation? • Who do we serve? • What end result do we desire – how will the “rules of the game” change? • What new behaviors will we see? • What will change right away? • What do we do? • What will we count? • What will we need to make it happen? • How will we know we’re on the right track?

10. The Program Logic Model Situation / Vision: Question 1 Target Audience: Question 2 Assumptions, Environmental / External Factors Strategies ResourcesQuestion 8: What will we need to make it happen? Activities Question 6: What do we do? Outputs Question 7: What will we count? Outcomes Long-Term Question 3: How will the rules of the game change? Short-Term Question 5: What will change right away? Intermediate Question 4: What new behaviors will we see? Indicators Question 9: How will we know we’re on track?

11. Vision • Mathematics instruction will reflect the common core instructional shifts, and will embed high-quality tasks and formative assessment strategies to gauge students’ conceptual understanding and re-engage students in mathematical thinking.

12. Current Situation • Current classroom instructional practices generally reflect training aligned to curricular materials and prior state content standards. • Teachers have had limited professional learning and experience with embedding high-quality, open-ended mathematics tasks and formative assessment into instructional practice. • Student achievement in mathematics has improved, but continues to show weaker performance in middle and high school grade levels and courses.

13. Target Audience • Mathematics teacher leaders • Site administrators • District curriculum / instruction / assessment administrators

14. Resources • SDCOE Mathematics and Assessment staff (managers and support staff) • Time for collaborative planning and delivery of professional development • Funding to contract for SVMI member resources • Funding for costs associated with professional development sessions

15. Activities • Four-part mathematics performance task and formative assessment professional development series • Contract for SDCOE and small districts in the county to access SVMI resources

16. Output and Indicators • Number of professional development sessions • 1 four-part series; 9 total sessions (Day 3 offered four times; Day 4 offered three times) • Number of participating teachers and administrators • 135 participants from 14 districts and 4 independent charters • Number of teachers that select and administer a task from SVMI MARS or other resource • 88% of participating classroom teachers administered a task • Number of teachers that score and analyze student work • 76% brought back student work to score and analyze

17. Short-term Outcomes and Indicators • Increased awareness of how mathematics performance tasks and formative assessment can improve instruction and student learning • 94% reported better understanding of assessment in relation to CCSS • 78% reported better understanding of analyzing student work to identify students’ mathematical understanding

18. Short-term Outcomes and Indicators • Increased awareness of where to find mathematics performance tasks, rubrics, and scored student papers • 78% reported better understanding of how to find and access identified websites to select performance tasks • Increased awareness of how to select and administer a performance task, and score student work using a rubric with training & standardizing papers • 89% reported the PD helped them plan to administer a performance task with their students • 86% reported the PD helped them score and analyze their students’ work

19. Intermediate Outcomes & Indicators • Teachers try out performance tasks in classrooms, and score and analyze student work • 88% tried out a performance task • 76% scored and analyzed student work • Teachers use student work for professional conversation • 100% of teachers who brought back work engaged in professional conversation on Day 4 • Proposed use of teacher learning logs to document continuing practice • Teachers share student work and instructional practices through PLCs • Proposed use of PLC logs and observations at a sample of sites

20. Long-term Outcomes & Indicators • Teachers and students engage in rigorous, coherent, and focused mathematics learning experiences • Classroom practice reflects high-quality mathematics instruction and assessment • PLCs are focused on student learning and instruction • Student achievement in mathematics improves • Teacher learning logs and classroom observations • PLC logs and observations • District / Site and Smarter Balanced assessment results

21. Learnings & Next Steps • What we learned about supporting teacher use of mathematics tasks and formative assessment strategies • More time was needed on re-engagement portion of mathematics performance task & formative assessment work • Teachers need continued support to embed tasks and formative assessment strategies into ongoing instructional practice • School site structures need to support teacher collaboration around selecting tasks, analyzing student work, and re-engaging students in mathematics learning

22. Learnings & Next Steps • What we learned about collecting “evidence of impact” • We were able to collect evidence of short-term outcomes in terms of increased awareness • We were able to collect limited evidence of intermediate outcomes in terms of changing practice • We need a structured plan and resources (human, time) to collect evidence of intermediate and longer-term outcomes

23. Learnings & Next Steps • What we learned about the program logic model • The approach helped us better define our service(s) and our desired outcomes • The tool helped us differentiate short, intermediate, and long-term outcomes and identify appropriate evidence to collect to assess impact • The process helped us understand that, to assess long-term outcomes / impact, we need a multi-year project and ongoing data collection methods

24. Contact Information • Sally J. Bennett-Schmidt, Assessment Director • San Diego County Office of Education • 858-292-3688 • sbennettschmidt@sdcoe.net