Moving to Greatness by Assessing Outcomes Using Data

1. Moving to Greatness by Assessing Outcomes Using Data

2. Session leaders • Tessie Bailey • Mary Brownell • Judi Littman

3. DISCLAIMER This content was produced under U.S. Department of Education, Office of Special Education Programs, Award No. H325A120003. Bonnie Jones and David Guardino serve as the project officers. The views expressed herein do not necessarily represent the positions or polices of the U.S. Department of Education. No official endorsement by the U.S. Department of Education of any product, commodity, service, or enterprise mentioned in this website is intended or should be inferred.

4. Activator Activity: Table Task • What questions do you or your colleagues have about assessing intermediate and long-term outcomes? • Write each question on a post-it note.

5. Assessing Outcomes: Starting with a Theory of Change

6. “. . .if there is no agreement on what good performance is and no way to tell what changes would improve performance, then it is very difficult -- often impossible -- to develop effective training methods.” Ericssson, A. & Pool, R. (2016). Peak: Secrets from the new science of expertise. New York: Houghton Mifflin Harcourt.

7. How to choose an outcome • What provides a signal on educator performance versus diagnostic information? • What portion of the measurement system do you have the most control over? • What are the human and financial costs of a particular system, including collecting data, analyzing it, and making sense of it? • How well aligned are the measures? • How will you use the measures (e.g., high stakes or development, simulated or natural observation)?

8. Theory of Change

9. Getting Started Don’t let great be the enemy of good as a starting point, but work on being great over time.

10. Defining Data Literacy: Four Essential Elements Tessie Rose Bailey, Ph.D. American Institutes for Research

11. Who are we? Check us out, https://ncsi.wested.org/

12. Lessons Learned About Data Use Ensure common understanding of data literacy. Provide a continuum of professional learning opportunities. Build infrastructure to support data literacy. Engage key stakeholders in the data use.

13. Build Common Understanding of Data Literacy at All Levels of the System Educator Preparation Increased Student Outcomes

14. Data Literacy Continuum: Essential Elements Data Exploration Data Management Data Use Improving Data Literacy • What? • Data definitions • Data selection • Data collection/ access • Data submission • Data fidelity • Data management system/storage • Considerations • Costs, time, feasibility, burden on stakeholders • What? • Data analysis • Data use/data-based decision making • Data sharing and reporting • Considerations • Balance of use…systems improvement, instructional decision making, reporting • Use depends on role • What? • Purpose and vision (set the path) • Research questions (what do we want/need to know) • Considerations • Who are the impacted stakeholders? • Establishing buy-in prior to moving forward • What? • Efficiency and effectiveness • Data integration • Sustainability of data use/management • Scaling processes • Improving fidelity • Did we answer our questions? • Considerations • How can we improve costs, time, feasibility, data system, burden on stakeholders for data use?

15. Data Literacy Continuum: Essential Elements. Data Exploration Data Management Data Use Improving Data Literacy • EXAMPLES • Do we prepare are teachers effectively? • Are faculty effective? Which teachers need more support? • How can we more efficiently use faculty? Budget? Time? • To what extent are teachers implementing with fidelity? • Does our course content align with evidence-based practices? • Are my students grasping what I am teaching? Are they generalizing learned skills to novel situations? • Are our assessments valid? Am I testing too much? • Are students prepared for the teaching in target settings: urban, rural, HS/MS, MTSS, etc..? • Are we providing sufficient practice opportunities? Are our practice opportunities high quality? • What? • Data definitions • Data selection • Data collection/ access • Data submission • Data fidelity • Data management system/storage • Considerations • Costs, time, feasibility, burden on stakeholders • What? • Efficiency and effectiveness • Data integration • Sustainability of data use/management • Scaling processes • Improving fidelity • Did we answer our questions? • Considerations • How can we improve costs, time, feasibility, data system, burden on stakeholders for data use? • What? • Data analysis • Data use/data-based decision making • Data sharing and reporting • Considerations • Balance of use…systems improvement, instructional decision making, reporting • Use depends on role • What? • Purpose and vision (set the path) • Research questions (what do we want/need to know) • Considerations • Who are the impacted stakeholders? • Establishing buy-in prior to moving forward

16. Data Literacy Continuum: Essential Elements.. • EXAMPLES • Do we have a valid indicator of the outcome of interest? • Do staff understand how different types of data can be used? • Do we have valid data? • Do we have the right amount of data? • Where store/access data? Data Exploration Data Management Data Use Improving Data Literacy • What? • Data definitions • Data selection • Data collection/ access • Data submission • Data fidelity • Data management system/storage • Considerations • Costs, time, feasibility, burden on stakeholders • What? • Data analysis • Data use/data-based decision making • Data sharing and reporting • Considerations • Balance of use…systems improvement, instructional decision making, reporting • Use depends on role • What? • Purpose and vision (set the path) • Research questions (what do we want/need to know) • Considerations • Who are the impacted stakeholders? • Establishing buy-in prior to moving forward • What? • Efficiency and effectiveness • Data integration • Sustainability at data use/management • Scaling processes • Improving fidelity • Did we answer our questions? • Considerations • How can we improve costs, time, feasibility, data system, burden on stakeholders for data use?

17. Data Literacy Continuum: Essential Elements… Data Exploration Data Management Data Use Improving Data Literacy • What? • Data definitions • Data selection • Data collection/ access • Data submission • Data fidelity • Data management system/storage • Considerations • Costs, time, feasibility, burden on stakeholders • What? • Data analysis • Data use/data-based decision making • Data sharing and reporting • Considerations • Balance of use…systems improvement, instructional decision making, reporting • Use depends on role • What? • Purpose and vision (set the path) • Research questions (what do we want/need to know) • Considerations • Who are the impacted stakeholders? • Establishing buy-in prior to moving forward • What? • Efficiency and effectiveness • Data integration • Sustainability of data use/management • Scaling processes • Improving fidelity • Did we answer our questions? • Considerations • How can we improve costs, time, feasibility, data system, burden on stakeholders for data use?

18. Data Literacy Continuum: Essential Elements…. Data Exploration Data Management Data Use Improving Data Literacy • What? • Data definitions • Data selection • Data collection/ access • Data submission • Data fidelity • Data management system/storage • Considerations • Costs, time, feasibility, burden on stakeholders • What? • Data analysis • Data use/data-based decision making • Data sharing and reporting • Considerations • Balance of use…systems improvement, instructional decision making, reporting • Use depends on role • What? • Purpose and vision (set the path) • Research questions (what do we want/need to know) • Considerations • Who are the impacted stakeholders? • Establishing buy-in prior to moving forward • What? • Efficiency and effectiveness • Data integration • Sustainability of data use/management • Scaling processes • Improving fidelity • Did we answer our questions? • Considerations • How can we improve costs, time, feasibility, data system, burden on stakeholders for data use?

19. Data Literacy Continuum: Essential Elements….. Data Exploration Data Management Data Use Improving Data Literacy • What? • Data definitions • Data selection • Data collection/ access • Data submission • Data fidelity • Data management system/storage • Considerations • Costs, time, feasibility, burden on stakeholders • What? • Data analysis • Data use/data-based decision making • Data sharing and reporting • Considerations • Balance of use…systems improvement, instructional decision making, reporting • Use depends on role • What? • Purpose and vision (set the path) • Research questions (what do we want/need to know) • Considerations • Who are the impacted stakeholders? • Establishing buy-in prior to moving forward • What? • Efficiency and effectiveness • Data integration • Sustainability of data use/management • Scaling processes • Improving fidelity • Did we answer our questions? • Considerations • How can we improve costs, time, feasibility, data system, burden on stakeholders for data use?

20. Activity • Place your sticky notes in the corresponding element. • Discussion: • What do you see? What do these questions indicate about the staff’s data literacy? • What areas are most difficult for schools? What about the district? • What areas of data literacy are strengths? Where could we improve?

21. Data Literacy Continuum: Essential Elements…… Data Exploration Data Management Data Use Improving Data Literacy Provide a Continuum of Professional Learning Build Infrastructure Engage Key Stakeholders

22. National Resources to Support Effective Use of Data at the Local Level • National Center on Intensive Intervention, www.intensiveintervention.org • National Center on Educational Outcomes, Moving Your Numbers, http://www.movingyournumbers.org/ • National Center on Improving Literacy, https://improvingliteracy.org/ • National Center on Systemic Improvement, https://ncsi.wested.org/

23. References and Resources • Data Quality Campaign (2018, Sept 12). What Parents and Teachers Think About Education Data. Retrieved from https://dataqualitycampaign.org/resource/what-parents-and-teachers-think-about-education-data/. • Garrett, R., Citkowicz, M., & Williams, R. (2019, in press). Meta-Analysis of RCT’s Targeting Classroom Practice. Review of Research in Education. • Ruedel, K., Nelson, G., & Bailey, T. (2018). Data Use Multi-State Spotlight: Using Data Fidelity Tools to Improve Data Quality. National Center on Systemic Improvement. Retrieved from https://ncsi-library.wested.org/resources/215 • Bailey, T., Nelson, G., Weingarten, Z. and Ruedel, R. (2018). State Data Use Spotlight: Florida Builds Local Data-Use Capacity. National Center on Systemic Improvement. Retrieved from https://ncsi-library.wested.org/resources/235 • Ruedel, K., Nelson, G., Bailey, T. & Blackmon, D. (2018). State Data Use Spotlight: West Virginia increases graduation rates for students with disabilities. National Center on Systemic Improvement. Retrieved from https://ncsi-library.wested.org/resources/180

24. THANK YOU!http://ncsi.wested.org | @TheNCSI

25. HLP Partner Work – Data Exploration • Vision – develop shared understanding of focal HLPs between EPP and P-12 partners in support of teacher candidates • Are we preparing candidates to enact focal HLPs? • Who or what can help us answer these questions? • P-12 partners • Teacher candidates • Professional knowledge base

26. HLP Partner Work – Data Exploration. • Vision – develop shared understanding of focal HLPs between EPP and P-12 partners in support of teacher candidates • Flexible Grouping/Setting Up and Managing Small Group Work • Are we preparing candidates to enact focal HLPs? • Who or what can help us answer these questions? • P-12 partners • Teacher candidates • Professional knowledge base

27. HLP Partner Work – Data Exploration.. • Vision – develop shared understanding of focal HLPs between EPP and P-12 partners in support of teacher candidates • Flexible Grouping/Setting Up and Managing Small Group Work • Are we preparing candidates to enact focal HLPs? • Do they “know” the HLPs? • Can they “do” the HLPs? • Who or what can help us answer these questions? • P-12 partners • Teacher candidates • Professional knowledge base

28. HLP Partner Work – Data Exploration… • Vision – develop shared understanding of focal HLPs between EPP and P-12 partners in support of teacher candidates • Flexible Grouping/Setting Up and Managing Small Group Work • Are we preparing candidates to enact focal HLPs? • Do they “know” the HLPs? • Can they “do” the HLPs? • Who or what can help us answer these questions? • P-12 partners • Teacher candidates • Professional knowledge base

29. General Education: Setting up and managing small group work • Teachers use small group work when instructional goals call for in-depth interaction among students and in order to teach students to work collaboratively. To use groups effectively, teachers choose tasks that require and foster collaborative work, issue clear directions that permit groups to work semi-independently, and implement mechanisms for holding students accountable for both collective and individual learning. They use their own time strategically, deliberately choosing which groups to work with, when, and on what. (TeachingWorks.com)

30. Special Education: Use flexible grouping. • Teachers assign students to homogeneous and heterogeneous groups based on explicit learning goals, monitor peer interactions, and provide positive and corrective feedback to support productive learning. Teachers use small learning groups to accommodate learning differences, promote in-depth academic-related interactions, and teach students to work collaboratively. They choose tasks that require collaboration, issue directives that promote productive and autonomous group interactions, and embed strategies that maximize learning opportunities and equalize participation. Teachers promote simultaneous interactions, use procedures to hold students accountable for collective and individual learning, and monitor and sustain group performance through proximity and positive feedback. • McLeskey, J., Barringer, M-D., Billingsley, B., Brownell, M., Jackson, D., Kennedy, M.,Lewis, T., Maheady, L., Rodriguez, J., Scheeler, M. C., Winn, J., & Ziegler, D. (2017, January). High leverage practices in special education. Arlington, VA: Council for Exceptional Children & CEEDAR Center.

31. HLP Partner Work – Data Management • How do we define or decompose the focal HLPs? • How do we define effective enactment of HLPs? • Can we reliably observe effective enactment? • How do we collect and share data?

32. General Education: Setting Up and Managing Small Group Work

33. Simulations Simulations & Observer Feedback Observer Feedback Form

34. Special Education: Flexible Grouping DRAFT

35. HLP Partner Work – Data Use • Instructional Improvement • Coursework scenarios • Simulations • Live enactment • Model for continuous improvement • Accreditation reporting