Measuring the Effectiveness of Improvement Strategies

1. Measuring the Effectiveness of Improvement Strategies Beyond District Improvement Plans Workshop Indianapolis – November 14, 2007 Division of Compensatory Education/Title I

2. Measuring the Effectiveness of Improvement Strategies Session Outcomes • Participants will understand the critical components of a well-written improvement plan. • Participants will be able to write clear, measurable goals focused on identified student learning needs. • Participants will be able to identify a variety of evidence useful in determining the effectiveness of improvement actions, including the effectiveness of implementation as well as evidence of progress toward achieving desired results.

3. Measuring the Effectiveness of Improvement Strategies Key Points • Effective improvement plans have a clear logic that link the plan’s components and guides action. • Effective implementation of planned strategies begins with clear, measurable goals. • The ongoing collection of evidence and data is needed to measure effectiveness of the actions in two areas: fidelity of implementation and impact on student learning.

4. School and District Improvement Plans Serve Multiple Purposes Improvement Plans • Document compliance • Guide improvement efforts

5. A logic model depicts the theory of change behind the action.

6. Well-planned improvement efforts that follow a clear logic model can impact student achievement • Research-based reading practices • Focused professional development • Coaching, support and feedback • Formative assessments guiding implementation • Establishing a strong reading culture Scott (2007) “Reading First” changed reading instruction resulting in student achievement

7. Types of Assessments for Improvement Planning • Planning (needs assessments) • Formative (feedback enroute) • Summative (outcome and impact) Guskey(2000)

8. Effective Implementation of Strategies Begins with Clear Goals “Would you tell me, please, which way I ought to go from here?” asked Alice. “That depends a good deal on where you want to get to,” said the cat. “I don’t much care where,” said Alice. “Then it doesn’t much matter which way you go,” said the cat. Lewis Carroll From Alice’s Adventures in Wonderland

9. Research Says… • The impact of setting instructional goals on student achievement ranges from 18 to 41 percentile points. (Marzano, 2003) • The power of student engagement in setting and monitoring goals is second only to active participation in impacting student learning. (Wang, Haertel,& Walberg, 1994). • The existence of common goals in schools was…rare and the lack of agreed upon goals makes school unique among organizations. (Schmoker, 1999).

10. Goals are often too broad or vague to guide effective action.

11. Goals are often too broad or vague to guide effective action.

12. SMART Goals: A Critical Step in Improving Student Learning SStrategic and Specific MMeasurable AAttainable R Results-based TTime-bound O’Neill & Cozemius (2006)

13. SMART Goals Tree Diagram:Making goals focused and concrete

14. Goals focus on the “greatest area of need.”

15. Indicators are the evidence used to determine if the goal is being achieved.

16. Measures are the assessments usedto gauge progress on the indicators.

17. Targets are the incremental steps toward a longer-range goal.

18. Breaking down the component parts helps to focus the goal toward action.

19. Measurable goals guide thoughtful action A well-written goal framing a research-based strategy is the foundation for action that can be measured and adjusted as the improvement team implements the plan. McKeever,Nine Lessons of Successful School Leadership Teams, (2003)

20. Clearly identify what teachers need to know and be able to do

21. Identify measures of effective implementation of new practice

22. Teacher professional development should affect student achievement Institute for Education Sciences (2007)

23. An improvement plan that is useful for guiding action… • Follows a logical path • Identifies critical student learning needs • Has clear, measurable goals • Uses research-proven practices • Includes measures of changing practice • And measures of impact of practice on student learning

24. Bartholomew Consolidated School CorporationColumbus, IndianaDr. George Van HornDirector of Special Education

25. Disaggregating Student Data

26. Creating Specific, Measurable Goals

27. Measuring Effect of Practices on Student Achievement

28. Framework for Evaluating Professional Development Collect and Analyze Five Levels of Information1. Participants' reactions 2. Participants' learning 3. Organization’s support and change4. Participants' use of new knowledge and skills5. Students’ learning outcomesGuskey (2002)

29. Framework for Evaluating Professional Development Collect and Analyze of Five Levels of Information 1. Participants' reactions 2. Participants' learning 3. Organization’s support and change 4. Participants' use of new knowledge and skills 5. Students’ learning outcomes Guskey (2002)

30. What guideposts will help to assess progress along the way? “Feedback is the breakfast of champions.” Blanchard (2001)

31. Types of Evidence • Classroom observations or Walkthroughs Principal – Coach - Peer • Teacher logs or documentation of activity • Journals of reflective practice • Teacher portfolios • Teacher surveys • Instructional materials (e.g., lessons, units) • Videotapes • Pre-post documentation of knowledge (K-W-L) Of increased teacher knowledge and/or change in classroom practices

32. Types of Evidence Of impact of instruction on student learning Local assessments • District assessments • Local rubrics or scoring guides • Structured warm up activities • Assessments embedded in units of study • Talking with students • Grades

33. Types of Evidence From published formative assessments • Measures of Academic Progress (NWEA) • Scholastic Reading Inventory (Scholastic) • Progress Assessment Series (Pearson) • Texas Assessment of Knowledge & Skills (Merit) Of impact of instruction on student learning

34. Types of Evidence Of impact of instruction on student learning Descriptive information • Attendance data • Discipline referral data • Surveys • Promotion and graduation data

35. Characteristics of Measures of Effectiveness • Yield reliable information • Can be used efficiently • Are minimally intrusive • Provide feedback that guides action • Are aligned with standards • Are consistent with larger, summative measures • Indicate progress toward a larger goal

36. A well written goal framing a research-based strategy can be measured and adjusted as the plan is implemented.

37. Formative assessments allow for monitoring of the strategy’s impact on student learning.

38. The gathered evidence guides the effective implementation of new instructional skills.

39. Activity: Identifying measures of effectiveness to guide improvement 1. Work with a LEA colleagues or join with another team/person if you are alone. 2. Discuss the sample goal statement and identify two measuresofteachers’ effective implementation of the practiceand two measures effectiveness of this practice for increasing student achievement. 3. Use the “Characteristics of Measures of Effectiveness” tool to review your work.

41. Characteristics of Measures of Effectiveness • Yield reliable information • Can be used efficiently • Are minimally intrusive • Provide feedback that guides action • Are aligned with standards • Are consistent with larger, summative measures • Indicate progress toward a larger goal

42. Measuring the Effectiveness of Improvement Strategies • Useful improvement plans have clear logic that drives the plan’s components and guides the actions. • Effective implementation of planned strategies begins with clear, measurable goals. • A well written goal framing a research based strategy can be measured and adjusted. • Measures of effectiveness evaluate the degree of implementation with fidelity and the impact on student learning.

43. Break-out Session Assistance with measuring effectiveness within district improvement plans • A set-aside time to review and revise district improvement plans and apply knowledge gained here. • Available assistance from Great Lakes East Comprehensive Center: • Stacy Rush – Students with disabilities • Maria Derrick-Mescua – Students learning English • Doug Walker • Jayne Sowers

44. References Blanchard, K. & Johnson, S. (2001). The one minute manager. London: HarperCollins. Guskey, T. R., (1999). Evaluating professional development. Thousand Oaks, CA: Corwin Press. Guskey, T. R. (2002) Does it make a difference? Evaluating professional development. Educational Leadership, 59(6), 45–51. Institute for Education Sciences, (2007). Reviewing the evidence on how teacher professional development affects student achievement. Washington, DC: U.S. Department of Education. Retrieved 11/7/07 from http://ies.ed.gov/ncee/edlabs/regions/southwest/pdf/REL_2007033.pdf

45. References, continued McKeever, B. (2003). Nine lessons of successful school leadership teams. San Francisco, CA: WestEd. Marzano, R. J. (1998). A theory-based meta-analysis of research on instruction. Aurora, CO: Mid-continent Research for Education and Learning. O’Neill, J. & Cozemius, A. (1999) The power of SMART goals: Using goals to improve student learning. Bloomington, IN. Schmoker, M. (1999). Results: The key to continuous school improvement (2nd ed). Alexandria, VA: ASCD. Scott, C. (2007). Reading First: Locally appreciated, nationally troubled. Washington, DC. Center on Education Policy. Retrieved 11/6/07 from http://www.cep-dc.org/index.cfm?fuseaction=document.showDocumentByID&nodeID=1&DocumentID=228

46. References, continued Wang, M. C., Haertel, G. D., & Walberg, H. J. (1993). Toward a knowledge base for school learning. Review of Educational Research, 63(3), 249–294.