Myth or Fact: What does the research say?

1. Myth or Fact:What does the research say? Christina Theokas Education Trust 20th EDUCATION TRUST NATIONAL CONFERENCE NOVEMBER 14, 2008

2. Today We Will Discuss • What is the current relationship between research and practice? • What research evidence exists to inform decision making? • Class Size • Formative Assessment • Professional Learning Communities • Why students drop out of school

3. Today We Will Discuss • How do you read/interpret research findings? • How to be a good consumer of research findings in secondary sources • What are good sources for research information? • About programs, interventions • Topics relevant to school improvement

4. Some history on the perception of education research…(the good and bad)

5. 1971 “The body of educational research now available leaves much to be desired, at least by comparison with the level of understanding that has been achieved in numerous other fields… Research has found nothing that consistently and unambiguously makes a difference in student outcomes.”-Rand report to the President’s Commission on School Finance

6. 1999 “One striking fact is that the complex world of education- unlike defense, health care or industrial production- does not rest on a strong research base. In no other field are personal experience and ideology so frequently relied on to make policy choice, and in no other field is the research base so little used.”-National Research Council

7. Across the Last Decade • Many changes to improve research base IES • Defining quality of evidence • Funding “true experiments” on interventions and curriculum • Identifying “problems of practice” Districts and schools • Increased access and use of student achievement data

8. 2009 "It is what it is. I would certainly say the anecdotal or the real-life experiences that staff or members face are probably always going to trump research."-Education Northwest“Adding the phrase ‘research shows’ significantly increases public-approval ratings of specific education initiatives.”-Education Next

9. Research Practice Research Barriers Practice Barriers Tradition Belief systems Time Resistance to change Changing priorities • Relevance to local issues • Volume of research • Contradictory findings • Not accessible • Not timely Federal policy requires the use of “evidence” to ground improvement efforts. We need a common language and conception of what is reliable and valid data and how to turn data into evidence to help make informed choices.

10. Why is research important? • Information from research is more reliable than information from other sources such as stories, personal experiences, opinions or logical arguments because research is based on systematic gathering of information

11. Class Size

12. Different Types of Evidence • Personal experience • Parents or school board members desire it • The opinion of teachers in a district • The logical argument that smaller classes are better learning environments for students N/A Class Size

13. What don’t we know from that evidence? • What exactly a “small class” is? • Are different class sizes better for different groups of students (by age, ethnicity, poverty, achievement)? • What about small classes are different? • What is the anticipated gain in achievement? • Would another investment yield a bigger return?

14. Anchor Research Studies on Class Size • The STAR Experiment, TN (1985-1989) The Lasting Benefits Study • Randomized experiment • ~ 12,000 students K-3, 1,300 teachers, 76 schools, 42 districts • Small classes 13-17 students, compared to 22-26 • The SAGE Project, WI (1996-2001) • Quasi-experimental • K-3 students in high poverty districts, 30 schools in 21 districts • 15 students per teacher (21-25 in comparison schools) Word, E. et.al., (1990). Student/Teacher achievement ratio: TN K-3 class-size study. Nashville, TN: TN State Department of Education. Molnar, A., et al. (1999). Evaluating the SAGE Program, Educational Evaluation and Policy Analysis, 21 (2), 165-177.

15. What do we know from research… • Statistically significant improvements in reading and math • Benefits were greatest for students who started in small classes in kindergarten • No supporting evidence from research with middle and high school students • Class size reductions need to be substantial • Adding an aid or second teacher to a large classroom to maintain the pupil teacher ratio did not produce increased achievement • These effects appear to be sustained beyond the treatment years • Longer periods in small classes produced higher increases in achievement in later grades for all types of students Word, E., et al. (1990); Molnar, A., et al. (1999); Hanushek, E. (1999); Finn, J., et al. (2001). #1 Does class size reduction work?

16. Small Class Advantage in Months of Schooling Finn. J.D., & Achilles, C. (1999). TN class size study: Findings, implications, misconceptions. Educational Evaluation and Policy Analysis, 21 (2), 97-109. (Average GE of Small Classes Minus Average GE of Regular Classes)

17. What do we know from research… Finn, J.D. (1998). Class Size and students at risk: What is known? What is next? Washington, DC: U.S. Department of Education. #2 Do disadvantaged and minority students benefit more?

18. Can Small Classes Help Close the Achievement Gap • There is some evidence for the cumulative effects of small classes for minority students, particularly in reading, but not for low-income students • All types of students, low, medium and high achievers, benefit from small classes • Lasting benefits are similar for students at all achievement levels • There are some exceptions certain grades, for reading and science, that are promising for affecting the achievement gap Konstantopoulos, S., & Chung, V. (2009). What are the long-term effects of small classes on the achievement gap? Evidence from the Lasting Benefits Study. American Journal of Education, 116, 125-154.

19. What do we know from research… • The research evidence here is more elusive and begins to point out the difficulty in using CSR as a wholesale strategy • Teaching strategies did not significantly differ between small and larger classes. • Teachers spent more time on instruction • Student engagement was higher • Making smaller classes requires hiring teachers, who are often less experienced and these teachers are placed with the low-income and minority students Finn. J., et al (2003). The “Whys” of class size: Student behavior in small classes. Review of Educational Research, 73 (3), 321-368. Hanushek, E. (1999). Some findings from an independent investigation of the TN STAR experiment and from other investigations of class size effects. Educational Evaluation and Policy Analysis, 21 (2). 143-163. #3 How and why does class size reduction work?

20. What do we know from research… This is what it might cost to reduce class size to 17 students in a 1,000-student school (based on an average starting salary of $31,000*). * American Federation of Teachers (2002). Survey and Analysis of Teacher Salary Trends. Washington, DC: American Federation of Teachers. #4 Is class size reduction cost effective?

21. What do you think? How do the practical, economical and best evidence from science come together to inform your opinion?

22. Professional Learning Communities (PLC)

23. What are PLC? • Different names and definition • Communities of Practice • Teacher Learning communities • Grade or subject teams • High school departments • Teacher collaboration • Shared planning • Lesson study Source: USDOE ,FRSS:74 Survey on Prof Development and Training in Public Schools (199982000) * N=5000

24. “PLC are groups of educators who work together to analyze and improve their classroom practice… engaging in an ongoing cycle of questions that promote deep team learning.” Source: Dufour, R. (2004). What is a professional learning community. Educational Leadership, 61 (8), 6-11.

25. Core Attributes of PLC • Shared mission, focus, goals • Focus on learning not teaching • Collective inquiry and reflection • Action orientation and experimentation • Shared leadership and responsibility • Effectiveness is judged on results • Common formative assessments • Objective analysis of data comparing teachers Dufour, R. (2004). What is a professional learning community. Educational Leadership, 61 (8), 6-11. Hord, S. (1997). The Big Ideas

26. “Rather than becoming a reform initiative itself, a professional learning community becomes the supporting structure for schools to continuously transform themselves through their own internal capacity.” Source: Morrissey, M. (2000). PLC: An ongoing exploration. Autsin, TX: SEDL

27. In “solo practice,” Teachers In learning communities, Teachers: Work collaboratively on problems that focus on student learning Focus on jointly creating new knowledge and see learning as on ongoing process Have structured time to observe and reflect on each other’s work and serve as critical friends in support of each other Commit to shared norms, with shared responsibility for growth in learning of all teacher and students in the school, understand others styles and techniques • Are isolated in individual classrooms and work alone • Teach to a set of externally fixed curricular standards that remain static over time • Are given little or not time to work collaboratively with colleagues • Perform in isolation according to externally determined professional standards, establish personal norms that may or may not be the same as their colleagues Source: Carroll et al., (2005). Induction into learning communities. Washington DC: National Commission on Teaching and America’s Future.

28. Research Evidence • Wide variation • Based on local school and district characteristics • Within school variation • Communal groups can organize around the wrong things and reinforce perceptions about teaching and students that does not contribute to or focus on continuous improvement Louis, Marks & Kraus (1996). Teachers professional community in restructuring schools. American Educational Research Journal, 33 (4), 757-798. Implementation

29. Research Evidence • Increased efficacy and collective responsibility • More positive attitudes towards teaching • Lower rates of absenteeism • Higher retention • More authentic pedagogy • Higher order thinking, substantive conversation, deep knowledge, connections to the real world Shachar & Shmuelevitz, 1997; Brownell et al. (1997); Erb (1995); Lee, Smith & Croniger (1995) Teacher Practice

30. Research Evidence We do not have much evidence suggesting which collaborative instructional practices lead to improved student learning but the pattern of change is clear. School Wide Professional Community Challenging Intellectual work Student Achievement Increased motivation and effort Evans-Stout (1998); Lee & Smith (1995); Louis & Marks (1998); Goddard, Goddard & Tschannen-Moran (2007) Student Achievement

31. Formative Assessment (FA) or Assessment for Learning

32. What are formative assessments? • Formative assessment is a process used by teachers and students during instruction that provides feedback to adjust ongoing teaching and learning to improve student’s achievement of intended instructional outcomes. Council of Chief State School Officers

33. Tiers of Assessment Scope Summative Assessment Final performance evaluation Interim/Benchmark Assessments Instructional, evaluative, diagnostic, predictive Standardized data that can be aggregated Formative Assessment Occur in the natural course of teaching, integrated into the lesson Ongoing information about what students are learning and how teachers may improve instruction to remediate gaps and misunderstandings Frequency of Administration Source; Perie, Marion, Gong & Wurtzel (2007). The role of interim assessment in a comprehensive assessment system. Washington, DC: The Aspen Institute.

34. Is this formative assessment? • A high school biology teacher frequently reads aloud a prepared biology-related statement, than asks students to hold their hands under their chins and signify whether the statement is true or false by showing a “thumbs-up” for true and a “thumbs-down” for false. Depending on the number of students who respond incorrectly the teacher may have students present arguments for both sides, he may pair students and ask them to discuss the concept further, or he may decide that he needs to present the same concept using a different instructional approach.

35. Is this formative assessment? • District-developed monthly exams are administered to all students at the end of each of the school year’s first 8 months in reading and math. The exams are based on state-authorized curricular goals for the grade and subject. Results of these tests are sent home to parents. The content covered often does not coincide with what is being taught.

36. Strategies for Evidence gathering • On-the-fly assessments: occurs spontaneously based on what the teachers hears/observes • Planned for interaction: teachers decide how they will elicit students’ thinking during the course of instruction • Curriculum embedded: Feedback solicited at key points during the learning progression Heritage, M. (2007). FA: What do teachers need to know and do? Phi Delta Kappan, 89 (2) 140-145.

37. What data and research are available? • Formative assessments, as discussed, are not standard, so rigorous research assessing causality is limited • ~ 250 studies • Poor to good quality in terms of methodological rigor • Largely with special education populations • Some in lab contexts or with online learning • Specific research on components of FA (feedback) Black & Williams (1998). Assessment and Classroom Learning. Assessment in Education, 5(1), 7-74.

38. What conclusions can be drawn about FA? • Formative assessment has the potential to significantly improve student learning • Effect sizes for formative assessment are typically larger than most other types of interventions (structural or curricular) • There is some evidence that lower performing students benefit more suggesting some impact on the achievement gap is possible • Certain conditions need to be satisfied for FA to be effective Fuch & Fuch (1986); Black & Williams (1998, 2003); Stiggins (2005)

39. Mean Change Scores in Different Types of Classrooms Source: Meisels et al.(2003). Creating a system of accountability: The impact of instructional assessment on elementary children’s achievement scores. Education Policy Analysis Archives, 11(9),

40. Five Attributes of Effective FA • Learning goals and criteria for success are communicated • The learning progression should be identified • Assessments must emphasize the skills, knowledge and attitudes regarded as most important, not just those easy to assess • Descriptive feedback to students that is linked to the instructional outcome is provided • Teachers and students are partners in learning • Students are instructional resources to one another (self and peer assessment) Council of Chief State School Officers FAST SCASS; Stiggins (2005)

41. The Northwest Regional Educational Laboratory has put large sections of its helpful training kit, Improving Classroom Assessment: A Toolkit for Professional Developers online at :http://www.nwrel.org/assessment/toolkit98.asp

42. Dropouts

43. The conversation on dropouts and graduation rates • The scope of the problem • 6,829 students are lost from HS each day (Diploma Counts, 2008) • Problems with graduation rate calculations • The social and economic impact of the problem for individuals and communities • Characteristics of schools with high dropout rates • Characteristics of students who dropout • Risk factors (individual, family, school and community) • Early Warning Systems (9th grade course performance and attendance)

44. This epidemic is inconsistent with the aspirations of most students, so what is going on?

45. Top Reasons Given by Dropouts for Leaving School

46. Dropouts Consistently Report More Push Factors as Reasons Push Factors Pull Factors Students leave school because of circumstances outside of school For example, acquiring a job that does not require a diploma Student interests pulled away from school resulting in detachment • Students leave school due to something in the school environment • For example, giving failing grades after a certain number of absences • Practices slowly alienate students leading to disengagement

47. Dropping Out is Not an Event • Dropouts are not a homogenous group • Patterns vary across subgroups, regions, locations • No single factor or reason causes dropout • Process of Disengagement • Timing of experiences • Turning points • Additive effect Hammond, Linton, Smink & Drew (2007). Dropout risk factors and exemplary programs. Clemson, SC: National Dropout Prevention Center.

48. Contrary to popular belief… • Lowering standards to reduce dropouts will likely have the opposite effect • Rigor, relevance and relationships are key • Students lack motivation • Students report a willingness to work harder and a desire to learn • Families don’t value education • Parents consistently report in surveys that they want their children to go to college. Minority families more often see education as the key to their child’s future.

49. Disengaged Students Recommendations • Opportunities for real world learning • Better teachers who keep classes interesting • Smaller classes with more individual instruction • Extra support (tutoring, summer school, extra time with teachers) • Increase supervision at school; ensure strong relationships with at least one adult at school Bridgeland, Dilulio & Morison (2006). The silent epidemic: Perspectives of high school dropouts. Civic Enterprises.

50. Interpreting Research Studies • What makes the study important? • Who were the participants in the study? • Is there a comparison group • What data were collected? • Tests, questionnaires, surveys, interview, observation • Do the findings make sense? • Are the results statistically significant? • Are the results practically significant? (effect size) • What are the practical implications? N/A Some general questions to ask