Coaches as Critical Consumers of Research

1. Coaches as Critical Consumers of Research Statewide Coaches Meeting Oregon Reading First Center Rachell Katz Jeanie Mercier-Smith April 24, 2008

2. Objectives • Scientifically Based Reading Research • Consumer’s Guide • Scientifically Validated/ Researched Instruction • What Works Clearinghouse • “Look fors” in research studies • Follow up on Tim Shanahan • Resources to keep us up to date on research

3. The Challenge in Education • No “Food and Drug Administration” for Education • Medical profession has a rigorous process for researching drugs and surgical procedures before they can be used on people. • In education we rely on publishers to produce core programs and interventions that will result in improvements in students’ skills.

4. Research Based vs Proven Effective by Research • “Until an instructional practice has been implemented, evaluated and found to produce better results than its alternatives, there is no research basis for recommending it.” (Grossen, 1996) • What are some challenges for practitioners about this statement?

5. Scientifically Based Reading Research • Reading Instruction is based on the science of reading- the five big ideas that have been identified by an extensive review of the research to be critical in reading development. • National Reading Panel • Lots of programs are using the buzzwords, but may not actually teach them. • Effective instructional strategies (modeling, explicit instruction, multiple opportunities to respond, systematic error correction).

6. Evaluating Core Programs • Does it teach all the relevant essential elements? • Are the design and delivery adequate for the majority of learners?

7. Examining Program Content The “Consumers Guide” provides a common metric for evaluating: • Scope of review and prioritization of skills • Quality and nature of the delivery of instruction

8. Examining Scope of Review & Prioritization • The reading program’s scope and sequence should provide evidence of breadth and depth of coverage on essential skills. High Priority Items in Kindergarten

9. High Priority Items: Grade 1 Phonics Instruction

10. Scientifically Researched/ Validated Interventions • Studies that have carefully controlled implementation of practices or programs to show that the program has resulted in an increase of students skills when compared to another instructional approach.

11. “Research Shows…” • NCITE Tool: Evaluating Trustworthiness in Educational Research That is Used to Support a Change in Teaching or School Practice. • Is the statement the result of a single study that was conducted with 40 students? • Is the statement the result of a research review? • Lots of research converging on the same conclusion • Meta-analysis

12. Findings that are not causal often are stated as causal Mathematica Policy Research Inc.

13. The correlation prompts a causal recommendation But.. The study showed a correlation It did not measure the extent to which reducing class size will reduce expulsions Mathematica Policy Research Inc.

14. What does “effective” mean? Program 3 appears to be most effective Mathematica Policy Research Inc.

15. What would have happened without the program matters Program 1 appears to be most effective Mathematica Policy Research Inc.

16. Observations • Showing causality is hard • Common for findings that are not causal to be stated as causal • Syntheses help to see a bigger picture but often present selective findings Mathematica Policy Research Inc.

17. What Works Clearinghouse http://ies.ed.gov/ncee/wwc/ • “What’s Been Well Researched Clearinghouse” • Review of the research that has been conducted on various educational issues • Only reading programs involved in rigorous research studies of the highest quality were given a positive rating. • Therefore a strong program that has not been tested in a high quality research study is not going to receive an endorsement by WWC.

19. WWC Ratings • Improvement Index- difference between the percentile rank of the average student in the intervention group and the percentile rank of the average student in the comparison group. (range of -50 to +50 with positive numbers indicating results favorable to the intervention group) • Evidence Rating- considers four factors • Quality of the research design • Statistical significance of the findings • Size of the difference between participants in the intervention and comparison conditions • Consistency in the findings across studies • Extent of Evidence- how much evidence was used to determine the intervention rating, focusing on the number and size of the studies. Two possible categories: 1. small 2. medium to large

20. Purpose of What Works Clearinghouse • Implementation and feasibility of implementing programs is not the focus • Focus is on reviewing the research that has been conducted to prove effectiveness of programs

21. “Look Fors” in Research Studies • Random assignment to conditions • Equality of groups at start • No confounds ( i.e. teachers with more experience teach intervention group, practicum students teach control group) • Limitations • Type of population • Length of the Study • Others? • ____________________________________________ • ____________________________________________ • ____________________________________________

22. Practice • Identify one potential problem with this research study. Discuss with a partner. • Within a school, 30 students are randomly assigned either to a teacher who uses a new reading intervention or a teacher who uses the standard curriculum. The outcome used in the analysis is the spring score on a standardized reading test.

23. More Practice • Identify one potential problem with this research study. Discuss with a partner. • The teachers identified 31 children whose mean score on the fall Metropolitan Achievement Test was at about the 10th percentile. Twelve children who did not have conflicts with other requested enrichment classes were scheduled for the fall reading intervention class (intervention plus tutoring group), and seven children were scheduled for the spring class (intervention only group). The remaining 12 children made up a control group.

24. Be a Critical Consumer of StrategiesExample Oregon Reading First Brown Bag Presentation with Timothy Shanahan (March 17th, 2008)

25. Be a Critical Consumer of StrategiesExample • Article referenced by Tim Shanahan in Oregon RF Brown Bag Presentation: Effect of Difficulty Levels on Second-Grade Delayed Readers Using Dyad Reading; Journal of Educational Research (2000) • Possible take home message that I heard: Practice in frustration-level material may result in more reading gains • What assumptions could I make? Practice = Fluency Practice? Reading gains = Increased ORF Reading gains = Increased Comprehension? Let’s look at the research…

26. Summary/Abstract “The authors investigated how far above a poor reader’s instructional level dyad reading should be used to promote the greatest growth in reading level, word recognition, comprehension, and rate. Fifty-one poor readers were identified and randomly assigned to 1 of 3 instructional groups: a) dyad reading at their instructional reading level; b) dyad reading 2 grades above their instructional reading level; and c) dyad reading 4 grades above their instructional reading level. For 95 days/sessions, all groups read 15 min daily during their classroom recreational reading time. They all improved with dyad reading regardless of the difficulty level of materials. Results suggest that the difficulty level of materials used for dyad reading may make a difference in student progress.” • Effect of Difficulty Levels on Second-Grade Delayed Readers Using Dyad Reading; Journal of Educational Research (2000)

27. Be a Critical Consumer of Strategies • Is there converging evidence? • How many studies? • What is the quality of the research? • For each study: • What was the intervention/instructional strategies? • How many participants? • Who were the participants? • Does the analysis make sense? • What were the results? • What were the limitations?

28. Coaches as Consumers of Research • Where can we go for a summary of independent research of programs? • What Work’s Clearinghouse • Association for Supervision and Curriculum Development (ASCD) • Professional Journals Exceptional Children, The Reading Teacher Others? _____________________________________ • Other resources: • National Reading Panel • National Literacy Panel on ELLs • Florida Center for Reading Research

29. Other Resources… Teaching All Students to Read in Elementary School: A Guide for Principals [K-3] Effective Instruction for Adolescent Struggling Readers: A Practice Brief [4-12] Effective Instruction for Adolescent Struggling Readers: A Practice Brief A Synopsis of Writing Next: Effective Strategies to Improve Writing of Adolescents in Middle & High Schools [4-12] The Secondary Literacy Instruction and Intervention Guide [6-12] More…. http://www.centeroninstruction.org

30. http://www.dww.ed.gov/media/EL/ReadingK5/TopicLevel/el_practice_guide.pdfhttp://www.dww.ed.gov/media/EL/ReadingK5/TopicLevel/el_practice_guide.pdf

31. Oregon Reading First Website • Research summaries posted: • Assessment, Coaches, Administrators • Project Data & Norms Does phonemic decoding skill predict reading proficiency for English Learners? Three Year Report on Oregon Reading First: Impact and Implementation Oregon Reading First Teacher Report (2006-07) (relation between DIBELS and SAT-10)

32. Steps to take when you hear “Research Shows…” • Research based or validated by a controlled study? • How many studies have supported the claim? • Check for the “look fors” when reading a study.