Access Center 3 rd Annual Information Sharing Community Meeting

1. Access Center3rd Annual Information Sharing Community Meeting Works in Progress: A Report on Middle and High School Improvement Programs Steve Fleischman, CSRQ Center Director October 4, 2005

2. Presentation Overview Presentation Overview • Why We Need Better Evidence • How to Find and Judge Evidence • Evidence on Middle and High School Improvement Programs • Working together to improve Middle School programs and practice

3. Need for Better Evidence • Claims, Claims, Claims • Need and demand for better evidence to guide school improvement • Importance of matching research methods to questions asked • Importance of implementation • Judging the quality of research (Who does it? Who do you trust?)

4. Three BIG Questions: • What works? • How do you know? • So what?

5. Sources of Evidence Sources of Evidence for Decision Making in Education • Empirical Evidence • Professional Wisdom Why Are Both Needed? • Without professional wisdom education cannot • adapt to local circumstances • operate intelligently in the many areas in which research evidence is absent or incomplete. • Without empirical evidence education cannot • resolve competing approaches • generate cumulative knowledge • avoid fad, fancy, and personal bias Adapted from a presentation by Grover (Russ) Whitehurst, IES Director, US Department of Education (http://www.ed.gov/offices/OERI/presentations/evidencebase.html)

6. How will we know if it’s a strong study ? What to look for in research studies on program or practice effectiveness: • Detailed description of the study sample (Does the program serve kids like mine?) • Indication that the program is the likely source of change in students’ outcomes (RCTs are strongest source of evidence) • Pre and post test • Comparison group • Indication that the findings reported are based on appropriate methods of statistical analysis • Sufficient detail on the findings and implementation of the intervention to allow its replication

7. Resources for Judging Research • Slavin, R.E. (2003). A reader’s guide to scientifically based research. Educational Leadership, 60, 12-16.http://www.ascd.org/publications/ed_lead/200302/slavin.html • Fashola, O.S. (2004). Being an informed consumer of quantitative educational research. Phi Delta Kappan, 85, 532-538. • Stringfield, S. (1998). Choosing Success. American Educator. http://www.aft.org/pubs-reports/american_educator/fall98/ChoosingSuccess.pdf • Lauer, P. A. (2004). A policymaker’s primer on education research: How to understand, evaluate and use it.http://www.ecs.org/html/educationIssues/Research/primer/foreword.asp • Institute of Education Sciences, U.S. Department of Education (2003). Identifying and implementing educational practices supported by rigorous evidence: A user friendly guide.(Prepared by the Coalition for Evidence Based Policy, )http://www.ed.gov/rschstat/research/pubs/rigorousevid/rigorousevid.pdf

8. What do we know about effective programs? • Comprehensive School Reform • Educators’ Guide to Schoolwide Reform (AIR) http://www.aasa.org/Reform/ • CSR Meta-Analysis (Borman et al) http://www.csos.jhu.edu/CRESPAR/techReports/Report59.pdf • Catalog of School Reform Models (NWREL)http://www.nwrel.org/scpd/catalog/index.shtml • Reading • Florida Center for Reading Research http://www.fcrr.org/FCRRReports/reportslist.htm • Math (and, soon, other topics) • What Works Clearinghousehttp://www.whatworks.ed.gov • Substance Abuse • SAMHSA, U.S Department of Health and Human Serviceshttp://modelprograms.samhsa.gov/ • Safe & Supportive Schools • U Colorado, Center for the Study and Prevention of Violencehttp://www.colorado.edu/cspv/blueprints/

9. Center for Effective Collaboration and Practicehttp://cecp.air.org Center for Implementing Technology in Educationhttp://www.citeducation.org Comprehensive School Reform Quality (CSRQ) Centerhttp://www.csrq.org K8 Access Centerhttp://www.k8accesscenter.org/ National Center for Mental Health Promotion and Youth Violencehttp://www.promoteprevent.org National Center for Technology Innovationhttp://www.nationaltechcenter.org National Center on Education, Disability and Juvenile Justicehttp://www.edjj.org National Center on Student Progress Monitoringhttp://www.studentprogress.org National Coordinator Training and Technical Assistance Centerhttp://www.k12coordinator.org National Evaluation and Technical Assistance Center for the Education of Children Who Are Neglected, Delinquent, or At Riskhttp://www.neglected-delinquent.org National Reporting System for Adult Education (NRS)http://www.nrsweb.org Supplemental Educational Services Quality (SESQ) Centerhttp://tutorsforkids.org Technical Assistance Partnership for Child and Familyhttp://www.air.org/tapartnership What Works Clearinghousehttp://www.whatworks.ed.gov AIR Resources for Evidence–Based School Improvement

10. More AIR Resources • Safe, Supportive and Successful Schools: Step by Step (Osher, et al) (www.sopriswest.org) • ASCD Educational Leadership column on Research Matters (www.ascd.org) • CSRQ Center (www.csrq.org) • AIR’s School District Consulting Services (www.air.org)

11. CSRQ Center: What we do CSRQ Center • Produce consumer-friendly CSRQ Center Reports. • Develop partnerships to promote knowledge and use of CSRQ Center reports and tools. • Provide technical assistance in partnership with selected states, districts, and schools.

12. CSRQ Center Reports • Works in Progress: A Report on Middle and High School Improvement Programs (January 2005) • CSRQ Center Reports on Elementary School CSR Programs (Fall 2005) • CSRQ Center Reports on Education Service Providers (Fall 2005) • CSRQ Center Reports on Middle and High School CSR Programs (Fall 2006) • CSRQ Center Reports on Elementary School CSR Programs (revised and expanded, Fall 2006)

13. CSRQ Center Reports: Framework CSRQ Center CSRQ Center Reports are produced using Quality Review Tools (QRT). Reports features: • Basic Program Information • Dimensions of Quality. Strength of evidence of: • Positive Effects on Student Achievement • Positive Effects on Additional Outcomes • Family and Community Involvement • Strong Link between Research and Program Design • Program Provider’s Support for Implementation • Program Provider’s Financial Viability • Evidence of Program Provider’s Capacity to Deliver High-Quality Services to All Schools

14. Works in Progress Report Contents, Suggested Uses and Limitations: • Report Overview (see handout) • The Structure of Works in Progress • Key challenges • Responses to key challenges including research findings • Considerations • Resources and references • CSR Section

15. Works in Progress Report (2) Guidance on Middle School Key Issues: • Transition to Middle School • Literacy and Reading • English Language Learners • Violence and Bullying • Alcohol, Tobacco, and Other Drugs • Parental Involvement • Transition from Middle to High School • CSR Models and Key Issues

16. Works in Progress Report (3) Guidance on High School Key Issues: • Transition to High School • Literacy and Reading • English Language Learners • High School Dropouts • Violence • Alcohol, Tobacco, and Other Drugs • Transition from High School to Postsecondary Settings

17. Works in Progress Report (4) Guidance on High School Key Issues: • Transition to High School • Literacy and Reading • English Language Learners • High School Dropouts • Violence • Alcohol, Tobacco, and Other Drugs • Transition from High School to Postsecondary Settings

18. Application of WIP Works in Progress Report Each group is a school improvement team comprised of teachers and administrators. You are seeking to address the two primary issues in middle or high school you have identified. Discuss the following questions: • How might these resources be useful to you? • How could you use these resources to move to move to next steps on school improvement?

19. WIP: Limitations Works in Progress Report • Survey of Issues-Not Exhaustive (starting point, not ending one) • Need for Better Evidence • Suggestive Evidence; Not Definitive (not a what works” report, but a “desk reference”) • Need for Changes in Multiple Areas (need comprehensive approaches) • Effective Implementation is Key • Need for Alignment with Local Efforts

20. Working Together to Promote Evidence- Based Improvement • Take the lead on insisting that policies, programs, and approaches are evidence-based. (“Show me the evidence.”) • Once made, stick to policies that are based on sound evidence and give them time to work. (“No quick fixes.”) • Work in partnership with the CSRQ Center (“Better Evidence. Better Choices. Better Schools.”)

21. CSRQ Center: Contact Us American Institutes for Research 1000 Thomas Jefferson St, NW Washington, DC 20007-3835 www.csrq.org Steve Fleischman, Director sfleischman@air.org 202/403-5989