1 / 12

Making a Difference

Making a Difference. Update on the National i3 Evaluation of the Implementation and Impact of Diplomas Now. Diplomas Now i3 Validation Study. Goal: Validate effectiveness of the DN model Research Questions - Phase I (current):

taylorfrederick
Télécharger la présentation

Making a Difference

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Making a Difference Update on the National i3 Evaluation of the Implementation and Impact of Diplomas Now

  2. Diplomas Now i3 Validation Study Goal: Validate effectiveness of the DN model Research Questions - Phase I (current): • What is the impact of Diplomas Now on students’ outcomes, particularly with regard to attendance, in-school behavior, and course performance? • What lessons can be learned about implementation of the model during national expansion?

  3. Notable about this Evaluation It’s rigor: • Random assignment design: uses a lottery-like process to assign schools to two equivalent groups (“DN schools” and “Non-DN schools”) • Gold standard for evaluation research: Diplomas Now caused any differences in school and student outcomes • Fairest way to allocate scarce resources It’s size: • 11 school districts, 62 secondary schools • 32 DN, 30 non-DN

  4. Notable about this Study It’s generalizability: • The study districts are broadly representative of urban districts across the country • The schools represent some of the most challenged in the nation • The students represent those traditionally dealing with the most educational and life challenges It’s duration: • US DOE Office of Innovation and Improvement awarded extension grant for the evaluation: follow 2 student cohorts for 4 years • Importance of continuity of implementation of the program model

  5. Diplomas Now i3 Validation Study • Second report (May 2015) on the first and second years of implementation: “fidelity” and “service contrast” • The third report (early 2016) will include the impact of Diplomas Now on students’ ABCs: • Attendance • Behavior • Course Performance

  6. Fidelity of Implementation • How closely the model as implemented in the Diplomas Now schools matched the original design. • Fidelity measures created with support from DNIST to identify the components of the model and to set thresholds for successful implementation of each component. • Data from surveys of Diplomas Now field staff and partner organizations’ data systems.

  7. Fidelity of Implementation • In Year 1 and Year 2, the majority of the model was implemented as designed. • Fidelity measurement based on 111 components clustered in 9 “inputs” and reported on a 0-1 scale • Year 1 = 0.61 and Year 2 = 0.62 • While there is still room for growth, the level of fidelity necessary to achieve impacts on student outcomes is still unknown.

  8. Service Contrast • The extent to which the Diplomas Now model as implemented created differences between practices in Diplomas Now and Non-Diplomas Nowschools. • Created survey items that focused on the kinds of activities implemented in DN schools, but used language that any teacher or administrator could understand. • Teachers and administrators were surveyed in both DN and Non-DN schools.

  9. Service Contrast • Implementation of Diplomas Now made schools different in organization and staff practices than what would have been expected. • In Year 2, across 29 measures: • 26 indicated positive change made by Diplomas Now • 16 of these differences were statistically significant • From Year 1 to Year 2, differences increased, often driven by stability in DN schools and declines in Non-DN schools.

  10. 85 85 84 72 71 53 Career Academy Group Control Group Outcomes and Impacts by Implementation Quality (Career Academies example) 100 100 100 90 90 90 80 80 80 70 70 70 60 60 60 50 50 50 Graduation Rate (%) Graduation Rates (%) Graduation Rates (%) 40 40 40 30 30 30 20 20 20 10 10 10 0 0 0 Program A: Strong Program B: Moderate Program C: Weak

  11. Diplomas Now i3 Validation Study • Research Questions – Phase II (extension): • What is the impact of Diplomas Now on four-year student outcomes: the 9th grade transition of middle grades students and the graduation of high school students • What additional lessons can be learned about implementation of the model during national expansion, particularly about relationships between implementation and outcomes?

  12. For More Information… • See the second evaluation report: http://www.mdrc.org/publication/moving-down-track • See the evaluation project page: http://www.mdrc.org/project/diplomas-now Contacts: William Corrin, MDRC, william.corrin@mdrc.org Susan Sepanik, MDRC, susan.sepanik@mdrc.org

More Related