Conducting Randomized Evaluations in Schools

1. Conducting Randomized Evaluations in Schools Lisa Markman Associate Director Education Research Section Princeton University

2. Education Research Section • Promotes the use of research in education decision-making. • Conducts and promotes high quality education research through the use of experimental and quasi-experimental research designs

3. No Child Left Behind, Schools and Research • No Child Left Behind Act (2001) requires programs and practices to be based on “scientifically based research”. • Term “scientifically based research” appears throughout the law over 100 times. • Applies to curricular interventions, professional development programs, psychosocial interventions and supplemental education services.

4. What is Scientifically Based Research ? The term scientifically based research means research that involves the application of rigorous, systematic, and objective procedures to obtain reliable and valid knowledge relevant to education activities and programs (According to NJ Dept. of Ed)

5. Necessary Components of SBR • Employs systematic, empirical methods that draw on observation or experiment • Involves rigorous data analysis that are adequate to test the stated hypotheses and justify the general conclusions drawn • Relies on measurement or observable methods that provide reliable and valid data • Is evaluated using experimental or quasi-experimental designs with appropriate control or comparison groups (with a preference for random assignment) • Reports results with sufficient detail and clarity • Has been subject to external (peer) review (According to NJ Dept. of Ed)

6. Undesirable Features of Many Evaluations No comparison group or comparison group is not comparable to treatment group Only “successfully” treated schools/students are included in the analysis The evaluators are not independent of the developers of the program Desirable features of Evaluations Participants are randomly assigned to “treatment” status All initial participants are included in the analysis The evaluators are independent of the developers of the program Evaluations in Education

7. Recipe for an Ideal Measurement Step 1: Randomly assign students. Step 2: Obtain PERFECT test. Step 3: Test students in fall. Step 4: Test students in spring. Step 5: Set non-school parameters back to original conditions. Step 6: Repeat Steps 3 & 4. Step 7: Impact = (Step 4 – Step 3) Year 2 - (Step 4 – Step 3) Year 1

8. The “2nd Best” OptionRandomization Randomly assign some students, classes, school districts, or states to the reform in question; the others are randomly assigned to the status quo (or another reform).

9. Why Randomize ? To obtain an appropriate control group Allows one to answer what would happen in the absence of the intervention (creates a credible counterfactual)

10. Common Concerns About Randomized Evaluations • It is too disruptive • Unethical to deny service to some children • Legal obligations require providing services to all children • It does not answer all questions • Fear of “random”

11. Is It Too Disruptive ? • It can be. • Good researchers work with teachers, schools and school districts to understand school culture, and school needs.

12. Example of a Randomized Research Evaluation • To determine the effectiveness of a computerized math curriculum • 3 Urban School districts • Randomize on the class level by school

13. Randomized Evaluation of A Computerized Math Curriculum Current School Schedule

14. Randomized Evaluation of A Computerized Math Curriculum Current School Schedule Below is a sample schedule that would be returned to each school after random assignment

15. It’s Unethical/Illegal to Deny Service • We don’t know if program/reform actually works • Ways around completely excluding all students • All children receive “something” or can contrast two programs

16. It Doesn’t Answer All Questions • True • Benefit of Mix-Methods Approach

17. The Fear of “Random” • Usually can not treat all students • Way to organize groups to get meaningful information

18. To Sum…. • Need more high-quality impact evaluations to help guide everyday decisions • Conducting evaluations does not have to be super disruptive • Schools should make sure that researchers work with them • It’ll take cooperation from all to improve the quality of education research on impacts

19. Information for Librarians • What Works Clearinghouse • Education Databases

20. What Works Clearinghouse • Created as a resource for decision makers in education. Provides databases and reports regarding the effectiveness scientifically based programs and practices. • Current topics – • Adult Literacy, • Character Education, • Delinquency, Disorderly and Violent behavior, • Dropout Prevention, • English Language Learners, • Math (Elementary and Middle School) • Peer Assisted Learning • Reading, • Early Childhood • http://www.whatworks.ed.gov

21. Examples of Education Databases • National Center for Education Statistics (NCES) • National Assessment of Educational Progress (NAEP) • High School and Beyond (HS&B) • The Common Core of Data (CCD) • http://nces.ed.gov • National Longitudinal Study of Youth (NLSY) • NLSY97 • NLSY79 • http://www.bls.gov/nls • Check websites for a list of all databases