1 / 12

Statistical power in educational settings

Statistical power in educational settings. Workshop at Wellcome seminar on educational research, May 2008 Dylan Wiliam Institute of Education, University of London www.dylanwiliam.net. The argument…. Premise 1 Learning is insensitive to instruction Measures of learning even more so

costin
Télécharger la présentation

Statistical power in educational settings

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Statistical power in educational settings Workshop at Wellcome seminar on educational research, May 2008 Dylan Wiliam Institute of Education, University of London www.dylanwiliam.net

  2. The argument… • Premise 1 • Learning is insensitive to instruction • Measures of learning even more so • So even small system-wide gains in learning are educationally important • Premise 2 • Education systems are inherently multi-levelled • Taking account of clustering in data lowers statistical power • Educational experiments are inherently weak • Conclusion • RCTs in education frequently need to be very large, and therefore expensive

  3. Learning is slow… 860+570=? Source: Leverhulme Numeracy Research Programme

  4. …especially for deep learning… Hart, 1981

  5. …and measures are insensitive… Sequential tests of educational progress (ETS, 1957)

  6. TIMSS NAEP …and measures are insensitive…

  7. …so small gains in learning are worthwhile • Average rate of progress of cohorts is 0.3 standard deviations per year • Average cost of one year’s education for a cohort in England is £3bn • An effect size of 0.05 sd might be regarded as “small” • But system-wide, is worth £6bn

  8. …but hard to detect… • Statistical power: The likelihood that a statistical test will reject a false null hypothesis • Depends on • The level set for statistical significance • The size of the difference between compared groups (effect size) • The sensitivity of the measures • Clustering reduces statistical power, but is an inherent feature of educational settings, and especially for school-wide interventions • Teacher quality • Ability grouping

  9. …especially in educational settings p = #students n = #classrooms  = effect size c= classroom clustering s= school clustering (Konstantopoulos,2006)

  10. So… • The most important question is not “Are RCTs good?” but “When are RCTs good?” • How should we answer?

  11. Institute of Education Sciences (USA) • Five goals • identify existing programs, practices, and policies that may have an impact on student outcomes and the factors that may mediate or moderate the effects of these programs, practices, and policies; • develop programs, practices, and policies that are theoretically and empirically based; • evaluate the efficacy of fully developed programs, practices, and policies; • evaluate the impact of programs, practices, and policies implemented at scale; • develop and/or validate data and measurement systems and tools.

More Related