130 likes | 268 Vues
VALID AND RELIABLE PHYSICAL, LIFE, AND EARTH SCIENCE CONTENT ASSESSMENTS FOR MIDDLE SCHOOL TEACHERS. Thomas R. Tretter Sherri L. Brown William S. Bush University of Louisville. National Evaluation Institute conference, October 2009 Louisville, KY. Overview of Presentation.
E N D
VALID AND RELIABLE PHYSICAL, LIFE, AND EARTH SCIENCE CONTENT ASSESSMENTS FOR MIDDLE SCHOOL TEACHERS Thomas R. Tretter Sherri L. Brown William S. Bush University of Louisville National Evaluation Institute conference, October 2009 Louisville, KY
Overview of Presentation • Purpose of assessments • Structure • Validity • Reliability • Initial results
Purpose of Assessments • Need for skilled science teachers is high • One important component is strong science content knowledge • Indicators of teacher content knowledge often indirect (e.g. number of courses taken, GPA) • Diagnostic assessments would be useful to professional development providers and teachers themselves • Target middle school science teachers
Structure of Assessments • 3 content assessments: physical, life, earth/space • Teacher content knowledge synthesized from national standards (Breadth of Knowledge) • AAAS Benchmarks NSTA • NSES PRAXIS • NAEP INTASC • TIMSS • Four categories of knowledge (Depth of Knowledge) • Li and Shavelson (2001), Shulman (1986) • 1. Declarative 3. Schematic • 2. Inquiry 4. Pedagogical • STS (crosses both breadth and depth)
Item Specification Grid (Physical Science Example) Depth of Knowledge Breadth Items simultaneously evaluate two separate dimensions (and are distributed with each major content domain) Some items also simultaneously address STS knowledge 20 multiple choice (1 pt. each), 5 open response (3 pts each)
Validity • Content validity from basis in standards • Test items created by teams consisting of: • science educators, teachers, scientists • Construct validity from evaluation by reviewers • Each item evaluated by national sample of approx. 30 external reviewers from same 3 groups • Six nominally parallel versions of each item passing external review were generated and randomly assembled into 6 test versions for each content area
Internal Reliability (N=1787) Physical Life Earth/Space Science ScienceScience Version Alpha N Alpha N Alpha N 1 .871 289 .839 328 .754 98 2 .818 69 .868 163 .626 29 3 .873 154 .841 69 .815 17 4 .833 110 .812 110 .788 113 5 .860 33 .678 63 .848 46 6 .815 38 .784 38 .869 10 Cronbach Alpha
Initial Results • Certification grade level groups • Elementary Middle High • Content certification groups • No science Other science • General science Target science
Results-Grade level certification Physical Life Earth/Space Science ScienceScience 13.5 (5.96) 16.1 (5.54) 13.8 (5.44) (n=76) (n=95) (n=45) 15.3 (6.74) 19.0 (6.53) 17.5 (5.59) (n=163) (n=209) (n=143) 20.9 (6.89) 22.4 (5.26) 19.1 (4.77) (n=133) (n=189) (n=48) p<.001 p<.001 p<.001 Note. Maximum score on all assessments is 35. Brackets indicate Tukey HSD post hoc homogenous subsets. ELEM MID HIGH
NO SCIENCE GENERAL SCIENCE OTHER SCIENCE TARGET Results-Science certification Physical Life Earth/Space Science ScienceScience 13.2 (5.54) 17.3 (5.77) 14.6 (4.63) (n=195) (n=226) (n=130) 19.2 (7.26) 21.1 (6.10) 18.9 (6.26) (n=116) (n=155) (n=71) 18.8 (6.08) 19.4 (5.54) 18.1 (6.48) (n=45) (n=23) (n=29) 23.8 (7.01) 23.5 (5.31) 19.1 (4.69) (n=51) (n=116) (n=35) p<.001 p<.001 p<.001 Note. Maximum score on all assessments is 35. Brackets indicate Tukey HSD post hoc homogenous subsets.
Summary • Extensive development process enhances validity • Cronbach Alpha Reliability >0.8 for most versions • Feedback includes subscores on: • - content subdomains (3 or 4) • - type of knowledge subdomains (4) • - STS • Also item-by-item scores for each individual provided • Cost to score = $10 per teacher per assessment
Possible Uses • Pretesting to determine focus of content area for designing workshops or courses • Pre and post testing for evaluating workshops and courses • Diagnostic tool for teacher self-assessment • Measuring content knowledge for research—both as a treatment variable and an independent variable
More Information Homepage for this project (including mathematics assessments) at http://louisville.edu/education/research/centers/crmstd To request electronic copies of the assessments for use in projects, please email the assessment coordinator CRMSTD@louisville.edu