South Carolina Alternate Assessment (SC-Alt) Advisory Committee Meeting October 17, 2012
A.I.R. Staff • DeeAnn WagnerProject Director • Jennifer Chou Project Manager • Lynnett Wright Alternate Assessment Specialist
Teacher Comments “Tasks are too high for all of my students. The test is very burdensome to assemble.” “My students all did great. Very easy to organize and put together for testing.”
Procedures Used for MonitoringSC-Alt Scoring Fidelity Special procedures are used to monitor test administration effectiveness and scoring consistency A videotaping procedure was used for this purpose for the 2007-2011 administrations The “second rater procedure” was piloted in 2011 and was fully implemented to replace the videotaping for the 2012 administration
Second Rater Procedures The second rater procedure involves having a second trained test administrator (a “second rater”) observe an SC-Alt administration and score the student’s responses on a separate second rater answer folder The second rater scores the student responses independently without consulting or comparing scores with the regular test administrator
Second Rater Procedures(cont.) The scoring by the regular test administrator is used as the official scoring for the student The scoring of the second rater is compared to the scoring of the regular test administrator by the scoring contractor to calculate measures of scoring consistency
Second Rater Pilot Results Results of the pilot in 2011 indicated that the second rater procedure yielded comparable results to the videotaping procedure The second rater procedure was also overwhelmingly preferred over videotaping by test administrators and DTCs-Alt
Advantagesof the Second Rater Procedure The second rater procedure eliminated the problems associated with using video cameras and recording media Teachers reported the procedure to be less stressful than videotaping Teachers reported the procedure was less distracting to students than videotaping.
Second Rater Procedures for 2012 • The teachers were sampled to conduct second rater administrations in the fall prior to test administrator training, allowing the DTC-Alt to assign and train a second rater for each administration • Each sampled teacher was required to conduct a second rater administration with one student for the mathematics administration only
Second Rater Sampling Procedures All districts were sampled Sampling was implemented by teacher and student Teachers were sampled according to proportions of students in their district Approximately 1/3 of teachers and 10% of students were sampled
Second Rater Qualifications • Must meet the test administrator criteria: • Certified teacher • Administrator (e.g., school administrator, district level special education consultant, or other administrator) • Related services personnel • Must participate in test administration training
Second Rater Results Previous results have indicated consistently high rates of scoring agreement at all three form levels (elementary, middle, and high school) For 2012, the average item agreement statistics for mathematics second rater samples were: Elementary form – 97.7%, Middle School form – 97.6%, and High School form – 97.0%. These results are consistent with the scoring consistency results for previous years and confirm a high level of scoring consistency at all three levels.
Second Rater Results Full implementation of the second rater procedure in 2012 was very successful Fewer problems were reported than with videotaping and the identification and training of second raters was not a problem for districts The return rate for complete and scorable second rater materials was significantly greater than for videotaping (94% vs. 79% for 2011)
2013 Second Rater Procedures Teachers will be sampled to conduct second rater administrations from the 2012 SC-Alt data file and the names of the teachers will be provided to districts in November prior to regional test administrator training Second rater administrations for 2013 will be conducted in science only.
Teacher Comments I liked having the second-rater features of the test. Great change!” “Second-rater had trouble with monitoring eye gaze with non-familiar students.” Note: This was also problematic with the videotape raters. Appreciate SPQ and the security affidavit in the answer folder and revised answer worksheet.
Changes for 2013 • Green border instead of green arrow to indicate correct response. • Updated “Try 2” statement • Change in TA “Say” statement • “Tell (show)” changed to “Show (tell)” • “Secure Materials” statement added to response options and other print manipulatives
Task Sharing Plan Share tasks and training materials with New Mexico Tasks will be reviewed by Content and Bias Committee in SC Tasks will be field-tested in SC Results in increasing bank of tasks so that we can replenish at a more rapid pace
Issues for Committee Consideration • Response Options • Purpose of Response Cards • Tense of verbs • Other Response Methods to Consider • Open ended • Other • Use of pictures/picture symbols in ELA
Participation Guidelines for SC-Alt Background and New Guidance on Eligibility Criteria
Participation Ratesof SWD in SC-Altand PASS Is Better Identification Guidance Needed ?
Summary of Changes in Rates of Participation • The overall number of students tested in 2012 increased by 256, which was a 8.1% increase over 2011. The increases for the previous two years were 7.9 and 6.6% respectively. • The steady increases since 2007 have been largely due to the increases for Autism and Mild ID. • The percentages of increase from 2007 to 2012 have been 93% for Autism and 62% for Mild ID, compared to an increase since 2007 of 15.7% for all other students.
Previous Efforts to ImproveIEP Team Testing DecisionsHave Been Ineffective • The increases for 2012in suspect categories of disabilities have occurred in spite of collaboration with the OEC and districts over the past two years.
2012 Rates of Participation • In 2012, the number of Mild ID students approximately equaled the number of Moderate ID students for the first time. • Increases in Autism and Mild ID continued to be major contributors for 2012, however, there were also substantial increases for DD and OHI. • The number of LD students has continued to increase during the last three years and now constitutes 2.4% of SC-Alt students, contrary to what would be expected based on LD identification criteria.
Statewide Percent of All Studentsby Disability Group Taking the SC-Alt • Severe Intellectual Disability – 100% • Moderate Intellectual Disability – 96.5% • Autism – 36.9% • Mild Intellectual Disability – 30.8% • Traumatic Brain Injury – 26.7% • Orthopedic Disability – 23.8% • Developmental Delayed – 17.5% • Other Health Impaired – 2.8%
The 1% Rule for AA-AAS Assessments • The number of alternate assessment students counted as Proficient for AYP may not exceed 1% of all students tested (i.e., regular and alternate tested students) • 1% rule calculations are made at the district and state level, not at the school level
Statewide Calculations for Participation in SC-Alt • The percent of all tested students (SC-Alt and PASS) who were tested with SC-Alt in 2012 was 0.9% • 81.6% of SC-Alt students taking ELA scored Proficient (Achievement Levels 3 and 4) • The percent of all students (SC-Alt and PASS) scoring Proficient on SC-Alt ELA was 0.8% (significantly below the NCLB Reg of <= 1.0%)
Variation in District SC-Alt Identification • The percent of all students tested with SC-Alt was less than 1% for most districts • The percent of all students (SC-Alt and PASS) scoring Proficient on SC-Alt ELA was less than 0.8% for most districts • 27% of districts had high rates of participation related to problematic identification decisions
Variation in District SC-Alt Identification • The number of districts exceeding the 1% Proficient rule was 22 in 2012 and 23 in 2011, and almost all were the same districts • One-half of the districts exceeding 1% had percent proficient rates of 1.5% or higher • The percent of all students tested with SC-Alt in these districts ranged from 1.2% to 3.8%
Districts Exceeding the 1% Rulewith the Highest RatesNumber Tested and Percent of All Grade 3-8 Students:
Contrasts in District SC-Alt IdentificationNumber Tested and Percent of All Grade 3-8 Students:
Participation Guidelines Guidance for IEP Teams on Determining Participation in theSouth Carolina Alternate Assessment (SC-Alt) Manual Features
Definition ofSignificant Cognitive Disability Characterized by ability scores on both verbal and non-verbal scales that are at least 2 ½ ─ 3 standard deviations below the mean Deficits in adaptive behavior skills (skills that enable people to function effectively in their everyday lives) are at least 2 ½─3 standard deviations below the mean in two out of three domains.
Manual Features Initial Determination of Eligibility Participation Worksheet Review of Eligibility Participation Worksheet Eligibility Criteria Descriptors Description of Sources of Evidence
Committee Discussion Guidance for IEP Teams on Determining Participation in theSouth Carolina Alternate Assessment (SC-Alt)