What Impacts Alternate Assessment Scores

1. What Impacts Alternate Assessment Scores Presented by Diane M. Browder, PhD UNC Charlotte Ginevra Courtade-Little Charlotte Mecklenburg Schools

2. Authors: Diane M. Browder Meagan Karvonen Stephanie Davis University of North Carolina at Charlotte Kathy Fallin Ginevra Courtade-Little Charlotte Mecklenburg Schools

3. Background • IDEA 1997, NCLB • Alternate assessment process may be used to improve educational programs (Browder, Spooner, Algozzine, et. al., 2003) • Limited research that focuses on the relationship between educational programs and alternate assessment outcomes • Teachers of students with significant disabilities may feel pressure to improve alternate assessment scores without knowing how to do so • The need exists to develop training that relates directly to the requirements of AA and to evaluate this training in the context of real school programs

4. Purpose • To determine if training teachers would improve students’ outcome scores on alternate assessments

5. Hypothesis • Training teachers on three instructional variables that influence outcomes (curriculum access, data collection, and instructional effectiveness) would increase the alternate assessment scores • A manual was developed and used for teacher training that summarized current research on: • how to select skills for students with significant disabilities • how to develop data collection systems • how to improve instruction if students do not make adequate progress

6. Research Questions • Does training in curriculum, data collection, and data-based decisions increase alternate assessment scores? • Is there evidence the teachers used the instructional skills trained? • Is there evidence the students learned new skills? • How did the teachers and parents perceive gains made?

7. Method-Setting • Charlotte-Mecklenburg Schools • Large urban consolidated city-county district • served 112,500 students (grades pK-12) • 75% of those students were in grades that required testing • 294 students participated in the NCAAP (.3% of the assessed population)

8. Teachers 27 initially nominated by a school system administrator 93% female 81% Caucasian, 19% African American Experience (0-24ys, M=10yrs) 5 lateral entry teachers Pool was comparable to district as a whole Students Initially 29 selected by teacher and parent nominations 24% female 60% Caucasian, 34% African American, 3% Hispanic, 3% Asian 21% with severe/profound disabilities, 36% with autism, 39% with moderate mental disabilities, 4% with multiple disabilities Method

9. Method-Intervention • North Carolina Alternate Assessment Portfolio • Portfolio of evidence collected to demonstrate mastery and generalization of 5 teacher selected IEP goals related to state standard • Instructional implications addressed in training • Selecting appropriate objectives consistent with state standards • Creating data sheets that met state requirements for baseline data and ongoing student performance data • Developing instruction that would promote generalization and mastery of the objectives by the turn in deadline

10. Method-Intervention • Instructional Components Training • Curriculum-Guidelines for reviewing IEP to determine if it provided access to the curricular domains required by the state • Data collection-how to develop data collection systems (models provided) • Guidelines for making data based decisions

11. Method-Intervention • Method of teacher training • Teacher training manual • Five group inservice days • Introduction of project • One day devoted to each instructional component • Evaluation of the project • 3 site visits by project staff to answer questions and model data based decision making process

12. Method-Instrumentation • Primary DV-score on the NCAAP assigned by NCDPI • Student scores obtained for year prior to intervention and intervention year • Scores obtained for comparison group of students in CMS and state • To obtain a proficient score • Below mastery during baseline • At criteria for mastery by the end of the year • Maintained criteria for most of the last days of the school year • Generalized across people and settings • Initiated (unprompted correct or used skill in response to natural cues)

13. Method-Instrumentation • Additional variables measured • NCAAP portfolio quality score • Teacher use of instructional components • Data collected on curriculum, data sheets, data based decisions • Teacher survey to identify resources used • Behavioral data on IEP goals • Reliability of teacher data • Percentage of growth on skills to determine if criteria was set at an appropriate level • Stakeholder Perception Surveys • To determine if teachers and parents considered gains made on portfolio objectives important

14. Method-Design • Quasi-experimental pretest-posttest design • Pretest-scores from year prior to the intervention • Post-test scores from intervention year • Scores from the students in the school system whose teachers did not participate in the project were used as comparison group to apply a pretest-posttest control group design

15. Results-Impact of Teacher Training on NCAAP scores

16. Results-Impact of Teacher Training on NCAAP scores 2002

17. Results Did teacher training positively influence the state alternate assessment scores?

18. Results-Teachers’ Use of Instructional Components • 92% of teachers mastered the curriculum component • 96% mastered the data sheets component • 84% mastered the data based decisions component • All teachers reported using the training manual and data collection sheets

19. Results-Portfolio quality score • Students with teachers in the model • 2001-67% adequate or superior • 2002-96% adequate or superior • Comparison group • 2001-42% adequate or superior • 2002-71% adequate or superior

20. Results-Meaningful Growth on IEP objectives • Median average growth 68.7% • Range from 16%-171%

21. Results-Teacher and Parent Perceptions of Gains • 84% of teacher respondents reported students made more progress as a result of being in the project • 80% of teacher respondents thought that their students had better IEPs because of the project • Parents had overall strong, positive impressions of their students’ participating in the project

22. Discussion • Provides evidence that AA scores can be improved by training teachers in instructional variables • Students in the project had significantly higher scores than the previous year • High rates of proficiency were supported by behavioral data

23. Generalizability of Findings • Generalizable to a wide range of students with severe disabilities due to teacher training overriding the influence of student characteristics • Generalizable to any AA system with a primary focus on student acquisition of skills

24. Limitation of findings • Use of quasiexperimental design with all students in the Charlotte region not in the project serving as the control group • Possible selection bias • Instrumentation used to document student progress and performance (changes in way portfolios were scored from year to year)

25. Recommendations for Future Research • Teacher training to meet changing curricular demands (assessment of students on specific LA and math standards) • Teacher buy in to AA process • Validation that AA outcomes reflect educationally significant student learning

26. Recommendations for Practice • Use of collected data to make instructional decisions • Training in : • Data collection • How to enhance student gains • How to teach and assess language arts, math, and science