California English Language Development Test: Developing the Early Literacy Assessment Components

1. California English Language Development Test: Developing the Early Literacy Assessment Components Panel Presentation at the Tenth Annual Accountability Leadership Institute Burlingame, CA December 8, 2009

2. Section 1 –Introductions,Background, and Policy RequirementsLily Roberts, Administrator,California English Language Development Test (CELDT) Office K-1 Early Literacy Assessment

3. Presentation Overview • Background and policy requirements • Item development cycle • Guiding principles for test development and administration • Field test and lessons learned • Development and technical issues • Discussion • Questions and answers K-1 Early Literacy Assessment

4. Background • 2007: State legislature authorized development of the Kindergarten and grade one (K-1) early literacy assessment • 2008: Initial item development and expert panel was convened • 2009: Field test was conducted in March • 2009-10: Operational testing in all four domains for all grades (K-12) • 2010: Standard setting will be conducted to recommend performance levels K-1 Early Literacy Assessment

5. Policy Requirements • Title III, Elementary and Secondary Education Act requires testing in the domains of listening, speaking, reading, and writing in all grades • California Education Code (EC) Section 60810 defines the purposes for the CELDT: • Identify limited English proficient students • Determine the level of English language proficiency • Assess progress acquiring the skills of listening, speaking, reading, and writing K-1 Early Literacy Assessment

6. Reclassification Criteria • EC Section 313 specifies four criteria for local reclassification decisions, including: • Assessment of English language proficiency • Performance in basic skills • Teacher evaluation • Parental opinion and consultation K-1 Early Literacy Assessment

7. Section 2 -Initial Item DevelopmentEdynn Sato, WestEd K-1 Early Literacy Assessment

8. 2008 Item Development • Subject matter experts participated in guiding item development • Conducted two rounds of content and bias reviews • Developed 142 K-1 reading and writing test questions • Twelve reading and writing English Language Development (ELD) standards were identified as important and assessable for K-1 by subject matter experts K-1 Early Literacy Assessment

9. Item Development Cycle • Test specifications are created based on ELD standards • Guiding principles for item development are established • Item specifications are created to guide item development • Item writers and editors receive training • Items are written and edited • Graphics are created K-1 Early Literacy Assessment

10. Item Development Cycle (cont.) • Lead editors and content director review and sign-off on items that are ready for CELDT Office and committee reviews • Interdepartmental staff reviews items/graphics • Subject matter experts review items for content • Subject matter experts review items for bias • Necessary edits are made K-1 Early Literacy Assessment

11. Guidelines for Item Writing • General Guidelines • Typical representations • Reflect a range of cultural and ethnic diversity • Familiar to a range of students • Specific Guidelines • Specific, detailed, and clear • Contextual detail to enhance comprehensibility • People in graphics • Age, gender, ethnicity K-1 Early Literacy Assessment

12. Specific Questions that Need to be Asked during Content Review • Is the item aligned to the correct ELD Standard? • Does the item test a knowledge or skill that is appropriate for the assigned domain? • Is the item aligned to the correct grade span? • Does the item test for current and worthwhile concepts? • Does the item stem give a full sense of what is being asked? K-1 Early Literacy Assessment

13. Specific Questions (cont.) • Are the directions clear? • Does the stem avoid wordiness? • Is the key the only correct answer? • Does the item use distractors that relate to the stem in the same way as the key? • Does the item use distractors that are plausible misconceptions and errors? • Does the item contain clues to the correct answer? K-1 Early Literacy Assessment

14. Specific Questions (cont.) IF THE ITEM CONTAINS GRAPHICS… • Is the graphic required to answer the item? • Does the graphic provide sufficient information needed to answer the item? K-1 Early Literacy Assessment

15. Item Bias and Sensitivity Review • Ensure that the items avoid stereotype patterns • Ensure that the items avoid language, symbols, words, phrases, or examples that are biased or potentially offensive, inappropriate, or negative toward any group • Ensure that the items acknowledge the multicultural nature of our society • Ensure items do not unfairly advantage or disadvantage any group K-1 Early Literacy Assessment

16. Different Types of Bias • Bias can refer to age, gender, race, ethnicity, cultural background, social-economic background, religion, disability, or mental condition. • Bias in test items can be present as Content bias or Language bias. K-1 Early Literacy Assessment

17. Additional development activities: • Cognitive interviews of students • Interviews of teachers • Small-scale pilot of items K-1 Early Literacy Assessment

18. Section 3 -Field Test and Lessons Learned for Future CELDT EditionsCaroline Fahmy,Educational Data Systems, CELDT Test Contractor K-1 Early Literacy Assessment

19. Overview: Lessons Learned K-1 Early Literacy • Review of test and lessons learned related to: • Spring 2009 field test • 2009-10 Operational test • Modifications to be incorporated into the 2010-11 operational test K-1 Early Literacy Assessment

20. 2009 Field Test (FT) • 142 items were delivered to the new CELDT contractor in February • Conducted FT March 16-20, 2009 • Constructed four FT forms with ten reading and ten writing items each • Administered FT in 37 districts to 2,548 K-1 students in 57 schools across California K-1 Early Literacy Assessment

21. Number of Field Test Participants by Grade K-1 Early Literacy Assessment

22. 2009-10 Lessons Learned: Test Materials • Format student book to eliminate distractions and enhance ease of administration • Provide all specific, test question information in the student book for ease and speed of administration K-1 Early Literacy Assessment

23. Page Layout Changes Student Page ExaminerPage 1 2 2 1 23 K-1 Early Literacy Assessment

24. 2009-10 Lessons Learned:Test Examiner • Include examples on how to pronounce the letter sounds next to each item script • Allow examiner to repeat any question, but no prompting allowed • Provide allowable prompts in the sample practice items in the student test book K-1 Early Literacy Assessment

25. 2009-10 Lessons Learned:Student • Use formatting to minimize distractions • Use sample item administration as a time to really practice the items • Do not underestimate students’ English language ability K-1 Early Literacy Assessment

26. 2010-11 Lessons Learned: Test Materials • Student Book and Examiner’s Manuals will include all four domains • Examiner’s Manual will contain all instructions, questions, and pictures • Student Book will only contain: • Bubbles for responses • Areas for written responses K-1 Early Literacy Assessment

27. 2010-11 Lessons Learned: Test Materials (cont.) • Student Books will be greatly reduced in size • Scoring Guide will be combined with Examiner’s Manual (for all grades) K-1 Early Literacy Assessment

28. 2010-11 Lessons Learned: Test Examiner • Move instructions from Student Book to Examiner’s Manual • Refine the “Acceptable Responses Include” section (based on actual student responses) • Review the use of pictures in the writing domain K-1 Early Literacy Assessment

29. 2010-11 Lessons Learned: Student • Writing area - modify open box design to: • Help student know where to write • Help administrator and scorer better interpret upper case and lower case letters • Keep all writing in area clipped for image scoring • Use page formatting to accommodate left-handed students in copying words K-1 Early Literacy Assessment

30. Section 4 -Technical Considerationsand DiscussionRichard Duran, University of California, Santa BarbaraMember, CELDT Technical Advisory Group K-1 Early Literacy Assessment

31. K-1 Expert Panel and Technical Advisory Group • Input on development issues • Guiding principles • Alignment to ELD Standards • Input on technical issues • Pilot test • Field test • Operational test K-1 Early Literacy Assessment

32. Primary Development Issues • Guiding Principles and ELD Standards: Items must: • embody ELD Standards • be based on early literacy theory/research • be meaningful to EL K-1 students • easily administered • scored reliably • Other Concerns K-1 Early Literacy Assessment

33. Technical Issues • December 2008 pilot test on small K-1 sample of English learner and English-only students • p-values computed • Cognitive interviews conducted • Screened out some items and revised others • Procedures for administration reviewed • March 2009 K-1 field test of four forms - Ten reading and ten writing items per form - Reading items had already been calibrated on common reading scale - Computed basic test statistics, p-values, point biserial correlations K-1 Early Literacy Assessment

34. Technical Issues (cont.) • July 2009 implemented operational test and a linking study for K-1 writing items with a sample of grade two English learners • January 2010 will conduct a standard setting for K-1 reading and writing performance-level cut scores using a modified Bookmark process K-1 Early Literacy Assessment

35. Discussion • Achieved alignment of K-1 assessments with state and federal policy and professional testing standards • Contributed to state and national efforts to better ground English language development assessment and practice K-1 Early Literacy Assessment

36. For More Information Lily Roberts, Ph.D., Administrator CELDT Office Statewide Assessment Division Telephone: 916-319-0784 E-mail: CELDT@cde.ca.gov Web: www.cde.ca.gov/ta/tg/el K-1 Early Literacy Assessment