Assessing Academic Literacy: The role of text in comprehending written language

1. Assessing Academic Literacy: The role of text in comprehending written language Barbara Foorman, Ph.D. Florida Center for Reading Research Florida State University

2. What are the Issues? • Academic literacy assumes grade-level proficiency. • On the 2007 Reading NAEP, 33% below basic in G4; 26% below basic in G8. • For minorities, the % below basic on the 2007 Reading NAEP are: 53% in G4 & 45% in G8 for Blacks; 50% in G4 and 42% in G8 for Hispanics. • NCLB requires that students at-risk for reading disability receive intervention.

3. Goals for This Presentation • Explain relation of academic literacy to academic language • Definitions of reading comprehension • Characteristics of text difficulty • Measuring text difficulty • Assessing academic literacy

4. Word Meanings Text Academic Language is at the Core of Literacy Instruction • because it allows literate people to discuss literary products; previously referred to as extended discourse or decontextualized language. • because contextual cues and shared assumptions are minimized by explicitly encoding referents for pronouns, actions, and locations

5. 13 higher- SES children (professional) 23 middle/lower- SES children (working class) Cumulative Vocabulary words 6 welfare children Age of child in months Hart & Risley, 1995

6. Language Experience Professional Working-class Estimated cumulative words addressed to child Welfare Age of child in months Hart & Risley, 1995

7. Quality Teacher Talk(Snow et al., 2007) • Rare words • Ability to listen to children and to extend their comments • Tendency to engage children in cognitively challenging talk • Promotes emergent literacy & vocabulary & literacy success in middle grades

8. Home & Schoolexperiences: ages 3-6 Skills developed: ages 3-6 School performance Literacy Understanding literacy Kindergarten and first grade reading Print Print focus Conversation Conversational language Instruction and Practice in reading Extended discourse forms and nonfamiliar audiences Reading comprehension In Grade 4 Decontextualized language (Snow, 1991)

9. (Cunningham & Stanovich, 1998, adapted from Anderson, Wilson, & Fielding,1988)

10. Is Literacy Enough?(Snow et al., 2007) For adolescents, oral language and literacy skills need to be adequate, but also need: • Caring adult(s) at home • Caring adults at school who provide guidance about how to meet goals (often need smaller school) • Minimal risk: Not many school transitions; minimal family disturbances.

11. What is Reading Comprehension? • “the process of simultaneously extracting and constructing meaning through interaction and involvement with written language” (RAND, 2002, p. 11) • “Reading is an active and complex process that involves • Understanding written text • Developing and interpreting meaning; and • Using meaning as appropriate to type of text, purpose, and situation” (NAEP Framework, 2009)

12. Word recognition, vocabulary, background knowledge, strategy use, inference-making abilities, motivation Text structure, vocabulary, genre discourse, motivating features, print style and font Sociocultural TEXT READER ACTIVITY Purpose, social relations, school/classroom/peers/ families Environment, cultural norms Context A heuristic for thinking about reading comprehension (Sweet & Snow, 2003).

13. Understanding what has been read; the application to written text of: (a) nonlinguistic (conceptual) knowledge (b) general language comprehension skills (Rayner, Foorman, Perfetti, Pesetsky, & Seidenberg, 2001)

14. The Reading Pillar (NRC, 1998) Skilled Reading Fluency Speed and ease of reading with comprehension Conceptual Knowledge/vocabulary Strategic processing of text Comprehension Word Recognition Decoding using alphabetic principle Decoding using other cues Sight Recognition Print Awareness & Letter Knowledge Motivation to Read Oral Language including Phonological Awareness Emergent Reading

15. What Makes a Text Difficult?

16. Components of Reading Comprehension(Perfetti, 1999) Comprehension Processes General Knowledge Situation Model Linguistic System Phonology Syntax Morphology Text Representation Inferences Parser Meaning and Form Selection Lexicon Meaning Morphology Syntax Word Representation Identification Word Orthography Mapping to phonology Orthographic Units Phonological Units Visual Input

17. Vocabulary Demands in 6 G1 Basals (Foorman et al., 2004)

19. Some “rare” (G1 Basal) and “not-so-rare” (elementary literature) Words

20. Representation of Opportunity Words Across Basals

21. Opportunity Words in Grade 1 Basals

22. Conclusions on Vocabulary • Publishers need to provide teachers with cumulative vocabulary lists • These need to be made available electronically to textbook adopters and should include information on: • Frequency in text and lesson number • Separate entry for each definition used • Derivational forms • Printed word frequency in other relevant corpora

23. Conclusions on Vocabulary • Instruction needs to target oral language development from pre-school through high school • Printed word frequency and age of acquisition are useful tools for guiding selection of lexical entries to be taught • Assessment of vocabulary for the purpose of Reading First should focus on the link between assessment and instruction

24. Summary and Conclusions • Programs differ substantially in the composition of their print materials for Grade 1 students • Length of texts, grammatical complexity, numbers of unique and total words, repetition of words, coverage of important vocabulary • Differences exist in the decodability of types and tokens • Generally there is greater decodability for tokens than types, • most programs show improvements for types later in the year

25. Summary and Conclusions Programs vary in the approach they take to achieve decodability and in the degree to which materials can be expected to yield accuracy in reading. - Vary in phonic elements taught - Vary in opportunity to practice words containing these elements - Within 6-week blocks, 70% of words are singletons in 4 of the 6 basals - Vary in reliance on holistically-taught words

26. Implications for fluency • “…for dysfluent readers, the texts that are read and reread for fluency practice need to have sufficiently high percentages of words within…the word zone fluency curriculum and low percentages of rare words, especialy multisyllabic ones” ( p. 18) • “Repetition of core words makes science text ideal for fluency practice in the primary grades” (p. 11) Hiebert (2007)

27. Word Zone Fluency Curriculum

28. Jabberwocky(Lewis Carroll, 1872) ‘Twas brillig, and the slithy toves Did gyre and gimble in the wabe: All mimsy were the borogoves, And the mome raths outgrabe. “Beware the Jabberwock, my son! The jaws that bite, the claws that catch! Beware the Jubjub bird, and shun The frumious Bandersnatch!” And four more stanzas From Through the Looking-Glass and What Alice Found There

29. Discussion You know how to pronounce the words in Jabberwocky; some are real English words. • Which ones are real English words? • What is the distinction between those that are actual English words and those that aren’t? • Do the two paragraphs differ in these distinctions?

30. Alice’s reaction “It seems very pretty,” she said when she had finished it, but it’s rather hard to understand!” (You see she didn’t like to confess, even to herself, that she couldn’t make it out at all.) “Somehow it seems to fill my head with ideas—only I don’t exactly know what they are! However, somebody killed something: that’s clear at any rate—”

31. NAEP 2009 Reading Framework Characteristics of text difficulty: • Vocabulary reported out separately • Subscales for literary & informational text • Grade-level standards for text type

36. What Does Mean to be Proficient? • W score cutpoints on NAEP and state tests communicate grade-level proficiency or benchmark performance. • State curriculum standards need to be aligned with benchmarks/proficiency levels. • Are states’ proficiency levels comparable to NAEP’s?

37. % Proficient on State vs NAEP Reading 2005 [Porter, 2007]

38. Most state testing systems do not assess college and work readiness • 26 states require students to pass an exam before they graduate high school.* • Yet most states have testing systems that do not measure college and work readiness.** *Source: Center on Education Policy, State High School Exit Exams: States Try Harder, But Gaps Persist, August 2005. **Source: Achieve Survey/Research, 2006.

39. Graduation exams in 26 states establish the performance “floor” Figure reads: Alaska has a mandatory exit exam in 2005 and is withholding diplomas from students based on exam performance. Arizona is phasing in a mandatory exit exam and plans to begin withholding diplomas based on this exam in 2006. Connecticut does not have an exit exam, nor is it scheduled to implement one. Source: Center on Education Policy, based on information collected from state departments of education, July 2005.

40. How challenging are state exit exams? • Achieve conducted a study of graduation exams in six states to determine how high a bar the tests set for students. • The results show that these tests tend to measure only 8th, 9th or 10th grade content, rather than the skills students needs to succeed in college and the workplace.

41. The tests Achieve analyzed Source: Achieve, Inc., Do Graduation Tests Measure Up? A Closer Look at State High School Exit Exams, 2004.

42. Students can pass state English tests with skills ACT expects of 8th & 9th graders ACT (11th/12th) ACT PLAN (10th) ACT EXPLORE (8th/9th) FL MD MA NJ OH TX Source: Achieve, Inc., Do Graduation Tests Measure Up? A Closer Look at State High School Exit Exams, 2004.

43. % Students Proficient on FCAT(Level 3 and above)

44. Is 10th Grade FCAT Too Hard? • The St. Petersburg Times article (4/15/07) concluded correctly that the 10th Grade FCAT is harder than the 10th grade NRT. • Conclusion based on fact that Level 3 (proficient) performance is 56th %ile nationally at Gr 7; 80th %ile at Gr 10 • Or “Why wait until high school to implement world class standards?”

45. Absolute level of reading proficiency nationally 10 Grade level standard on the FCAT 9 8 7 Absolute level of reading proficiency 6 5 4 3 3 4 5 6 7 8 9 10

46. Passage Length in Words

47. % of Passage Types

48. FCAT Test Design • Cognitive Complexity (Webb’s Depth of Knowledge) • Content Categories for Reading - Words & phrases in context - Main idea, plot, & author’s purpose - Comparison; cause/effect - Reference & Research – locate, organize, interpret, synthesize, & evaluate information

49. To Make Proficiency Standards Meaningful and Fair • Agree on target for proficiency (e.g., college readiness) • Align elementary, middle, and high school targets • Align curriculum standards • Evaluate dimensionality of tests and prepare instruction accordingly • Equate state tests with NAEP to guarantee comparability and equity

50. From Barbara Tuckman’s The Zimmerman Telegram… The first message of the morning watch plopped out of the pneumatic tube into a wire basket with no more premonitory rattle than usual. The duty officer at the British Navel Intelligence twisted open the cartridge and examined the German wireless intercept it contained without noting anything of unusual significance. When a glance showed him that the message was in non-navel code, he sent it in to the Political Section in the inner room and thought no more about it. The date was January 17, 1917, past the halfway mark of a war that had already ground through thirty months of reckless carnage and no gain.