Project 4: Reading and writing development

1. SALSA Study into Adolescent Literacy of Students At-risk Project 4: Reading and writing development Assessing the reading and writing proficiency of at-risk adolescents in multilingual contexts Roel van Steensel, Ron Oostdam, Amos van Gelderen and Mirjam Trapman EARLI 2009, Amsterdam Roel van Steensel (SCO-Kohnstamm Instituut – UvA) http://salsa.socsci.uva.nl

2. Project SALSA • Large groups of students in secondary education lag behind in literacy skills OECD (2001) • Relatively little is known about the factors that influence at-risk students’ literacy development Curtis (2002) • Hence: Project SALSA Roel van Steensel, EARLI 2009

3. Project SALSA Environment Student Project 1 Literacy context in school De Milliano Project 3 Literacy-related skills and attitudes Trapman Project 4 Reading and writing development Project 2 Literacy context outside school Van Kruistum Roel van Steensel, EARLI 2009

4. Project SALSA • Central research question Which factors contribute to/impede the reading and writing development of at-risk adolescents in multilingual contexts? • Sample • Lower tracks of prevocational secondary education (vmbo): basic and middle-management program • Grades 7, 8 and 9 (age range: appr. 12-15) • 30% lowest scoring on CITO End of Primary School Test • Year 1 (2007/08): N=63 • 32 monolinguals, 31 bilinguals (based on student questionnaire) • International context Parallel projects in Canada en Switzerland Roel van Steensel, EARLI 2009

5. The test • Newly developed literacy test: • Reading: text comprehension • Writing: text production • This presentation: (construct) validity of the reading test Roel van Steensel, EARLI 2009

6. The test • 9 reading tasks • Text selection on the basis of: • Text type: narrative, expository, instructive, argumentative • Medium: school books, newspapers/magazines, internet, ‘official documents’ (house rules) • Subject: context-neutral, relevant for the target group Roel van Steensel, EARLI 2009

7. The test Roel van Steensel, EARLI 2009

8. The test • Item construction on the basis of hypothesized subskills: • Retrieving: retrieving relevant details from the text • Interpreting: making inferences from shorter passages in the text (e.g. cause-effect relationships) • Reflecting: reflecting on larger passages/the text as a whole Davis (1968); Rosenshine (1980); Goldman & Durán (1988); Alderson (2000); OECD (2003); Song (2008); Cerdán et al. (2009) Roel van Steensel, EARLI 2009

9. The test Local level/microstructure Retrieving Text comprehension Interpreting Global level/macrostructure Reflecting Roel van Steensel, EARLI 2009

10. Validity study • Messick (1989) - aspects of (construct) validity: • Construct (ir-)relevance: does the test only reflect aspects of the construct to be measured? Or are the test results influenced by variables irrelevant to the construct? • Internal structure: can (theoretical) expectations about relations between items be confirmed? • External structure: can (theoretical) expectations about relations with other measures be confirmed? Roel van Steensel, EARLI 2009

11. Validity study • Test administered to 200 students in the lower tracks of prevocational secondary education (school year 2007-08) • Additional tests administered to a subsample of 63 students Roel van Steensel, EARLI 2009

12. Construct (ir-)relevance • Assumption: difficulty of the tasks is determined by difficulty of the texts • Comparison between: • Text difficulty measured by Flesch/Douma readability index • Task difficulty indicated by mean task score as % of the maximum task score Roel van Steensel, EARLI 2009

13. Construct (ir-)relevance Roel van Steensel, EARLI 2009

14. Construct (ir-)relevance • INS1 en INS2 are both discontinuous texts: • INS1 is a set of house rules discontinuity makes the task easier • INS2 is a website discontinuity makes the task more difficult Roel van Steensel, EARLI 2009

15. Internal structure Local level/microstructure Retrieving Text comprehension Interpreting Global level/macrostructure Reflecting Roel van Steensel, EARLI 2009

16. Internal structure • Do the data show a: • 1-factor model? • 3-factor model? • 2-factor model? • Confirmatory factor analysis using Structural Equation Modeling (EQS 6.1) • Individual test items were combined into 6 parcels (RTR1, RTR2, INT1, INT2, RFL1, RFL2) Roel van Steensel, EARLI 2009

17. Internal structure RTR1 1-factor model RTR2 Text comprehension INT1 INT2 RFL1 Excellent fit! χ2=6.27, df=9, p=.713, N=175, SRMR=.026, RMSEA=.000, 90% confidence interval=.000-.060. RFL2 Roel van Steensel, EARLI 2009

18. Internal structure RTR1 3-factor model retrieving RTR2 Text comprehension INT1 interpreting INT2 RFL1 reflecting No improvement! Difference score 1-factor/3-factor model: χ2=3.36, df=3, p>.05. RFL2 Roel van Steensel, EARLI 2009

19. Internal structure RTR1 2-factor model RTR2 local level skills Text comprehension INT1 INT2 RFL1 global level skills No improvement! Difference score 1-factor/2-factormodel: χ2=2.41, df=1, p>.05. RFL2 Roel van Steensel, EARLI 2009

20. External structure • Assumption: text comprehension is facilitated by: • Well-developed decoding skills • A rich vocabulary • Well-developed morpho-syntactic skills • A good working memory • (Metacognitive) knowledge of reading and writing • And: text comprehension is associated with IQ Roel van Steensel, EARLI 2009

21. External structure ** p<.01 Roel van Steensel, EARLI 2009

22. Conclusions: support for validity? • Construct (ir-)relevance: •  Task difficulty largely corresponds with text difficulty • Exceptions (INS1, INS2) are explainable • Internal structure: •  No support for the presence of subskills •  Data do show that the test measures one construct • External structure: •  Substantial correlations with other skills • Exception (decoding skills) is explainable Roel van Steensel, EARLI 2009

23. Thank you for your attention! • Contact? • r.c.m.vansteensel@uva.nl • Or visit our website: http://salsa.socsci.uva.nl/ Roel van Steensel, EARLI 2009