Improving Comprehension Online Project, 2005-08

1. Improving Comprehension Online Project, 2005-08 Designing and testing a universally designed strategic digital literacy environment for diverse learners Bridget Dalton, Vanderbilt University Patrick Proctor, Boston College IES Research Conference Washington, DC ~ June 11, 2008 A goal 2 development award to CAST, Inc.

2. Bridget Dalton (Co-PI), Vanderbilt University,andElaine Mo, Kristin Robinson, Ge Vue, Mary O’Malley, & Boris Goldowski, CAST, Inc. Patrick Proctor (Co-PI), Yi-Chien Li, & Kevin O’Connor, Boston College Catherine Snow (Co-PI), Paola Uccelli, Sabina Neugebauer, Lorena Landeo Schenone, Harvard Graduate School of Education School partners: 3 semi-urban and 1 urban school in northeastern Massachusetts Research team

3. To develop and test a universally designed (Rose & Meyer, 2002) strategic digital reading approach (Dalton & Proctor, 2007) to improving reading achievement of 5th grade students, including bilingual students and struggling readers Project goal

4. Multiple perspectives required

5. Universal design for learning (Rose & Meyer, 2002) Design for the broadest range of learners from the beginning; avoid retrofitting Provide multiple means of • Representation • Expression • Engagement

6. Assumptions • Shift to universal design perspective • Attention to diversity and individual difference benefits individual and society • New literacies, while more complex, are more flexible and inclusive • potential to level the playing field for those who have not fared well with print literacy

7. Rand Reading Study Group’s (2002) reading comprehension heuristic reader reader text text activity activity comprehension comprehension Socioculturalcontext Socioculturalcontext

8. Strategic Digital Reading (Dalton & Proctor, 2007) reader text activity comprehension Socioculturalcontext

9. Comprehension in a new literacies landscape: Strategic digital reading reader text activity comprehension Socioculturalcontext

10. How does ICON support diverse learners in relation to… • Representation? • Expression? • Engagement? • What is unique for ELLs? • What features/supports are essential for some; good for many/all?

11. Iterative design, formative feedback and testing Y2. Compare Vocabulary, Strategies & Combo Y2. Compare Vocabulary, Strategies & Combo Versions Y1. Develop Vocabulary Y1. Develop Vocabulary Y3. Compare Combo Vs. Control Y3. Compare Combo Vs. Control

12. ICON optimal prototype (Yr. 3)

13. Embedded Strategies

14. Spanish language support

15. Coaches Level 1 coaches provide text-specific models and think alouds. As skill increases, students select strategies and coaches provide generic think-alouds.

16. Vocabulary: Connect It!

17. Vocabulary: Language Alert More than 60% of the power words are Spanish English cognates

18. Vocabulary: Web It!

19. Vocabulary: Caption It!

20. All 3 years/studies: Feasibility, appeal & usability • Teachers and students view ICON as a helpful reading tool, easy to use, & engaging • Technical support required; bandwidth issue • Variation in teacher enactment of ICON suggests need for additional study • English proficiency levels influence ways in which students use ICON and extent to which additional support is needed • Peer collaboration one means of support • Increased sensitivity to learner (needs, use of system, performance) is likely to benefit all

21. Y1 Study of Semantic Depth • 35 students, 24 bilinguals (Spanish and other low-incidence languages), 11 English monolinguals • Oral language skills (WJ picture vocab+listening comprehension) • Reading skills (WJ passage comp + MCAS ELA score) • Average semantic depth score for 8 target words (Anxiously, Bitter, Dense, Grasp, Ignore, Menacing, Powerless, Relieved)

22. Yr. 1 study of semantic depth(Proctor, Uccelli, Dalton, & Snow, in press) Effective teaching and learning activities targeted for further analysis: Caption-It

23. Semantic depth was a significant predictor of reading performance, mediated by interaction with English oral language proficiency *No effect for language status (bilingual vs. monolingual)

24. What did we learn from Y1 vocabulary study? • Caption It: Encouraging but preliminary – As both an activity and an assessment, it appeared to reduce the monolingual/bilingual gap in students’ performance, though oral language was heavily implicated • Semantic Depth: Promising but far from final • Semantic depth showed a positive association with reading comprehension, beyond the contribution of decoding and oral language skills • Semantic depth seems to play a more prominent role as oral language skills improve • Semantic depth refers to a cluster of skills: associations among these and with other dimensions of vocabulary depth need to be explored.

25. Year 2 study: Strategies vs. vocabulary vs. combo • Very hard to find research that compares effects of vocabulary versus comprehension instruction • Likely because the two are so strongly intertwined • Given that vocabulary is a primary focus of instruction for ELL students, we found this question intriguing, and asked 2 basic questions: • Does assignment to condition (Vocabulary-only, Strategy-only, Combo) affect students’ performance on standardized and researcher-developed measures of vocabulary and comprehension? • Do the effects vary by language status (monolingual, Spanish-English bilingual, other bilingual)?

26. Our hypotheses For standardized and researcher-developed vocabulary: Combo > Vocabulary > Strategy For standardized and researcher-developed comprehension: Combo > Strategy > Vocabulary

27. Y2 study of vocab vs strategies: What matters and for whom?(Dalton, Proctor, Uccelli, Mo, & Snow, in preparation) • 106 students, 21 Spanish-English bilinguals, 17 other-English bilinguals, 68 monolinguals in 6 classrooms, 3 districts • Random assignment to condition (vocabulary, strategy, combination) • 14-week intervention • Pre-post standardized testing, embedded vocabulary and comprehension quizzes

28. Y2 effect size overview

29. What did we learn from Y2 study of vocabulary vs. reading strategies vs. combined? • Overall, hypotheses held, and combination version showed strongest results across standardized and researcher measures • Did the small sample size mask learner by treatment interactions? • Theoretically, an interaction between student characteristics (reader type and/or language status) would make sense • Sample size may be too small • Student controls access to support and may not be making good decisions about when and how to use support • Thresholds of language proficiency

30. Year 3 study: Combo vs control Having established general effectiveness, time to move to a comparison between treatment and control using optimal version of ICON • Quasi-experimental study • 12 classrooms, classrooms assigned randomly to treatment or control condition, n = 227 (108 control, 119 intervention; 10.5% other bilinguals, 48.5% Spanish-English bilinguals, 41% English monolinguals) • For intervention group, 2 x 50 minutes per week, for 16 weeks • For control group, across the three districts, standard literacy curriculum included reading strategies focus, but limited vocabulary instruction • Initial training of teachers and students by research team, gradual release of ICON prototype teaching responsibility • Analyses conducted at student level, randomization at teacher level

31. Y3 measures • Gates-MacGinitie reading vocabulary and comprehension subtests pre- and post-intervention • Aprenda reading vocabulary - pre • Researcher-designed breadth of vocabulary (targeted words) post-intervention • 20-item multiple choice assessment • Reseacher-designed depth of vocabulary, post-intervention • 5-item definition, drawing, & captioning assessment

32. Y3 results General results: • No effect of condition on standardized measures; significant voc and comp gain for both groups • Strong effect of condition on researcher developed measures Condition Depth/Breadth Standard Vocab and comp

33. Condition effects on ICON vocabulary breadth Significant effect of condition on ICON voc. Breadth F(1,205) = 56.62, p < .001 Significant difference between Spanish bilinguals and English monolinguals (t = 5.1, p < .001) Strong readers significantly outperform average (t = 5.1, p < .001) and struggling (t = 12.6, p < .001) No interactions by language or reader status and condition

34. Condition effects on ICON vocabulary depth Exp. significantly outperform Control on vocabulary depth F(1,224) = 101.4, p < .001 English monolinguals significantly outperform Spanish bilinguals (t = 5.3, p < .001) and non-Spanish bilinguals (t = 2.2, p < .05) Strong readers significantly outperform average (t = 5.0, p < .001) and struggling (t = 9.4, p < .001) readers No interactions by language status, BUT average-reader X condition interaction (p = .048)

35. Spanish-English bilinguals and ICON depth of vocabulary For intervention Spanish-English bilinguals, Spanish Vocabulary scores explained English Vocabulary Depth performance, after controlling for condition and prior English proficiency.

36. Conclusions and next steps • Continue to analyze Year 3 data • Worklog responses, multimodal retellings, student feature use, teacher use of feedback support. • For whom does this intervention work best? • Goal 3: effects for Spanish-English bilinguals and struggling readers are intriguing • Goal 2: Work for transfer. Design for increased sensitivity to learner characteristics, especially language proficiency. • Distal effects on standardized measures