1 / 28

Jack Mostow, Joe Beck, Juliet Bey, Andrew Cuneo, June Sison, Brian Tobin Project LISTEN

An Embedded Experiment to Evaluate the Effectiveness of Vocabulary Previews in an Automated Reading Tutor. Jack Mostow, Joe Beck, Juliet Bey, Andrew Cuneo, June Sison, Brian Tobin Project LISTEN Carnegie Mellon University www.cs.cmu.edu/~listen Funding: NSF IERI, Heinz Endowments

thomasbarbara
Télécharger la présentation

Jack Mostow, Joe Beck, Juliet Bey, Andrew Cuneo, June Sison, Brian Tobin Project LISTEN

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. An Embedded Experiment to Evaluate the Effectiveness of Vocabulary Previews in an Automated Reading Tutor Jack Mostow, Joe Beck, Juliet Bey, Andrew Cuneo, June Sison, Brian Tobin Project LISTEN Carnegie Mellon University www.cs.cmu.edu/~listen Funding: NSF IERI, Heinz Endowments The Reading Tutor helps children read stories aloud. Should it preview new vocabulary? Within-student experiment design Effects on vocabulary and comprehension

  2. Before story: randomly assign 4 new vocabulary words in story to different previews • Control: • Do not pretest or explain word meaning. • Test-only: • Pretest but do not explain word meaning. • Pretest word meaning, then teach a synonym: • Relate to a simpler word appropriate to the story. • Pretest word meaning, then teach a definition: • Give a short explanation appropriate to the story. • How does Reading Tutor pretest word meaning?… 2

  3. Before story: pretest word meaning 3

  4. Before story: teach a synonym 4

  5. Before story: teach a definition 5

  6. During story: assist reading(only with decoding, not word meaning) 6

  7. During story: test comprehension by automatically inserting cloze questions 7

  8. … then continue assisted reading 8

  9. After story: post-test vocabulary 9

  10. … by matching the 4 words to their definitions 10

  11. Effects on vocabulary: post-test results Percent correct on post-test • N = 5668 trials • 1417 story readings • 364 students in grades K-9 • 7 urban & suburban schools • “Untaught”: just encounter • control • test-only • “Taught”: preview meaning • synonym • definition 56% 54% 52% 50% 48% control test-only synonym definition (95% confidence intervals of means) 11

  12. Exclude words with hasty pretest responses • Correctness varies with speed • N = 4251 pretest responses • 10.8% are faster than 2 seconds • 3.3% are slower than 15 seconds • “Hasty” responses are guesses • At chance level (25%) • Exclude those words in subsequent analyses Percent correct on pretest 80% 60% 40% 20% 0% 0 5 10 15 Response time on pretest (in seconds) 12

  13. More taught than untaught words were learned • Split by whether already knew • “Knew” = correct on pretest • Excluding hasty pretests • Compare per-student % correct • N = 291 students • If didn’t already know word: • Taught: 17% over chance • Untaught: 10% over chance • Preview > encounter alone! • P = .031 • Who learned?… Percent correct on post-test 60% 50% 40% 30% N=277 N=242 N=291 N=250 No Yes 13

  14. Who learned taught words better than untaught? • Disaggregate by WRMT • N = 291 students • No difference: • For WC up to GE 3 • Taught > untaught words: • WC above GE 3 Percent correct on post-test 80% 60% 40% 20% 0% • 2 3 4 5 6 • Word Comprehension GE pretest score 14

  15. Did teaching target, distractor(s), or sentence word(s) help? N = 2671 cloze questions on vocabulary words Effects on comprehension: cloze results 15

  16. Who did better on cloze if target word was taught? • N = 152 students’ cloze performance with target word taught or not • Exclude cloze question if a distractor or sentence word was taught Percent correct on post-test 70% 60% 50% 40% 30% 20% 10% 0% 1 2 3 4 5 Word Comprehension GE pretest score 16

  17. Benefits to vocabulary and comprehension • Briefly explaining a new word before a story helped child • Match the vocabulary word to its definition after the story • Better than just encountering it in the story • For students with Word Comprehension above GE 3 • And apparently • Answer comprehension questions involving the word • Whether as cloze target, distractor, or context • For students with Word Comprehension above GE 1 (!) • Do previews really help comprehension before vocabulary? • Thank you! Questions? Suggestions? 17

  18. Project LISTEN’s Reading Tutor (video) • John Rubin (2002). The Sounds of Speech (Show 3). On Reading Rockets (Public Television series commissioned by U.S. Department of Education). Washington, DC: WETA. www.readingrockets.org. 18

  19. Project LISTEN’s Reading Tutor (video) 19

  20. Problem: support vocabulary learning • Efficacy varies by age and ability (NRP 2000). • What are the specific vocabulary instruction needs of students at different grade and ability levels? • Computers can facilitate vocabulary learning (NRP 2000). • What is the optimal use of computers in vocabulary instruction? • Our question: when does previewing new words help? • Which previews? • Which outcomes? • Which students? • Which words? 20

  21. Within-student experiment design • The Reading Tutor helps children read stories aloud. • Before story: pick 4 random new vocabulary words in story • Pretest word meaning: “Which word means (definition)?” • Teach 2 of the 4 words • During story: • Assist reading: scaffold decoding but not meaning • Assess comprehension: insert cloze questions • After story: • Post-test all 4 words: “Which word means (definition)?” 21

  22. Before story • show example -- simulate screen shot sequence 22

  23. Before story: ask familiarity with word 23

  24. During story: assist and assess reading • Assist: scaffold decoding but not word meaning • Assess: automatically insert cloze questions • Before a sentence containing a vocabulary word • Delete the word and use it as the target • Use 3 other vocabulary words as distractors • (not necessarily the preview words) 24

  25. Results • Effect of previews on post-test • Effect of previews on cloze performance • By age • By word frequency 25

  26. Source df F Sig ANOVA for % correct on post-test Corrected Model 6 3.6 .002 Intercept 1 .48 .491 WRMT WC GE 1 7.3 .007 Age 1 .61 .437 Word Length 1 1.1 .290 Gender 1 .41 .521 Treatment 1 4.7 .031 Gender * Treatment 1 .47 .493 • Significant predictors: • Previous vocabulary • Preview 26

  27. Previous work • Aist 2002: factoids helped rare, single-sense words (grades 2-3) on next-day test • Vocabulary learning literature • Reinking & Rickman 1990: computer vocabulary help for 6th graders • Beck: • McKeown guidelines for explaining words • Mostow, Tobin, Cuneo 2002: cloze validation see Aist thesis 27

  28. Who learned taught words better than untaught? 28

More Related