Research on and With Novel Technologies for Comprehension:

1. Research on and With Novel Technologies for Comprehension: Tom Landauer*+ Peter Foltz*#Walter Kintsch+, Simon Dennis+, Bill Oliver+ + Dept. of Psychology & Institute of Cognitive Science University of Colorado* Pearson Knowledge Technologies, LLC # New Mexico State University landauer@psych.colorado.edu pfoltz@k-a-t.com IERI Reading Comprehension 11/12/04

2. Primary Objectives • Develop and evaluate novel techniques for improving vocabulary assessment and reading comprehension • Apply computational models of language simulating human understanding of words and text • Two approaches using same underlying technology • Vocabulary skills with a new vocabulary assessment instrument • Summarization skills with automated content-based feedback. • Validate and Use results to examine differences in technology effectiveness at different school grades and settings • Note: still early on in the testing. IERI Reading Comprehension 11/12/04

3. Vocabulary is essential to literary skills Requires Understanding of words as they fit into sentences. --How word meanings create sentence meanings. --How sentence meanings constrain word meanings. • Cloze tests are popular and effective approach for vocabulary and comprehension assessment and practice. • But…they rely on recognition process rather than recall Mom said to stop arguing and be __ . [friends trees flakes heal] How do we capitalize on the constructive nature of language? IERI Reading Comprehension 11/12/04

4. The open cloze test Eric rocked the __ gently A constructed response form would assess and practice precise knowledge of the possible meanings of a word (and of all the words around it and of the sentence formed of them) more authentically and effectively(and be a good test of writing and comprehension as well.) Focus on the active and constructive nature of language comprehension. More amenable to using with automated tutorial feedback than multiple choice. . IERI Reading Comprehension 11/12/04

5. How do we know if a student has chosen an appropriate word? Eric rocked the __ gently. dog, cradle, boat, house, crib, chair, … Goal: automatically model the contribution of meaning of each potential word in the sentence. IERI Reading Comprehension 11/12/04

6. The model • Combines • 1) a new Bayesian form of word “n-gram model that uses word-before as well as word-after information • 38 Million sentences • ~500 million words • 2) Latent Semantic Analysis (LSA) • Semantic model based on 75mb of running text • 3) Human judgements in calibration. IERI Reading Comprehension 11/12/04

7. Calibrating Open-Cloze items • From a large pool of correct (professionally authored, edited, and grammar and spelling check clean) sentences, select ones that are: • Between 8 and 10 words long • Have Lexile ratings appropriate to test-taker population • Remove from each, a word at least 3 from either end • Pretest with >= 20 representative test-takers • Two experts (e.g. teachers) rate the quality of each response on a 6 point scale • Apply ngram + LSA+proportion model to predict ratings IERI Reading Comprehension 11/12/04

8. The traditional-Cloze test The traditional Cloze test Our model scored 98+% correct on a sample of 368 professionally constructed items for which part-of-speech would not give the answer IERI Reading Comprehension 11/12/04

9. Current Performance of Open-Cloze • 2900 Responses over 300+ items • Human-human correlation of raters on appropriateness of words in sentences • Grad students raters 0.59 • Undergrad raters 0.39 • Correlation of model overall ratings to mean human rating 0.53 • Estimated correlation of summed-over twenty item score >= 0.84 For aggregate score over a class, grade, school, obviously extremely good. IERI Reading Comprehension 11/12/04

10. Administering Open-Cloze items • Select the 10% of the calibrated items that best predicted the mean ratings • Administer at least 20 such items per student • Score each answer using model: if the predicted rating is < x (where x is a parameter of the model that keeps FA errors below y%), ask student to try again, up to to two or three more times • For total individual assessment score compute percentage of all attempts (including multiple tries) that are predicted to be >= x.by model • Normalize to 6 point scale by your favorite method, curve, priors… • Report to student instantly and summed or meaned, create database entry and grade-book entries IERI Reading Comprehension 11/12/04

11. Current directions • Integrate into tutorial system. • Missing word cloze passages automatically generated from curriculum sources • Make students try again if word is below threshold • Tutorial feedback on appropriateness of words, suggest other words • Extend to Colorado elementary, middle and high schools • Test with standardized test items, other comprehension measures, grades, teacher ratings as part of Colorado Literacy Project IERI Reading Comprehension 11/12/04

12. IERI Reading Comprehension 11/12/04

13. Summary Street • Gets kids to practice reading-to-learn and writing what they learned • Automated content feedback on their written summaries • Also grammar, spelling, plagiarism, redundancy • In use in Colorado Literacy Tutor, funded by NSF, DoEd, NIH with 1000s of kids, 34 schools: elementary to high school, >80,000 summaries submitted • May improve reading comprehension on state exams (tested on Colorado CSAP items) IERI Reading Comprehension 11/12/04

14. IERI Reading Comprehension 11/12/04

15. IERI Reading Comprehension 11/12/04

16. IERI Reading Comprehension 11/12/04

17. IERI Reading Comprehension 11/12/04

18. IERI Reading Comprehension 11/12/04

19. IERI Reading Comprehension 11/12/04

20. Summary Street (red)Produces Better Essays as Judged by Teachers:2-Week Trial: 6th Grade Students IERI Reading Comprehension 11/12/04

21. Conclusions • Computational approach to simulating human understanding can be used for literacy assessment, vocabulary instruction, and comprehension training • Can assess free text responses • Essays, summaries, cloze tests, … • Applicability to young children • Open cloze: Yes • Summary Street: 3rd grade + IERI Reading Comprehension 11/12/04

22. Future directions • Development and testing of integrated learning environment to improve reading comprehension, writing skills and ability to learn from text. • Automated vocabulary training using free response tests with feedback and training • Assessment and feedback on student summaries of curriculum material • Other methods to adaptively choose the next set of words/topics/curriculum material to maximize domain coverage and knowledge • Validate and use results to examine differences in technology effectiveness at different school grades and settings IERI Reading Comprehension 11/12/04

23. Questions pfoltz@k-a-t.com landauer@psych.colorado.edu IERI Reading Comprehension 11/12/04