Kathy McCoy

1. Kathy McCoy Artificial Intelligence Natural Language Processing Applications for People with Disabilities

2. Primary Research Areas • Natural Language Generation – problem of choice. • Deep Generation --- structure and content of coherent text • Surface Generation – particularly using TAG (multi-lingual generation and machine translation) • Discourse Processing • Second Language Acquisition • Applications for people with disabilities affecting their ability to communicate

3. Projects • ICICLE – CALL system for teaching English as a second language to ASL natives (Chris Pennington, Rashida Davis, Mike Bloodgood, David Derman) • Augmentative Communication – Center for Applied Science and Engineering in Rehabilitation (ASEL) – Word Prediction and Contextual Information • Text Summarization (Greg Silber) • Multi-lingual SentenceGeneration (Raymond Kozlowski, Vijay Shanker) • Graph-to-Text(Sandee Carberry, Dan Chester, Stephanie Elzer, Nancy Green)

4. Developing Intelligent Communication Aids for People with Disabilities Kathleen F. McCoy Computer and Information Sciences & Center for Applied Science and Engineering in Rehabilitation University of Delaware

5. Augmentative Communication • Intervention that gives non-speaking person an alternative means to communicate User Population • May have severe motor impairments • Unable to speak • Unable to write • Cannot use sign language • May have cognitive impairments and/or developmental disabilities • May be too young to have developed literacy skills

6. Row-Column Scanning

7. Row-Column Scanning II

8. Language Representation: Words

9. Still Need to Spell!

10. Predicting Fringe Vocabulary • Word Prediction of Spelled Words (infrequent context-specific words) Methods • Statistical NLP Methods • Syntax/Semantic Filters • Other Contextual Clues • Geographic Location, Time of Day, Conversational Partner, Topic of Conversation

11. Language Representation: Phrases

12. Delivering Text in Context – What must Technology Facilitate? Context: Video Store want Grisham’s “A Time to Kill” Prestored Message: Many of Grisham’s books have been made into movies. Choice: • Edit message to be perfect • Let listener know and edit to be perfect • Deliver message quickly as is

13. Hypothesis: Perception Different Depending on Kind of Mismatch Grice Theory of Language Maxims • Quantity – amount of information provided • Quality – truth value and adequacy of message • Relation – relevance • Manner – way message is delivered

14. Experiments • Book Store Situation • Actors playing roles • AAC User wants to purchase books • Subjects are clerks, asked to put themselves in clerk’s position

15. Findings • Some maxims are more important than others • Need to develop technologies to support prestored text delivery obeying the important maxims • Current work: looking at repair strategies and how the repair process can be supported

16. Modelingthe Acquisition of English in the ICICLE System Kathleen F. McCoy Department of Computer and Information Sciences University of Delaware

17. People • Current People • Chris Pennington • Mike Bloodgood • Rashida Davis • David Durmond • Recent Graduate • Lisa Masterman Michaud • Others • Greg Silber, Meghan Boyle, Mohamed Mostagir, Stephanie Baker, Heejong Yi • Graduates: Matthew Huenerfauth, Jill Janofsky, Litza Stark, David Schneider

18. The ICICLE Project Interactive Computer Identification and Correction of Language Errors • Interactive writing tutor for native signers of American Sign Language (ASL) • Purpose: analyze student-written English texts and provide individualized feedback and instruction on grammar

19. The ICICLE Project • system provides student with tutorial instruction on the errors • student has opportunity to make corrections and request re-analysis • Cycle of user input, system response • student provides piece of text • system analyzes text for grammatical errors The ICICLE System

20. Current Implementation the system shows which sentences have errors the student enters text here explanations shown here

21. Writing From Deaf Students • Literacy is a serious issue for the Deaf population. • Lots of variation in level of acquisition. • Marked Differences from writing of hearing peers. • Dropped be: She really pretty. • Missing Possessives: She age is 13. • Subject/verb agreement, plural markers, determiners: She really like go with friend to mall.

22. Work on ICICLE • Previous work focused on developing grammar and mal-rules and modeling the user’s level of acquisition (so different analyses can be found depending on it) Current Work • Tutorial Responses • Probabilistic Parsing • NEED SYSTEM HELP!!!!!

23. What Mal-Rules do We Use? “She is teach piano on Tuesdays.” • Beginner: Over-application of auxiliary IS, missing simple present morphology: • She teaches piano on Tuesdays. • Intermediate: Botched progressive tense: • She is teaching piano on Tuesdays. • Advanced: Botched passive voice: • She is taught piano on Tuesdays.

24. Text Summarization Greg Silber

25. Lexical Chain Interpretation • What is important to include in a summary? • Focus on nouns • Coherent text will repeat noun concepts • A particular noun concept may be referred to with different lexical items. E.g., computer, machine, Sun • Developed a linear time algorithm for determining interpretation of words that allow most “coherent text” • Those concepts repeated most often belong in a summary

26. Current Status • Algorithm can pick out important noun concepts • Current Question: What do you say about those nouns???? • Looking at picking out predicates that link important noun concepts in a text • Generating a coherent summary