Linguistically Rich Statistical Models of Language

1. Linguistically Rich Statistical Models of Language Joseph Smarr M.S. Candidate Symbolic Systems Program Advisor: Christopher D. Manning December 5th, 2002

2. Grand Vision • Talk to your computer like another human • HAL, Star Trek, etc. • Ask your computer a question, it finds the answer • “Who’s speaking at this week’s SymSys Forum?” • Computer can read and summarize text for you • “What’s the cutting edge in NLP these days?”

3. We’re Not There (Yet) • Turns out behaving intelligently is difficult • What does it take to achieve the grand vision? • General Artificial Intelligence problems • Knowledge representation, common sense reasoning, etc. • Language-specific problems • Complexity, ambiguity, and flexibility of language • Always underestimated because language is so easy for us!

4. Are There Useful Sub-Goals? • Grand vision is still too hard, but we can solve simpler problems that are still valuable • Filter news for stories about new tech gadgets • Take the SSP talk email and add it to my calendar • Dial my cell phone by speaking my friend’s name • Automatically reply to customer service e-mails • Find out which episode of The Simpsons is tonight • Two approaches to understanding language: • Theory-driven: Theoretical Linguistics • Task-driven: Natural Language Processing

5. Theoretical Linguistics vs. NLP Theoretical Linguistics • Goal: • Understand people’s Knowledge of language • Method: • Rich logical representations of language’s hidden structure and meaning • Guiding principles: • Separation of (hidden) knowledge of language and (observable) performance • Grammaticality is categorical (all or none) • Describe what are possible and impossible utterances Natural Language Processing • Goal: • Develop practical tools for analyzing speech / text • Method: • Simple, robust models of everyday language use that are sufficient to perform tasks • Guiding principles • Exploit (empirical) regularities and patterns in examples of language in text collections • Sentence “goodness” is gradient (better or worse) • Deal with the utterances you’re given, good or bad

6. Theoretical Linguistics vs. NLP Linguistics NLP

7. Linguistic Puzzle • When dropping an argument, why do some verbs keep the subject and some keep the object? • John sang the song John sang • John broke the vaseThe vase broke • Not just “quirkiness of language” • Similar patterns show up in other languages • Seems to involve deep aspects of verb meaning • Rules to account for this phenomenon • Two classes of verbs (unergative & unaccusative) • Remaining argument must be realized as subject

8. Exception: Imperatives • “Open the pod bay doors, Hal” • Different goals lead to study of different problems. In NLP... • Need to recognize this as a command • Need to figure out what specific action to take • Irrelevant how you’d say it in French • Describing language vs. working with language • But both tasks clearly share many sub-problems

9. Theoretical Linguistics vs. NLP • Potential for much synergy between linguistics and NLP • However, historically they have remained quite distinct • Chomsky (founder of generative grammar): • “It must be recognized that the notion ‘probability of a sentence’ is an entirely useless one, under any known interpretation of this term.” • Karttunen (founder of finite state technologies at Xerox) • Linguists’ reaction to NLP: “Not interested. You do not understand Theory. Go away you geek.” • Jelinek (former head of IBM speech project): • “Every time I fire a linguist, the performance of our speech recognition system goes up.”

10. Potential Synergies • Lexical acquisition (unknown words) • Statistically infer new lexical entries from context • Modeling “naturalness” and “conventionality” • Use corpus data to weight constructions • Dealing with ungrammatical utterances • Find “most similar / most likely” correction • Richer patterns for finding information in text • Use argument structure / semantic dependencies • More powerful models for speech recognition • Progressively build parse tree while listening

11. Finding Information in Text • US Government has sponsored lots of research in “information extraction” from news articles • Find mentions of terrorists and which locations they’re targeting • Find which companies are being acquired by which others and for how much • Progress driven by simplifying the models used • Early work used rich linguistic parsers • Unable to robustly handle natural text • Modern work is mainly finite state patterns • Regular expressions are very practical and successful

12. Our Price:$##.## Web Information Extraction • How much does that text book cost on Amazon? • Learn patterns for finding relevant fields

13. Example of Tagged Sentence: Uba2pis located largelyinthe nucleus. NP_SEG VP_SEG PP_SEG NP_SEG Improving IE Performance on Natural Text Documents • How can we scale IE back up for natural text? • Need to look elsewhere for regularities to exploit • Idea: Consider grammatical structure • Run shallow parser on each sentence • Flatten output into sequence of “typed chunks”

14. Power of Linguistic Features 21% increase 65% increase 45% increase

15. Linguistically Rich(er) IE • Exploit more grammatical structure for patterns • e.g. Tim Grow’s work on IE with PCFGs S{pur, acq, amt} VP{acq, amt} VP{acq, amt} NP{pur} MD NNP NNP NNP PP{amt} will VB {pur} {pur} {pur} NP{acq} NP{amt} acquire IN First Union Corp NNP NNP NNP CD CD NNP for {acq} {acq} {acq} {amt} {amt} {amt} Sheland Bank Inc three million dollars

16. Cotrimoxazole Wethersfield Alien Fury: Countdown to Invasion Classifying Unknown Words • Which of the following is the name of a city? • Most linguistic grammars assume a fixed lexicon • How do humans learn to deal with new words? • Context (“I spent a summer living in Wethersfield”) • Makeup of the word itself (“phonesthetics”) • Idea: Learn distinguishing letter sequences

17. oxa : field What’s in a Name?

18. Generative Model of PNPs Length n-gram model and word model P(pnp|c) = Pn-gram(word-lengths(pnp)) *Pword ipnp P(wi|word-length(wi)) Word model: mixture of character n-gram model and common word model P(wi|len) = llen*Pn-gram(wi|len)k/len + (1-llen)* Pword(wi|len) N-Gram Models: deleted interpolation P0-gram(symbol|history) = uniform-distribution Pn-gram(s|h) = lC(h)Pempirical(s|h) + (1- lC(h))P(n-1)-gram(s|h)

19. pairwise 1-all n-way Experimental Results

20. Knowledge of Frequencies • Linguistics traditionally assumes Knowledge of Language doesn’t involve counting • Letter frequencies are clearly an important source of knowledge for unknown words • Similarly, we saw before that there are regular patterns to exploit in grammatical information Take home point: • Combining Statistical NLP methods with richer linguistic representations is a big win!

21. Language is Ambiguous! • Ban on Nude Dancing on Governor’s Desk – from a Georgia newspaper column discussing current legislation • Lebanese chief limits access to private parts – talking about an Army General’s initiative • Death may ease tension – an article about the death of Colonel Jean-Claude Paul in Haiti • Iraqi Head Seeks Arms • Juvenile Court to Try Shooting Defendant • Teacher Strikes Idle Kids • Stolen Painting Found By Tree

22. Language is Ambiguous! • Local HS Dropouts Cut in Half • Obesity Study Looks for Larger Test Group • British Left Waffles on Falkland Islands • Red Tape Holds Up New Bridges • Man Struck by Lightning Faces Battery Charge • Clinton Wins on Budget, but More Lies Ahead • Hospitals Are Sued by 7 Foot Doctors • Kids Make Nutritious Snacks

23. Coping With Ambiguity • Categorical grammars like HPSG provide many possible analyses for sentences • 455 parses for “List the sales of the products produced in 1973 with the products produced in 1972.” (Martin et al, 1987) • In most cases, only one interpretation is intended • Initial solution was hand-coded preferences among rules • Hard to manage as number of rules increase • Need to capture interactions among rules

24. Statistical HPSG Parse Selection • HPSG provides deep analyses of sentence structure and meaning • Useful for NLP tasks like question answering • Need to solve disambiguation problem to make using these richer representations practical • Idea: Learn statistical preferences among constructions from hand-disambiguated collection of sentences • Result: Correct analysis chosen >80% of the time StatNLP methods + Linguistic representation = Win

25. Towards Semantic Extraction • HPSG provides representation of meaning • Who did what to whom? • Computers need meaning to do inference • Can we extend information extraction methods to extract meaning representations from pages? • Current project: IE for the semantic web • Large project to build rich ontologies to describe the content of web pages for intelligent agents • Use IE to extract new instances of concepts from web pages (as opposed to manual labeling) • student(Joseph), univ(Stanford), at(Joseph, Stanford)

26. Towards the Grand Vision? • Collaboration between Theoretical Linguistics and NLP is important step forward • Practical tools with sophisticated language power • How can we ever teach computers enough about language and the world? • Hawking: Moore’s Law is sufficient • Moravec: mobile robots must learn like children • Kurzweil: reverse-engineer the human brain • The experts agree: Symbolic Systems is the future!

27. Upcoming Convergence Courses • Ling 139M Machine Translation Win • Ling 239E Grammar Engineering Win • CS 276B Text Information Retrieval Win • Ling 239A Parsing and Generation Spr • CS 224N Natural Language Processing Spr Get Involved!!