cs60057 speech natural language processing n.
Skip this Video
Loading SlideShow in 5 Seconds..
CS60057 Speech &Natural Language Processing PowerPoint Presentation
Download Presentation
CS60057 Speech &Natural Language Processing

play fullscreen
1 / 159

CS60057 Speech &Natural Language Processing

315 Views Download Presentation
Download Presentation

CS60057 Speech &Natural Language Processing

- - - - - - - - - - - - - - - - - - - - - - - - - - - E N D - - - - - - - - - - - - - - - - - - - - - - - - - - -
Presentation Transcript

  1. CS60057Speech &Natural Language Processing Autumn 2007 Lecture 8 9 August 2007 Natural Language Processing

  2. POS Tagging • Task: • assign the right part-of-speech tag, e.g. noun, verb, conjunction, to a word in context • POS taggers • need to be fast in order to process large corpora • should take no more than time linear in the size of the corpora • full parsing is slow • e.g. context-free grammar  n3, n length of the sentence • POS taggers try to assign correct tag without actually parsing the sentence Natural Language Processing

  3. POS Tagging • Components: • Dictionary of words • Exhaustive list of closed class items • Examples: • the, a, an: determiner • from, to, of, by: preposition • and, or: coordination conjunction • Large set of open class (e.g. noun, verbs, adjectives) items with frequency information Natural Language Processing

  4. POS Tagging • Components: • Mechanism to assign tags • Context-free: by frequency • Context: bigram, trigram, HMM, hand-coded rules • Example: • Det Noun/*Verb the walk… • Mechanism to handle unknown words (extra-dictionary) • Capitalization • Morphology: -ed, -tion Natural Language Processing

  5. POS Tagging • Words often have more than one POS: back • The back door = JJ • On my back = NN • Win the voters back = RB • Promised to back the bill = VB • The POS tagging problem is to determine the POS tag for a particular instance of a word. These examples from Dekang Lin Natural Language Processing

  6. How hard is POS tagging? Measuring ambiguity Natural Language Processing

  7. Algorithms for POS Tagging • Ambiguity – In the Brown corpus, 11.5% of the word types are ambiguous (using 87 tags): • Worse, 40% of the tokens are ambiguous. Natural Language Processing

  8. Problem Setup • There are M types of POS tags • Tag set: {t1,..,tM}. • The word vocabulary size is V • Vocabulary set: {w1,..,wV}. • We have a word sequence of length n: W = w1,w2…wn • Want to find the best sequence of POS tags: T = t1,t2…tn Natural Language Processing

  9. Information sources for tagging All techniques are based on the same observations… • some tag sequences are more probable than others • ART+ADJ+N is more probable than ART+ADJ+VB • Lexical information: knowing the word to be tagged gives a lot of information about the correct tag • “table”: {noun, verb} but not a {adj, prep,…} • “rose”: {noun, adj, verb} but not {prep, ...} Natural Language Processing

  10. Algorithms for POS Tagging • Why can’t we just look them up in a dictionary? • Words that aren’t in the dictionary http://story.news.yahoo.com/news?tmpl=story&cid=578&ncid=578&e=1&u=/nm/20030922/ts_nm/iraq_usa_dc • One idea: P(ti| wi) = the probability that a random hapax legomenon in the corpus has tag ti. • Nouns are more likely than verbs, which are more likely than pronouns. • Another idea: use morphology. Natural Language Processing

  11. Algorithms for POS Tagging - Knowledge • Dictionary • Morphological rules, e.g., • _____-tion • _____-ly • capitalization • N-gram frequencies • to _____ • DET _____ N • But what about rare words, e.g, smelt (two verb forms, melt and past tense of smell, and one noun form, a small fish) • Combining these • V _____-ing I was gracking vs. Gracking is fun. Natural Language Processing

  12. POS Tagging - Approaches • Approaches • Rule-based tagging • (ENGTWOL) • Stochastic (=Probabilistic) tagging • HMM (Hidden Markov Model) tagging • Transformation-based tagging • Brill tagger • Do we return one best answer or several answers and let later steps decide? • How does the requisite knowledge get entered? Natural Language Processing

  13. 3 methods for POS tagging 1. Rule-based tagging • Example: Karlsson (1995) EngCGtagger based on the Constraint Grammar architecture and ENGTWOL lexicon • Basic Idea: • Assign all possible tags to words (morphological analyzer used) • Remove wrong tags according to set of constraint rules (typically more than 1000 hand-written constraint rules, but may be machine-learned) Natural Language Processing

  14. Sample rules N-IP rule: A tag N (noun) cannot be followed by a tag IP (interrogative pronoun) ... man who … • man: {N} • who: {RP, IP} --> {RP} relative pronoun ART-V rule: A tag ART (article) cannot be followed by a tag V (verb) ...the book… • the: {ART} • book: {N, V} --> {N} Natural Language Processing

  15. After The First Stage • Example: He had a book. • After the fırst stage: • he he/pronoun • had have/verbpast have/auxliarypast • a a/article • book book/noun book/verb Tagging Rule Rule-1: if (the previous tag is an article) then eliminate all verb tags Rule-2: if (the next tag is verb) then eliminate all verb tags Natural Language Processing

  16. Rule-Based POS Tagging • ENGTWOL tagger (now ENGCG-2) • http://www.lingsoft.fi/cgi-bin/engcg Natural Language Processing

  17. 3 methods for POS tagging 2. Transformation-based tagging • Example: Brill (1995) tagger - combination of rule-based and stochastic (probabilistic) tagging methodologies • Basic Idea: • Start with a tagged corpus + dictionary (with most frequent tags) • Set the most probable tag for each word as a start value • Change tags according to rules of type “if word-1 is a determiner and word is a verb then change the tag to noun” in a specific order (like rule-based taggers) • machine learning is used—the rules are automatically induced from a previously tagged training corpus (like stochastic approach) Natural Language Processing

  18. An example 1. Assign to words their most likely tag • P(NN|race) = .98 • P(VB|race) = .02 2. Change some tags by applying transformation rules Natural Language Processing

  19. Types of context • lots of latitude… • can be: • tag-triggered transformation • The preceding/following word is tagged this way • The word two before/after is tagged this way • ... • word- triggered transformation • The preceding/following word this word • … • morphology- triggered transformation • The preceding/following word finishes with an s • … • a combination of the above • The preceding word is tagged this ways AND the following word is this word Natural Language Processing

  20. Learning the transformation rules • Input: A corpus with each word: • correctly tagged (for reference) • tagged with its most frequent tag (C0) • Output: A bag of transformation rules • Algorithm: • Instantiates a small set of hand-written templates (generic rules) by comparing the reference corpus to C0 • Change tag a to tag b when… • The preceding/following word is tagged z • The word two before/after is tagged z • One of the 2 preceding/following words is tagged z • One of the 2 preceding words is z • … Natural Language Processing

  21. Learning the transformation rules (con't) • Run the initial tagger and compile types of errors • <incorrect tag, desired tag, # of occurrences> • For each error type, instantiate all templates to generate candidate transformations • Apply each candidate transformation to the corpus and count the number of corrections and errors that it produces • Save the transformation that yields the greatest improvement • Stop when no transformation can reduce the error rate by a predetermined threshold Natural Language Processing

  22. Example • if the initial tagger mistags 159 words as verbs instead of nouns • create the error triple: <verb, noun, 159> • Suppose template #3 is instantiated as the rule: • Change the tag from <verb> to <noun> if one of the two preceding words is tagged as a determiner. • When this template is applied to the corpus: • it corrects 98 of the 159 errors • but it also creates 18 new errors • Error reduction is 98-18=80 Natural Language Processing

  23. Learning the best transformations • input: • a corpus with each word: • correctly tagged (for reference) • tagged with its most frequent tag (C0) • a bag of unordered transformation rules • output: • an ordering of the best transformation rules Natural Language Processing

  24. Learning the best transformations (con’t) let: • E(Ck) = nb of words incorrectly tagged in the corpus at iteration k • v(C) = the corpus obtained after applying rule v on the corpus C ε = minimum number of errors desired for k:= 0 step 1 do bt := argmint (E(t(Ck))// find the transformation t thatminimizes // the error rate if ((E(Ck) - E(bt(Ck))) < ε)// if bt does not improve the taggingsignificantly then goto finished Ck+1 := bt(Ck)// apply rule bt to the current corpus Tk+1 := bt// bt will be kept as the currenttransformationrule end finished: the sequence T1 T2 … Tk is the ordered transformation rules Natural Language Processing

  25. Strengths of transformation-based tagging • exploits a wider range of lexical and syntactic regularities • can look at a wider context • condition the tags on preceding/next words not just preceding tags. • can use more context than bigram or trigram. • transformation rules are easier to understand than matrices of probabilities Natural Language Processing

  26. How TBL Rules are Applied • Before the rules are applied the tagger labels every word with its most likely tag. • We get these most likely tags from a tagged corpus. • Example: • He is expected to race tomorrow • he/PRN is/VBZ expected/VBN to/TO race/NN tomorrow/NN • After selecting most-likely tags, we apply transformation rules. • Change NN to VB when the previous tag is TO • This rule converts race/NN into race/VB • This may not work for every case • ….. According to race Natural Language Processing

  27. How TBL Rules are Learned • We will assume that we have a tagged corpus. • Brill’s TBL algorithm has three major steps. • Tag the corpus with the most likely tag for each (unigram model) • Choose a transformation that deterministically replaces an existing tag with a new tag such that the resulting tagged training corpus has the lowest error rate out of all transformations. • Apply the transformation to the training corpus. • These steps are repeated until a stopping criterion is reached. • The result (which will be our tagger) will be: • First tags using most-likely tags • Then apply the learned transformations Natural Language Processing

  28. Transformations • A transformation is selected from a small set of templates. Change tag a to tag b when - The preceding (following) word is tagged z. - The word two before (after) is tagged z. - One of two preceding (following) words is tagged z. - One of three preceding (following) words is tagged z. - The preceding word is tagged z and the following word is tagged w. - The preceding (following) word is tagged z and the word two before (after) is tagged w. Natural Language Processing

  29. 3 methods for POS tagging • Stochastic (=Probabilistic) tagging • Assume that a word’s tag only depends on the previous tags (not following ones) • Use a training set (manually tagged corpus) to: • learn the regularities of tag sequences • learn the possible tags for a word • model this info through a language model (n-gram) • Example: HMM (Hidden Markov Model) tagging - a training corpus used to compute the probability (frequency) of a given word having a given POS tag in a given context Natural Language Processing

  30. Topics • Probability • Conditional Probability • Independence • Bayes Rule • HMM tagging • Markov Chains • Hidden Markov Models Natural Language Processing

  31. 6. Introduction to Probability • Experiment (trial) • Repeatable procedure with well-defined possible outcomes • Sample Space (S) • the set of all possible outcomes • finite or infinite • Example • coin toss experiment • possible outcomes: S = {heads, tails} • Example • die toss experiment • possible outcomes: S = {1,2,3,4,5,6} Natural Language Processing

  32. Introduction to Probability • Definition of sample space depends on what we are asking • Sample Space (S): the set of all possible outcomes • Example • die toss experiment for whether the number is even or odd • possible outcomes: {even,odd} • not {1,2,3,4,5,6} Natural Language Processing

  33. More definitions • Events • an event is any subset of outcomes from the sample space • Example • die toss experiment • let A represent the event such that the outcome of the die toss experiment is divisible by 3 • A = {3,6} • A is a subset of the sample space S= {1,2,3,4,5,6} Natural Language Processing

  34. Introduction to Probability • Some definitions • Events • an event is a subset of sample space • simple and compound events • Example • deck of cards draw experiment • suppose sample space S = {heart,spade,club,diamond} (four suits) • let A represent the event of drawing a heart • let B represent the event of drawing a red card • A = {heart} (simple event) • B = {heart} u {diamond} = {heart,diamond} (compound event) • a compound event can be expressed as a set union of simple events • Example • alternative sample space S = set of 52 cards • A and B would both be compound events Natural Language Processing

  35. Introduction to Probability • Some definitions • Counting • suppose an operation oi can be performed in ni ways, • a set of k operations o1o2...ok can be performed in n1 n2 ...  nk ways • Example • dice toss experiment, 6 possible outcomes • two dice are thrown at the same time • number of sample points in sample space = 6  6 = 36 Natural Language Processing

  36. Definition of Probability • The probability law assigns to an event a nonnegative number • Called P(A) • Also called the probability A • That encodes our knowledge or belief about the collective likelihood of all the elements of A • Probability law must satisfy certain properties Natural Language Processing

  37. Probability Axioms • Nonnegativity • P(A) >= 0, for every event A • Additivity • If A and B are two disjoint events, then the probability of their union satisfies: • P(A U B) = P(A) + P(B) • Normalization • The probability of the entire sample space S is equal to 1, i.e. P(S) = 1. Natural Language Processing

  38. An example • An experiment involving a single coin toss • There are two possible outcomes, H and T • Sample space S is {H,T} • If coin is fair, should assign equal probabilities to 2 outcomes • Since they have to sum to 1 • P({H}) = 0.5 • P({T}) = 0.5 • P({H,T}) = P({H})+P({T}) = 1.0 Natural Language Processing

  39. Another example • Experiment involving 3 coin tosses • Outcome is a 3-long string of H or T • S ={HHH,HHT,HTH,HTT,THH,THT,TTH,TTT} • Assume each outcome is equiprobable • “Uniform distribution” • What is probability of the event that exactly 2 heads occur? • A = {HHT,HTH,THH} 3 events/outcomes • P(A) = P({HHT})+P({HTH})+P({THH}) additivity - union of the probability of the individual events • = 1/8 + 1/8 + 1/8 total8 events/outcomes • = 3/8 Natural Language Processing

  40. Probability definitions • In summary: Probability of drawing a spade from 52 well-shuffled playing cards: Natural Language Processing

  41. Moving toward language • What’s the probability of drawing a 2 from a deck of 52 cards with four 2s? • What’s the probability of a random word (from a random dictionary page) being a verb? Natural Language Processing

  42. Probability and part of speech tags • What’s the probability of a random word (from a random dictionary page) being a verb? • How to compute each of these • All words = just count all the words in the dictionary • # of ways to get a verb: # of words which are verbs! • If a dictionary has 50,000 entries, and 10,000 are verbs…. P(V) is 10000/50000 = 1/5 = .20 Natural Language Processing

  43. Conditional Probability • A way to reason about the outcome of an experiment based on partial information • In a word guessing game the first letter for the word is a “t”. What is the likelihood that the second letter is an “h”? • How likely is it that a person has a disease given that a medical test was negative? • A spot shows up on a radar screen. How likely is it that it corresponds to an aircraft? Natural Language Processing

  44. More precisely • Given an experiment, a corresponding sample space S, and a probability law • Suppose we know that the outcome is some event B • We want to quantify the likelihood that the outcome also belongs to some other event A • We need a new probability law that gives us the conditional probability of A given B • P(A|B) Natural Language Processing

  45. An intuition • Let’s say A is “it’s raining”. • Let’s say P(A) in Kharagpur is 0.2 • Let’s say B is “it was sunny ten minutes ago” • P(A|B) means “what is the probability of it raining now if it was sunny 10 minutes ago” • P(A|B) is probably way less than P(A) • Perhaps P(A|B) is .0001 • Intuition: The knowledge about B should change our estimate of the probability of A. Natural Language Processing

  46. Conditional Probability • let A and B be events in the sample space • P(A|B) = the conditional probability of event A occurring given some fixed event B occurring • definition: P(A|B) = P(A  B) / P(B) Natural Language Processing

  47. A A,B B Conditional probability • P(A|B) = P(A  B)/P(B) • Or Note: P(A,B)=P(A|B) · P(B) Also: P(A,B) = P(B,A) Natural Language Processing

  48. Independence • What is P(A,B) if A and B are independent? • P(A,B)=P(A) ·P(B) iff A,B independent. P(heads,tails) = P(heads) · P(tails) = .5 · .5 = .25 Note: P(A|B)=P(A) iff A,B independent Also: P(B|A)=P(B) iff A,B independent Natural Language Processing

  49. Bayes Theorem • Idea: The probability of an A conditional on another event B is generally different from the probability of B conditional on A. There is a definite relationship between the two. Natural Language Processing

  50. Deriving Bayes Rule The probability of event A given event B is Natural Language Processing