LIN3022 Natural Language Processing Lecture 9

1. LIN3022 Natural Language ProcessingLecture 9 Albert Gatt

2. In this lecture • We continue with our discussion of parsing algorithms • We introduce dynamic programming approaches • We then look at • probabilistic context-free grammars • Statistical parsers

3. Part 1 Dynamic programming approaches

4. Top-down vs bottom-up search Top-down Bottom-up Never considers derivations that do not end up at root S. Wastes a lot of time with trees that are inconsistent with the input. Generates many subtrees that will never lead to an S. Only considers trees that cover some part of the input. NB: With both top-down and bottom-up approaches, we view parsing as a search problem.

5. Beyond top-down and bottom-up • One of the problems we identified with top-down and bottom-up search is that they are wasteful. • These algorithms proceed by searching through all possible alternatives at every stage of processing. • Wherever there is local ambiguity, these possibly alternatives multiply. • There is lots of repeated work. • Both S  NP VP and S  VP involve a VP • The VP rule is therefore applied twice! • Ideally, we want to break up the parsing problem into sub-problems and avoid doing all this extra work.

6. Extra effort in top-down parsing Input: a flight from Indianapolis to Houston. NP  Det Nominal rule. (Dead end) NP  Det Nominal + Nominal  Nominal PP + Nominal  Nominal PP NP  Det Nominal PP + Nominal  Noun PP (Dead end)

7. Dynamic programming • In essence, dynamic programming involves solving a task by breaking it up into smaller sub-tasks. • In general, this is carried out by: • Breaking up a problem into sub-problems. • Creating a table which will contain solutions to each sub-problem. • Resolving each sub-problem and populating the table. • “Reading off” the complete solution from the table, by combining the solutions to the sub-problems.

8. Dynamic programming for parsing • Suppose we need to parse: • Book that flight. • We can split the parsing problem into sub-problems as follows: • Store sub-trees for each constituent in the table. • This means we only parse each part of the input once. • In case of ambiguity, we can store multiple possible sub-trees for each piece of input.

9. Part 2 The CKY Algorithm and Chomsky Normal Form

10. CKY parsing • Classic, bottom-up dynamic programming algorithm (Cocke-Kasami-Younger). • Requires an input grammar based on Chomsky Normal Form • A CNF grammar is a Context-Free Grammar in which: • Every rule LHS is a non-terminal • Every rule RHS consists of either a single terminal or two non-terminals. • Examples: • A  BC • NP  Nominal PP • A  a • Noun  man • But not: • NP  the Nominal • S VP

11. Chomsky Normal Form • Any CFG can be re-written in CNF, without any loss of expressiveness. • That is, for any CFG, there is a corresponding CNF grammar which accepts exactly the same set of strings as the original CFG.

12. Converting a CFG to CNF • To convert a CFG to CNF, we need to deal with three issues: • Rules that mix terminals and non-terminals on the RHS • E.g. NP  the Nominal • Rules with a single non-terminal on the RHS (called unit productions) • E.g. NP  Nominal • Rules which have more than two items on the RHS • E.g. NP  Det Noun PP

13. Converting a CFG to CNF • Rules that mix terminals and non-terminals on the RHS • E.g. NP  the Nominal • Solution: • Introduce a dummy non-terminal to cover the original terminal • E.g. Det  the • Re-write the original rule: • NP  Det Nominal • Det  the

14. Converting a CFG to CNF • Rules with a single non-terminal on the RHS (called unit productions) • E.g. NP  Nominal • Solution: • Find all rules that have the form Nominal  ... • Nominal  Noun PP • Nominal  Det Noun • Re-write the above rule several times to eliminate the intermediate non-terminal: • NP  Noun PP • NP  Det Noun • Note that this makes our grammar “flatter”

15. Converting a CFG to CNF • Rules which have more than two items on the RHS • E.g. NP  Det Noun PP • Solution: • Introduce new non-terminals to spread the sequence on the RHS over more than 1 rule. • Nominal  Noun PP • NP  Det Nominal

16. The outcome • If we parse a sentence with a CNF grammar, we know that: • Every phrase-level non-terminal (above the part of speech level) will have exactly 2 daughters. • NP  Det N • Every part-of-speech level non-terminal will have exactly 1 daughter, and that daughter is a terminal: • N  lady

17. Part 3 Recognising strings with CKY

18. Recognising strings with CKY Example input: The flight includes a meal. • The CKY algorithm proceeds by: • Splitting the input into words and indexing each position. (0) the (1) flight (2) includes (3) a (4) meal (5) • Setting up a table. For a sentence of length n, we need (n+1) rows and (n+1) columns. • Traversing the input sentence left-to-right • Use the table to store constituents and their span.

19. The table Rule: Det the [0,1] for “the” the flight includes a meal

20. The table Rule1: Det the Rule 2: N  flight [0,1] for “the” [1,2] for “flight” the flight includes a meal

21. The table [0,2] for “the flight” Rule1: Det the Rule 2: N  flight Rule 3: NP  Det N [1,2] for “flight” [0,1] for “the” the flight includes a meal

22. A CNF CFG for CYK (!!) • S  NP VP • NP  Det N • VP  V NP • V  includes • Det  the • Det a • N  meal • N  flight

23. CYK algorithm: two components • Lexical step: for j from 1 to length(string) do: let w be the word in position j find all rules ending in w of the form X  w put X in table[j-1,1] • Syntactic step: fori = j-2 to 0 do: for k = i+1 to j-1 do: for each rule of the form A B Cdo: if B isin table[i,k] & C is in table[k,j] then add A to table[i,j]

24. CKY algorithm: two components • We actually interleave the lexical and syntactic steps: for j from 1 to length(string) do: let w be the word in position j find all rules ending in w of the form X  w put X in table[j-1,1] fori = j-2 to 0 do: for k = i+1 to j-1 do: for each rule of the form A B Cdo: if B isin table[i,k] & C is in table[k,j] then add A to table[i,j]

25. CKY: lexical step (j = 1) • Lexical lookup • Matches Det the • The flight includes a meal.

26. CKY: lexical step (j = 2) • Lexical lookup • Matches N  flight • The flight includes a meal.

27. CKY: syntactic step (j = 2) • Syntactic lookup: • look backwards and see if there is any rule that will cover what we’ve done so far. • The flight includes a meal.

28. CKY: lexical step (j = 3) • Lexical lookup • Matches V  includes • The flight includes a meal.

29. CKY: lexical step (j = 3) • Syntactic lookup • There are no rules in our grammar that will cover Det, NP, V • The flight includes a meal.

30. CKY: lexical step (j = 4) • Lexical lookup • Matches Det a • The flight includes a meal.

31. CKY: lexical step (j = 5) • Lexical lookup • Matches N  meal • The flight includes a meal.

32. CKY: syntactic step (j = 5) • Syntactic lookup • We find that we have NP  Det N • The flight includes a meal.

33. CKY: syntactic step (j = 5) • Syntactic lookup • We find that we have VP  V NP • The flight includes a meal.

34. CKY: syntactic step (j = 5) • Syntactic lookup • We find that we have S  NP VP • The flight includes a meal.

35. From recognition to parsing • The procedure so far will recognise a string as a legal sentence in English. • But we’d like to get a parse tree back! • Solution: • We can work our way back through the table and collect all the partial solutions into one parse tree. • Cells will need to be augmented with “backpointers”, i.e. With a pointer to the cells that the current cell covers.

36. From recognition to parsing

37. From recognition to parsing NB: This algorithm always fills the top “triangle” of the table!

38. What about ambiguity? • The algorithm does not assume that there is only one parse tree for a sentence. • (Our simple grammar did not admit of any ambiguity, but this isn’t realistic of course). • There is nothing to stop it returning several parse trees. • If there are multiple local solutions, then more than one non-terminal will be stored in a cell of the table.

39. Part 4 Probabilistic Context Free Grammars

40. CFG definition (reminder) • A CFG is a 4-tuple: (N,Σ,P,S): • N = a set of non-terminal symbols (e.g. NP, VP) • Σ= a set of terminals (e.g. words) • N and Σ are disjoint (no element of N is also an element of Σ) • P = a set of productions of the form Aβwhere: • A is a non-terminal (a member of N) • βis any string of terminals and non-terminals • S = a designated start symbol (usually, “sentence”)

41. S  NP VP S  Aux NP VP NP  Det Nom NP  Proper-Noun Det  that | the | a … CFG Example

42. Probabilistic CFGs • A CFG where each production has an associated probability • PCFG is a 5-tuple: (N,Σ,P,S, D): • D isa function assigning each rule in P a probability • usually, probabilities are obtained from a corpus • most widely used corpus is the Penn Treebank

43. Example tree

44. S NP VP VBZ NNP NNP NP Mr Vinken is NP PP NN NN IN chairman of NNP Elsevier Building a tree: rules • S  NP VP • NP  NNP NNP • NNP  Mr • NNP Vinken • …

45. Characteristics of PCFGs • In a PCFG, the probability P(Aβ) expresses the likelihood that the non-terminal A will expand as β. • e.g. the likelihood that S  NP VP • (as opposed to SVP, or S  NP VP PP, or… ) • can be interpreted as a conditional probability: • probability of the expansion, given the LHS non-terminal • P(Aβ) = P(Aβ|A) • Therefore, for any non-terminal A, probabilities of every rule of the form A  β must sum to 1 • in this case, we say the PCFG is consistent

46. Uses of probabilities in parsing • Disambiguation: given n legal parses of a string, which is the most likely? • e.g. PP-attachment ambiguity can be resolved this way • Speed: we’ve defined parsing as a search problem • search through space of possible applicable derivations • search space can be pruned by focusing on the most likely sub-parses of a parse • parser can be used as a model to determine the probability of a sentence, given a parse • typical use in speech recognition, where input utterance can be “heard” as several possible sentences

47. Using PCFG probabilities • PCFG assigns a probability to every parse-tree t of a string W • e.g. every possible parse (derivation) of a sentence recognised by the grammar • Notation: • G = a PCFG • s = a sentence • t = a particular tree under our grammar • t consists of several nodes n • each node is generated by applying some rule r

48. Probability of a tree vs. a sentence • We work out the probability of a parse tree t by multiplying the probability of every rule (node) that gives rise to t (i.e. the derivation of t). • Note that: • A tree can have multiple derivations • (different sequences of rule applications could give rise to the same tree) • But the probability of the tree remains the same • (it’s the same probabilities being multiplied) • We usually speak as if a tree has only one derivation, called the canonical derivation

49. Picking the best parse in a PCFG • A sentence will usually have several parses • we usually want them ranked, or only want the n best parses • we need to focus on P(t|s,G) • probability of a parse, given our sentence and our grammar • definition of the best parse for s: • The tree for which P(t|s,G) is highest

50. Probability of a sentence • Given a probabilistic context-free grammar G, we can the probability of a sentence (as opposed to a tree). • Observe that: • As far as our grammar is concerned, a sentence is only a sentence if it can be recognised by the grammar (it is “legal”) • There can be multiple parse trees for a sentence. • Many trees whose yield is the sentence • The probability of the sentence is the sum of all the probabilities of the various trees that yield the sentence.