Download
1 / 36
360 likes | 489 Vues
This article explores the methodologies for estimating parameters in Probabilistic Context-Free Grammars (PCFGs), emphasizing supervised and unsupervised learning approaches. It examines the steps involved in preprocessing treebank data, overcoming challenges like sparsity through smoothing techniques, and the significance of labeled data. Furthermore, it delves into learnability concepts and the conditions under which a class of languages can be effectively learned, referencing foundational work by Gold and Horning. The document illustrates the complexities and insights associated with grammar induction and automatic language processing.
E N D
Learning PCFGs: Estimating Parameters,Learning Grammar Rules Many slides are taken or adapted from slides by Dan Klein
Treebanks An example tree from the Penn Treebank
The Penn Treebank • 1 million tokens • In 50,000 sentences, each labeled with • A POS tag for each token • Labeled constituents • “Extra” information • Phrase annotations like “TMP” • “empty” constituents for wh-movement traces, empty subjects for raising constructions
Supervised PCFG Learning • Preprocess the treebank • Remove all “extra” information (empties, extra annotations) • Convert to Chomsky Normal Form • Possibly prune some punctuation, lower-case all words, compute word “shapes”, and other processing to combat sparsity. • Count the occurrence of each nonterminal c(N) and each observed production rule c(N->NL NR) and c(N->w) • Set the probability for each rule to the MLE: P(N->NL NR) = c(N->NL NR) / c(N) P(N->w) = c(N->w) / c(N) Easy, peasy, lemon-squeezy.
Complications • Smoothing • Especially for lexicalized grammars, many test productions will never be observed during training • We don’t necessarily want to assign these productions zero probability • Instead, define backoff distributions, e.g.: Pfinal(VPtransmogrified -> Vtransmogrified PPinto) = αP(VPtransmogrified -> Vtransmogrified PPinto) + (1-α) P(VP -> V PPinto)
Problems with Supervised PCFG Learning • Coming up with labeled data is hard! • Time-consuming • Expensive • Hard to adapt to new domains, tasks, languages • Corpus availability drives research (instead of tasks driving the research) • Penn Treebank took many person-years to manually annotate it.
Unsupervised Learning • Systems take raw data and automatically detect data • Why? • More data is available • Kids learn (some aspects of) language with no supervision • Insights into machine learning and clustering
Grammar Induction and Learnability • Some have argued that learning syntax from positive data alone is impossible • Gold, 1967: non-identifiability in the limit • Chomsky, 1980: poverty of the stimulus • Surprising result: it’s possible to get entirely unsupervised parsing to work (reasonably) well.
Learnability • Learnability: formal conditions under which a class of languages can be learned • Setup: • Class of languages Λ • Algorithm H (the learner) • H sees a sequence X of strings x1 … xn • H maps sequences X to languages L in Λ • Question is: for what classes Λ do learners H exist?
Learnability [Gold, 1967] • Criterion: Identification in the limit • A presentation of L is an infinite sequence of x’s from L in which each x occurs at least once • A learner H identifies L in the limit if, for any presentation of L, from some point n onwards, H always outputs L • A class Λ is identifiable in the limit if there is some single H which correctly identifies in the limit every L in Λ. • Example: L = {{a},{a,b}} is identifiable in the limit. • Theorem (Gold, 67): Any Λwhich contains all finite languages and at least one infinite language (ie is superfinite) is unlearnable in this sense.
Learnability [Gold, 1967] • Proof sketch • Assume Λ is superfinite, H identifies Λ in the limit • There exists a chain L1⊂ L2⊂ … L∞ • Construct the following misleading sequence • Present strings from L1 until H outputs L1 • Present strings from L2 until H outputs L2 • … • This is a presentation of L∞ but H never outputs L∞
Learnability [Horning, 1969] • Problem, IIL requires that H succeeds on all examples, even the weird ones • Another criterion: measure one identification • Assume a distribution PL(x) for each L • Assume PL(x) puts non-zero probability on all and only the x in L • Assume an infinite presentation of x drawn i.i.d. from PL(x) • H measure-one identifies L if the probability of [drawing a sequence X from which H can identify L] is 1. • Theorem (Horning, 69): PCFGs can be identified in this sense. • Note: there can be misleading sequences, but they have to be (infinitely) unlikely
Learnability [Horning, 1969] • Proof sketch • Assume Λ is a recursively enumerable set of recursive languages (e.g., the set of all PCFGs) • Assume an ordering on all strings x1 < x2 < … • Define: two sequences A and B agree through n iff for all x<xn, x is in A x is in B. • Define the error set E(L,n,m): • All sequences such that the first m elements do not agree with L through n • These are the sequences which contain early strings outside of L (can’t happen), or which fail to contain all of the early strings in L (happens less as m increases) • Claim: P(E(L,n,m)) goes to 0 as m goes to ∞ • Let dL(n) be the smallest m such that P(E) < 2–n • Let d(n) be the largest dL(n) in first n languages • Learner: after d(n), pick first L that agrees with evidence through n • This can only fail for sequences X if X keeps showing up in E(L, n, d(n)), which happens infinitely often with probability zero.
Learnability • Gold’s results say little about real learners (the requirements are too strong) • Horning’s algorithm is completely impractical • It needs astronomical amounts of data • Even measure-one identification doesn’t say anything about tree structures • It only talks about learning grammatical sets • Strong generative vs. weak generative capacity
Unsupervised POS Tagging • Some (discouraging) experiments [Merialdo 94] • Setup: • You know the set of allowable tags for each word (but not frequency of each tag) • Learn a supervised model on k training sentences • Learn P(w|t), P(ti|ti-1,ti-2) on these sentences • On n>k, reestimate with EM
Grammar Induction Unsupervised Learning of Grammars and Parameters
Right-branching Baseline • In English (but not necessarily in other languages), trees tend to be right-branching: • A simple, English-specific baseline is to choose the right chain structure for each sentence.
Learn PCFGs with EM [Lari and Young, 1990] • Setup: • Full binary grammar with n nonterminals {X1, …, Xn} (that is, at the beginning, the grammar has all possible rules) • Parse uniformly/randomly at first • Re-estimate rule expecations off of parses • Repeat • Their conclusion: it doesn’t really work
EM for PCFGs: Details • Start with a “full” grammar, with all possible binary rules for our nonterminals N1 … Nk. Designate one of them as the start symbol, say N1 • Assign some starting distribution to the rules, such as • Random • Uniform • Some “smart” initialization techniques (see assigned reading) • E-step: Take an unannotated sentence S, and compute, for all nonterminals N, NL, NR, and all terminals w: E(N | S), E(N->NL NR, N is used| S), E(N->w, N is used| S) • M-step: Reset rule probabilities to the MLE: P(N->NL NR) = E(N->NL NR|S) / E(N | S) P(N->w) = E(N->w | S) / E(N | S) • Repeat 3 and 4 until rule probabilities stabilize, or “converge”
Definitions This is the sum of P(T, S | G) over all possible trees T for w1m where the root is N1.
E-Step • We can define the expectations we want in terms of π, α, β quantities:
Inside Probabilities Base case: Induction: Nj Nl Nr wp wd wd+1 wq
Outside Probabilities Base case: Induction:
A nested distributional model • We’d like a model that • Ties spans to linear contexts (like distributional clustering) • Considers only proper tree structures (like PCFGs) • Has no symmetries to break (like a dependency model)
