THE MATHEMATICS OF STATISTICAL MACHINE TRANSLATION Sriraman M Tallam

1. THE MATHEMATICS OF STATISTICAL MACHINE TRANSLATION Sriraman M Tallam

2. The Problem • The problem of machine translation is discussed. • Five Statistical Models are proposed for the translation process. • Algorithms for estimating their parameters are described. • For the learning process, pairs of sentences that are translations of one another are used. • Previous work shows statistical methods to be useful in achieving linguistically interesting goals. • natural extension - matching up words within pairs of aligned sentences. • Results show the power of statistical methods in extracting linguistically interesting correlations.

3. Statistical Translation • Warren Weaver first suggested the use of statisitical techniques for machine translation. [Weaver 1955] • Fundamental Equation for Machine Translation Pr(e|f) = Pr(e) Pr(f|e) --------------- Pr(f) ê = argmax Pr(e) Pr(f|e)

4. Statistical Translation • A translator when writing a French sentence, even a native speaker, conceives an English sentence and then mentally translates it. • Machine translation’s goal is to find that English sentence. • Equation summarizes the 3 computational challenges presented by statistical translation. • Language Model Probability Estimation - Pr(e) • Translational Model Probability Estimation - Pr(f|e) • Search Problem - maximizing their product • Why not reverse the translation models ? • Class Discussion !!

5. Alignments • What is a translation ? • Pair of strings that are translations of one another • (Qu’ aurions-nous pu faire ? | What could we have done ?) • What is an alignment ?

6. Alignments • The mapping in an alignment could be from one-one to many-many. • The alignment in the figure is expressed as • (Le programme a ete mis en application | And the(1) program(2) has(3) been(4) implemented(5,6,7)). • This alignment though acceptable has a lower probability. • (Le programme a ete mis en application | And(1,2,3,4,5,6,7) the program has been implemented). • A(e,f) is the set of alignments of (f|e) • If e has length ‘l’ and f has length ‘m’, there are 2lm alignments in all.

7. Cepts • What is a cept ? • To express the fact that each word is related to a concept, in a figurative sense, a sentence is a web of concepts woven together • The cepts in the example are The, poorand don’t have any money • There is the notion of an empty cept.

8. Translation Models • Five Translation models have been developed. • Each model is a recipe for computing Pr(f|e), which is called the likelihood of the translation (f,e). • The likelihood is a function of many parameters ( !). • The idea is to guess values for these parameters and to apply the EM algorithm iteratively.

9. Translation Models • Models 1 and 2. • all possible lengths are equally possible • In Model 1, all connections for each french position are equally likely. • In Model 2, connection probabilities are more realistic • These models lead to unsatisfactory alignments very often • Models 3,4 and 5. • No assumptions on the length of the French string • Models 3 and 4 make more realistic assumptions on the connection probabilities • Models 1 - 4 are a stepping stone for the training of Model 5 • Start with Model 1 for initial estimates and pipe thru the models, 2 - 5.

10. Translation Models • The likelihood of f | e is, over all elements of A(e,f) • Then, • choose the length of the French string given the English • for each french word position, choose the alignment, given previous alignments and words • choose the identity of the word at this position given our knowledge of the previous alignments and words.

11. Model 1 Assumptions • We assume Pr(m|e) is independent of e and m • All reasonable lengths of the French string are equally likely. • Also, depends only on l. • All connections are equally likely, and for a word there are (l + 1) connections, so this quantity is equal to (l + 1) -1 • is called the translation probability of fj given eaj

12. Model 1 • The joint likelihood function for Model 1 is, and for j = 1 … m, and aj from 1 … l • Therefore, • subject to,

13. Model 1 • Technique of Lagrange Multipliers, • EM algorithm is applied repeatedly.  = X = Y = • The expected number of times e connects to f is t (f | e) f, e, l set of aj

14. Model 1

15. Model 1 -> Model 2 • Model 1 does not take into account where words appear in either string • All connections are equally probable • In Model 2, alignment probabilities are introduced and, which satisfy the constraints,

16. Model 2 • The likelihood function now is, and the cost function is,

17. Fertitlity and Tablet • Fertility of a english word is the number of French words it is connected to - i • Each english word translates to a set of French words called the Tablet - Ti • The collection of Tablets is the Tableau - T. • The final French string is a permutation of the words in the Tableau - 

18. Joint Likelihood of a Tableau and Permutation • The joint likelihood of a Tableau and Permutation is, • and ,

19. Model 3 Assumptions • The fertility probability of an english word only depends on the word. • The translation probability is, • The distortion probability is,

20. Model 3 • The likelihood function for Model 3 is now,

21. Deficiency of Model 3 • The fertility of word i does not depend on the fertility of previous words. • Does not always concentrate its probability on events of interest. • This deficiency is no serious problem. • It might decrease the probability of all well-formed strings by a constant factor.

22. Model 4 • Allowing Phrases in the English String to move and be translated as units in the French String • Model 3 doesn’t account for this well, because of the word by word movement. where, A and B are functions of the French and English words. • Using this they account for facts that an adjective appears before a noun in English and reverse in Frernch. - THIS IS GOOD !

23. Model 4 • For example, implemented produces mis en application, all occuring together, whereas notproduces ne pas which occurs with a word in between. • So, d>1(2 | B(pas)) is relatively large when compared to d>1(2 | B(en)) • Models 3 and 4 are both deficient. Words can be placed before the first position or beyond the last position in the French string. Model 5 removes this deficiency.

24. Model 5 • They define to be the number of vacancies up to and including position j just before forming the words of the ith cept. • And, this gives rise to the following distortion probability equation, • Model 5 is powerful but must be used in tandem with the other 4 models.

25. Results

26. Changing Viterbi Alignments with Iterations

27. Key Points from Results • Words like nodding have a large fertility because they don’t slip gracefully into French. • Words like should do not have a fertility greater than one but they translate into many different possible words, their translation probability is spread more. • Words like the have zero fertlility some times since English prefers an article in some places where French does not.