SYNTAX BASED MACHINE TRANSLATION

1. SYNTAX BASED MACHINE TRANSLATION UNDER GUIDANCE OF PROF PUSHPAK BHATTACHARYYA PRESENTED BY ROUVEN RӦHRIG (10V05101) ERANKI KIRAN (10438004) SRIHARSA MOHAPATRA (10405004) ARJUN ATREYA (09405011)

2. OUTLINE Motivation Introduction Synchronous grammar Syntax based Language Model for SMT Hierarchical Phrase-Based MT Example Hindi translations Joshua Toolkit Conclusions

3. Motivation • Consider the following English-Japanese example: • (1) The boy stated that the student said that the teacher danced • (2) shoonen-ga gakusei-ga sensei-ga odotta to itta to hanasita • The-boy the-student the-teacher danced that said that stated • -> Easy to translate the words. • -> Very hard find the correct reordering! • Syntax-based machine translation techniques start with the syntax. • Some can deliver guaranteed correct syntax! David Chiang - An Introduction to Synchronous Grammars, Proceedings of the 43rd Annual Meeting of the ACL, Ann Arbor, 21 June 2006.

4. Introduction (1) • Syntax-based Language Model • Noisy channel model • Uses 3 steps starting from the parse tree • 1. Reordering - create foreign language syntax tree • 2. Insertion - add extra words which are required in target language • 3. Translation - Translation of leaf words Eugene Charniak, Kevin et al. - Syntax based Language Models for Statistical Machine Translation Brown Univ.(2002)

5. Introduction (2) • Basic phrase-based model • Uses phrases instead of words • Instance of noisy channel model • Modeled as known: • argmaxP(e | f) = argmaxP(e, f) = arg max(P(e) x P(f | e)) • Then 1. Segmentation of e into phrases ē1… ēI , • 2. Reordering of ēi • 3. Translation of ēi using P(f ̄ | ē) • Problem: usually phrases reordered independent of their content •  It is desirable to include a larger scope David Chiang - A Hierarchical Phrase-Based Model for Statistical Machine Translation, Proceedings of the 43rd Annual Meeting of the ACL, Ann Arbor, June 2005.

6. Introduction (3) • Hierarchical Phrase-Based Model • Consists of words and phrases. • For example: • English: "Australia is one of the few countries that have diplomatic relations with North Korea" • German: ''Australien ist eines der weniges Länder, das diplomatische Beziehungen mit Nord-Korea hat" • One example of of a hierarchical phrase is • <[1] mit [2] hat, have [1] with [2]> • [i] are placeholders for sub-phrases. • Captures the fact of different placing in German and English David Chiang - A Hierarchical Phrase-Based Model for Statistical Machine Translation, Proceedings of the 43rd Annual Meeting of the ACL, Ann Arbor, June 2005.

7. Synchronous grammar (1) • Production of a syntactic correct source language string • will always deliver a syntactic correct target language string • Generalizes context-free grammars (CFGs) • Generates pair of strings e.g. • (1) S → ⟨NP[1] VP[2] ,NP[1] VP[2] ⟩ • (2) VP → ⟨V[1] NP[2], NP[2] V[1] ⟩ • [i] model the relations of non-terminal symbols • Applying rule (1) and (2) produces: • Replacing S → ⟨NP[1] VP[2], NP[1] VP[2]⟩ • => ⟨NP[1] V[3] NP[4] ,NP[1] NP[4] V[3]⟩ - David Chiang - An Introduction to Synchronous Grammars, Proceedings of the 43rd Annual Meeting of the ACL, Ann Arbor, 21 June 2006.

8. Synchronous grammar (2) • ⟨NP[1] V[3] NP[4] ,NP[1] NP[4] V[3]⟩ • When applying a rule, both sides have to be replaced similarly! • When replacing NP[1] on the left side, then also NP[1] on the ride side. • NP → ⟨I, watashiwa⟩ • NP → ⟨the box, hako wo⟩ • V → ⟨open, akemasu⟩ • => ⟨I open the box ,watashiwahakowoakemasu⟩ David Chiang - An Introduction to Synchronous Grammars, Proceedings of the 43rd Annual Meeting of the ACL, Ann Arbor, 21 June 2006.

9. Synchronous grammar (3) • Solution for everything? • -> Lowering or raising of tree is not possible! • Example: • John misses Mary • Mary manque à John • (Mary is-missed by John) • S → <NP[0] VP[1], NP[0] VP[1]> •  “à John“ is part of the VP • NP → <John, John> • NP → <Mary, Mary> • Not possible to replace correctly! An Introduction to Synchronous Grammars - David Chiang, Proceedings of the 43rd Annual Meeting of the ACL, Ann Arbor, 21 June 2006.

10. Syntax based Language Model for SMT • Noisy channel model – where source Language Sentence E is distorted by the channel into the Foriegn Language F. argmaxE p(E|F) = argmaxE p(E)p(F|E) .. .. ..(1)LM TM • Base SMT System: • It is a parse tree-string tranlsation model (english parse tree[input]-->French sentence [output] • p(E|F) ∞ ∑p(E, π)p(F|E, π) • where π – parse tree of english sentence • This model performs 3 types of operations – reorders, insertion, translation • The direction of real translation(decoding) is reverse of translation Model • Extract CFG rules from parsed corpus of english, using std. Bottom up parser. • A decoder is given chinese sentence to get best english parse tree p(E), p(F|E) Eugene Charniak, Kevin et al. - Syntax based Language Models for Statistical Machine Translation Brown Univ.(2002)

11. Syntax based Language Model for SMT Computes the inside, outside probabilities of parse forest and eliminate edges which fall below a empirical set of 0.00001 threshold. • Parsing/Language Model: comprises of 2 stages based on Penn tree bank corpus • a. Non-Lexical PCFG (create large parse forest for sentence) • b. Pruning step p(eki,j | w1,n) = α(ni,jk) p(rule(ei,jk)) πnnl,m єrhs(ei,jk) β(nl,mn) • c. Lexical PCFG( examine edges and pull out most probable parse tree from forest) • Issues while parsing – incompatibilities with translation model, phrasal translations, non-linear word ordering. p(w1,n) Eugene Charniak, Kevin et al. - Syntax based Language Models for Statistical Machine Translation Brown Univ.(2002)

12. VB PRP VB1 VB2 He adores VB TO listening TO NN music to VB PRP VB2 VB1 Adores desu He ha VB TO listening no NN TO music to Syntax based Translation Model for SMT Input: ”He adores listening to music” [english parse tree] Output: Kare ha ongaku wo kiku no ga daisuki desu [Japanese sentence] Channel Input Reordering VB PRP VB2 VB1 adores He VB TO listening R-table NN TO music to SVO  SOV Translation Insertion VB PRP VB2 VB1 desuki desu kare ha VB TO ga ga kiku no NN TO Ongaku wo T-table N-table Kenji Yamada, Kevin et al. - Syntax based Translation Model - Southern California Univ.(2002)

13. Syntax based Translation Model for SMT The model parameters probabilities of n(v|N), r(p|R), and t(t|T) decide the behaviour of the translation model. Kenji Yamada, Kevin et al. - Syntax based Translation Model - Southern California Univ.(2002)

14. Hierarchical Phrase-Based MT • Use of heirarchical Phrases not words as translation units • A phrase is a sequence of words • Uses Bi-text to infer the syntax for both source and destination language • The syntax is a synchronous grammar • Inherent reordering • Phrase to phrase alignment • Phrase to phrase translation • Handling divergence • The translation has two phases – training and decoding • The Bi-text is a word aligned corpus: - a set of triples < f, e, ~ > • f is the French sentence (source language) • e is the corresponding English sentence (target language) • ~ is the many-to-many mapping between phrases in the sentences A Hierarchical Phrase-Based Model for Statistical Machine Translation - David Chiang, Proceedings of the 43rd Annual Meeting of the ACL, Ann Arbor, June 2005.

15. Hierarchical Phrase-Based MT • A phrase grammar rule is represented as (an example) • X  < X1fi jX2 , X2ek lX1 > • Where (i, j) is the source phrase boundary and (k,l) is the target phrase boundary • The above example shows the attachment of a subordinate clause is reversed in English • In training phrase the minimal set of all the above rules is extracted • A Derivation D is a set of triples [ R, i, j ] . • Each triple is a step in derivation. • R is the rule used • fi j is the phrase in source language that was rewritten using the grammar • In decoding phase given a French sentence f, D(f) rewrites the sentence in English. An alternate notation for f an e is f(D) and e(D) respectively. David Chiang - A Hierarchical Phrase-Based Model for Statistical Machine Translation, Proceedings of the 43rd Annual Meeting of the ACL, Ann Arbor, June 2005

16. Hierarchical Phrase-Based MT • The following is a partial left-most derivation to the sentence • English: "Australia is one of the few countries that have diplomatic relations with North Korea" David Chiang - A Hierarchical Phrase-Based Model for Statistical Machine Translation, Proceedings of the 43rd Annual Meeting of the ACL, Ann Arbor, June 2005.

17. Hierarchical Phrase-Based MT To decode the CKY parser with beam search has been used Highest probability single derivation is given below: - • The arg max is computed over each derivation tree D yields f • The corresponding English sentence is given by e(D) • In each cell of the CYK parser, the beam search eliminates • Each item that has a score worse than β times the best score in the same cell • Each item that is that is worse than the b-th best item in the same cell • b = 40, β = 10*exp(−1) for X cells; b = 15, β = 10*exp(−1) for S cells David Chiang - A Hierarchical Phrase-Based Model for Statistical Machine Translation, Proceedings of the 43rd Annual Meeting of the ACL, Ann Arbor, June 2005.

18. Hierarchical Phrase-Based MT w(r) is the weight of the rule r [the first formula] Plm is the language model probability for sentence e |e| denotes length of sentence e λlm and λwp denote the respective exponent factors exp(−wp*|e|) is the word penalty Φi and λi denote the feature weight and the exponent David Chiang - A Hierarchical Phrase-Based Model for Statistical Machine Translation, Proceedings of the 43rd Annual Meeting of the ACL, Ann Arbor, June 2005 Franz Josef Och and Hermann Ney - The alignment template approach to statistical machine translation, Computational Linguistics 2004

19. Hierarchical Phrase-Based MT David Chiang - A Hierarchical Phrase-Based Model for Statistical Machine Translation, Proceedings of the 43rd Annual Meeting of the ACL, Ann Arbor, June 2005.

20. Example translation: - Hierarchical Phrase Based MT • S  { X1 X2 , X1ne X2} ---- ENGLISH | hindi • X  { RAM , ram } • X  { HAD TOLD , kaha tha } • THE ENGLISH SENTENCE : - RAM HAD TOLD • S ⇒ < X1 X2 , X1ne X2> • ⇒ < RAM X2 , ramne X2 > • ⇒ < RAM HAD TOLD , ramnekaha tha > • Compared to pure statistical parsing, the hierarchical phrase based (in general syntax based) MT handles dependency and divergence better.

21. Example translation: - Synchronous CFG Translation • S { NP VP | NP ne VP } ---- ENGLISH | hindi • NP  {N | N} • N  { RAM | ram } • VP  { VPAST_P | VPAST_P } • VPAST_P  { HAD TOLD| kaha tha } • THE ENGLISH SENTENCE PARSE TREE • S ⇒NP VP ⇒N VP ⇒RAM VP ⇒RAM VPAST_P⇒RAM HAD TOLD • HINDI TRANSLATION BY APPLYING THE DUAL OF EACH RULE • S ⇒NP ne VP ⇒N neVP⇒ram ne VP⇒ram neVPAST_P⇒ram ne kaha tha

22. Joshua Toolkit • Open source toolkit • Parsing based Machine Translation • Joshua decoder is written in Java with implementation of several algorithms • Chart-parsing • n-gram language model integration • Beam and cube pruning • Unique k-best extraction

23. Goals • Extendibility : Implementation is organized as packages for customization. • End to End Cohesion : Integrated with suffix-array grammar extraction(Burch, et al., 2005) and minimum error rate training(Och, 2003) • Scalability : Parsing and pruning algorithms are implemented with dynamic programming

24. Experiment Data • Training • Chinese - English • 570K parallel data • Language model was built on 130M words • Decoding • SCFG – 3M rules, 49M n-grams • Results shows that it is 22 times faster decoder than others • Translation quality is better than BLEU-4 (Papineni et al., 2002)

25. Joshua Features • Decoding Algorithms • Grammar Formalism • Handles only SCFGs currently • Chart Parsing • Generates one best or k-best translations using CKY algorithm • Pruning • Increases computational efficiency

26. Joshua Features • Decoding Algorithms • Hyper-graphs and k-best extraction • For each source sentence hyper-graph is generated containing set of derivations • K-best extraction is used to retrieve subset of derivations • Parallel and Distributed Computing • Parallel decoding • Distributed language model

27. Conclusions • Syntax based language and translation models provide a promising technique for use in noisy channel SMT. • Syntax based LM can be combined with several MT systems • Parsing Models such as YC, YT, BT have shown perfect translations of 45% by improving the English syntax of translations. • By using syntactic linguistic information of different word orders and case markers the quality of translation can be improved.

28. Conclusions Hierarchical phrase based translation does not require synchronous grammar as input – uses bitext to generate Hierarchical phrase pairs can be learned without any syntactically-annotated training data Improve translation accuracy over pure statistical phrase-based MT by 7.5% The major challenge in future is to produce a complete provable MT Another goal is to reduce the number of derivation trees with a more syntactically-motivated grammar

29. References • 1. Translation-Eugene Charniak, Kevin et al. - Syntax based Language Models for Statistical Machine Brown Univ.(2002) • 2. David Chiang - A Hierarchical Phrase-Based Model for Statistical Machine Translation, Proceedings of the 43rd Annual Meeting of the ACL, Ann Arbor, June 2005. • 3. David Chiang - An Introduction to Synchronous Grammars, Proceedings of the 43rd Annual Meeting of the ACL, Ann Arbor, 21 June 2006. • 4. Franz Josef Och and Hermann Ney - The alignment template approach to statistical machine translation, Computational Linguistics 2004 • 5. ZhifeiLi, Chris Callison-Burch, Chris Dyer, JuriGanitkevitch, Ann Irvine, SanjeevKhudanpur, Lane Schwartz, Wren N. G. Thornton, ZiyuanWang, JonathanWeese and Omar F. Zaidan - Joshua 2.0: A Toolkit for Parsing-Based Machine Translation with Syntax, Semirings, Discriminative Training and Other Goodies - Proceedings of the Joint 5th Workshop on Statistical Machine Translation and MetricsMAT, Uppsala, Sweden, 15-16 July 2010.

30. Thank You