An Integrated Phrase Segmentation/Alignment Algorithm for Statistical Machine Translation

1. An Integrated Phrase Segmentation/Alignment Algorithm for Statistical Machine Translation Joy Advisor: Stephan Vogel and Alex Waibel LTI Student Research Symposium

2. Outline • Background • Phrase Alignment Algorithms in SMT • Segmentation Approaches • Integrated Segmentation and Alignment Algorithm (ISA) • Experiments • Discussions LTI Student Research Symposium

3. Statistical Machine Translation • Statistical Machine Translation (Brown et al, 93) • Noisy Channel Model • Translating from F to E • Given a testing sentence f, generate translation e*, which is • Pr(e): Language Model (LM) • Pr(f|e): Translation Model (TM) LTI Student Research Symposium

4. Training • Training • Using large English corpora (e.g. Wall Street Journal) to train an LM • Using bilingual corpora (e.g. Canadian Hansard) to train the TM • To get the building blocks for Pr(f|e) • Word to word translation or phrase to phrase translations • Reordering information • Other features LTI Student Research Symposium

5. Alignment • Alignment for one sentence pair (e,f): • Suppose e has l words: and f has m words • Then alignment a can be represented as: Of m values, each between 0 and l. aj=i means fj is “aligned” to ei, where e0stands for NULL word • In short: alignment tells us which word in e is translated into which word in f LTI Student Research Symposium

6. Alignment Example LTI Student Research Symposium

7. Alignment Models • Alignment algorithms: • IBM model 1 to 5 (Brown et al.) • HMM model similar to IBM2 (Vogel) • Competitive linking (Melamed) • Flow Network (Gaussier) • Others LTI Student Research Symposium

8. IBM Model 1 • IBM model 1 • Easy to train • Simple to understand • Used very often in MT research • One serious problem for IBM models • Word-to-word alignment assumption LTI Student Research Symposium

9. Phrase-to-phrase Alignment • Phrase-to-phrase alignment is better • Mismatch between languages • Phrases encapsulate the context of words • Phrases encapsulate local reordering LTI Student Research Symposium

10. Outline • Background • Phrase Alignment Algorithms in SMT • Segmentation Approaches • Integrated Segmentation and Alignment Algorithm (ISA) • Experiments • Discussions LTI Student Research Symposium

11. Alignment Algorithms • Based on initial word alignment • Train word alignment • Read off phrase-to-phrase alignments from Viterbi path • Examples: • HMM phrase alignment (Vogel) • Alignment templates from IBM 4 (Och) • Bilingual bracketing (Wu, B. Zhao) • Popular in SMT research LTI Student Research Symposium

12. Outline • Background • Phrase Alignment Algorithms in SMT • Segmentation Approaches • Integrated Segmentation and Alignment Algorithm (ISA) • Experiments • Discussions LTI Student Research Symposium

13. Segmentation Approaches • Identify monolingual phrases and segment/bracket phrases into one unit (super-word) (Zhang 2000) • Train the regular word-to-word alignment •  LTI Student Research Symposium

14. Problems in Segmentation Approaches • Segmentation uses only monolingual information • Good segmentations may make alignment even harder •  LTI Student Research Symposium

15. Outline • Background • Alignment Algorithms in SMT • Segmentation Approaches • Integrated Segmentation and Alignment Algorithm (ISA) • Experiments • Discussions LTI Student Research Symposium

16. Integrated Segmentation and Alignment • Let’s look at an example first LTI Student Research Symposium

17. Integrated Segmentation and Alignment • Represent a sentence pair (e,f) as a matrix D • D(i,j) = I’(ei,fj).I’ is a modified point-wise mutual information • A partition over D is a series of non-overlapping rectangle regions d1, d2,…,dm. • Region dk(rs,re,cs,ce) indicates: are aligned to • Segmentation and alignment are achieved at the same time LTI Student Research Symposium

18. Integrated Segmentation and Alignment • Best partition should yield maximum • Computationally intractable to search all possible partitions • Exponential to sentence length • DP: not a good idea. • An optimal policy has the property that whatever the initial state and the initial decisions are, the remaining decisions must constitute an optimal policy with regard to the state resulting from the first decision. -- Richard Bellman's Principle of Optimality • But here, decision of how to expand the first cell changes the search space for the rest of the cells • Using a computationally cheap algorithm to find the “good” partitions LTI Student Research Symposium

19. An Example LTI Student Research Symposium

20. Computational Cheap Algorithm • Assumption: • if the translation for e1e2 is f, I’(e1 , f) should be very “similar” to I’(e2 , f). • Example: • Algorithm • Step1: find the cell in D with max value of I’ • Step2: expand this cell to a rectangle region where all cells in the region has similar I’ as this cell • Repeat Step1 and Step2 until no more regions can be found LTI Student Research Symposium

21. Example: Apply the Algorithm LTI Student Research Symposium

22. Estimate the probabilities for phrase translations • The decoder needs the conditional probabilities P(f|e) • Can not be estimated directly: data sparseness • Convert I’(f,e) to P(f|e) • IBM model 1 style: • Context-dependent style where: and LTI Student Research Symposium

23. Outline • Background • Phrase Alignment Algorithms in SMT • Segmentation Approaches • Integrated Segmentation and Alignment Algorithm (ISA) • Experiments • Discussions LTI Student Research Symposium

24. Experiments • Chinese-English • Small data track • Evaluation: NIST score against 4 human references LTI Student Research Symposium

25. Results • Baseline: IBM model1 + HMM phrase • Compare to using ISA only, and ISA+Baseline LTI Student Research Symposium

26. T-test • Student's t-test at the sentence level LTI Student Research Symposium

27. Compared to IBM1 • Large data track (2.6M English words, 414K Chinese words) LTI Student Research Symposium

28. No IBM1 is Better • Small data track (LDC+IBM1+ISA) • ISA is better even on unigram match than IBM1 LTI Student Research Symposium

29. Summary • Integrated Alignment and Segmentation • Simple algorithm • Enhanced translation quality • Better than IBM models • Higher quality than HMM alignment • A major component in the CMU SMT system LTI Student Research Symposium

30. ISA Toolkit • Location: • /afs/cs.cmu.edu/user/joy/Release/PhraseAlign • Documentation: • /afs/cs.cmu.edu/user/joy/Release/PhraseAlign/documentation/readme.txt • Speed • Example: 4172 sentence pairs (133K En words, 20K Ch words) • About 160 seconds for the alignment (10 loops for each sentence pair) LTI Student Research Symposium

31. Selected References • Franz Josef Och, Christoph Tillmann, Hermann Ney, “Improved Alignment Models for Statistical Machine Translation,” Proceedings of the Joint Conference of Empirical Methods in Natural Language Processing and Very Large Corpora, pp. 20-28. University of Maryland, College Park, MD, June 1999. • Stephan Vogel, Hermann Ney, and Christoph Till-mann, “HMM-based Word Alignment in Statistical Translation,” Proceedings of COLING '96: The 16th International Conference on Computational Linguistics, pp. 836-841. Copenhagen, August 1996. • Stephan Vogel, Ying Zhang, Fei Huang, Alicia Tribble, Ashish Venogupal, Bing Zhao, Alex Waibel, “The CMU Statistical Translation System,” to appear in the Proceedings of MT Summit IX, New Orleans, LA, U.S.A., September 2003. • Ying Zhang, Ralf D. Brown, Robert E. Frederking and Alon Lavie, “Pre-processing of Bilingual Corpora for Mandarin-English EBMT,” Proceedings of MT Summit VIII, Santiago de Compostela, Spain, September 2001. • Ying Zhang, Stephan Vogel, Alex Waibel, "Integrated Phrase Segmentation and Alignment Algorithm for Statistical Machine Translation," in the Proceedings of International Conference on Natural Language Processing and Knowledge Engineering (NLP-KE'03), Beijing, China, October 2003. LTI Student Research Symposium