1 / 58

Automatic conversion of a Japanese text corpus into f-structure

Automatic conversion of a Japanese text corpus into f-structure. OYA, Masanori ( 大矢 政徳 ) National Center for Language Technology School of Computing, Dublin City University. 1. Overview. Japanese grammar Kyoto Text Corpus (KTC) Converting KTC into dependency trees

joella
Télécharger la présentation

Automatic conversion of a Japanese text corpus into f-structure

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Automatic conversion of a Japanese text corpus into f-structure OYA, Masanori (大矢 政徳) National Center for Language Technology School of Computing, Dublin City University NCLT Seminar series

  2. 1. Overview • Japanese grammar • Kyoto Text Corpus (KTC) • Converting KTC into dependency trees • Converting KTC into f-structure • Problems • Evaluation • Summary NCLT Seminar series

  3. 2. Japanese grammar • Syntax • Writing system • SOV as the basic word order • Use of particles for grammatical functions • Tense, aspect and mood are specified by verbal or adjectival morphology • “bunsetsu” (sentential units) • Ellipses of core arguments • Topicalization • Two types of relative clauses • Case particles derived from verbs • Adverbial nouns • Coordination NCLT Seminar series

  4. 2. Japanese grammar • Writing system: three different types of scripts • Chinese characters (1945 and more) • Nouns (possible to be written in Hiragana or Katakana) • Stems of verbs and adjectives • Hiragana (104) • Inflections of verbs and adjectives • Particles • Words that have no Chinese counterparts • Katakana (124) • Nouns borrowed from foreign languages • Technical and scientific names • Onomatopoeia • No spaces are given between words NCLT Seminar series

  5. 2. Japanese grammar The chart of Hiragana The chart of Katakana NCLT Seminar series

  6. 2. Japanese grammar • SOV as the basic word order; scrambling is prevalent • Use of particles for grammatical functions Example: 太郎はダブリンの大学に行った。 Taro-wa dabulin-no daigaku-ni it-ta Taro-TOP Dublin-in college-to go-PST “Taro went to a college in Dublin.” • “-wa”, “-ga”, “-wo” and “-ni” – used for core grammatical functions • Other particles – used for adjuncts (postpositional phrases or complementizer) (Tsujimura 2006) • The particle “-ni” is ambiguous; it can be used as the OBL case marker or a postposition for temporal or locative adverbials (semantic distinction is possible). NCLT Seminar series

  7. 2. Japanese grammar • Tense, aspect and mood are specified by verbal or adjectival morphology Example: 太郎はダブリンの大学に行っている。 Taro-wa dabulin-no daigaku-ni it-teiru Taro-TOP Dublin-in college-to go-PROG.PRES “Taro is going to a college in Dublin.” 太郎はダブリンの大学に行ったのだろうか。 Taro-wa dabulin-no daigaku-ni it-ta-nodarou-ka Taro-TOP Dublin-in college-to go-PST-AUX-INT “(I wonder) whether Taro went to a college in Dublin.”etc. NCLT Seminar series

  8. 2. Japanese grammar • “Bunsetsu”, or syntactic units • One bunsetsu = a content word + a particle or inflection ≈ Chinese characters + hiragana or katakana Example: 太郎はダブリンの大学に行っている。 Taro-wa dabulin-no daigaku-ni it-teiru Unit 0 Unit 1 Unit 2 Unit 3 • Spaces represent bunsetsu boundaries. • Hyphens represent morphological boundaries within a bunsetsu. NCLT Seminar series

  9. 2. Japanese grammar • Ellipses of core arguments • Pro-drop: contextually-evident units are absent from the sentence • Gender, person and number of the subject are not specified by verbal or adjectival morphology Example: ダブリンの大学に行った。 dabulin-no daigaku-ni it-ta Dublin-in college-OBL go-PST “I/We/You/He/She/It/They/(Someone in the context) went to a college in Dublin.” -Personal pronouns are also available, but they are not equivalent with personal pronouns in English (e.g., variations of 1st singular personal pronouns: ‘ore’, ‘atashi’, ‘boku’, ‘watashi’, ‘watakushi’, etc.; variations of 2nd singular personal pronouns: ‘kimi’, ‘anata’, ‘anta’, ‘omae’, etc) NCLT Seminar series

  10. 2. Japanese grammar • Topicalization • Topicalized units have the particle “wa” • Non-topicalized units are the focus of the sentence Example: ダブリンの大学には太郎が行った。 dabulin-no daigaku-ni-wa Taro-gait-ta Dublin-in college-OBL-TOP Taro-NOM go-PST “To a college in Dublin, Taro went.” or “It is Taro who went to a college in Dublin NCLT Seminar series

  11. 2. Japanese grammar • Relative clauses • If a clause ends with a verb in a sentence-ending form and it comes before a noun, then the clause is a relative clause: Japanese has no relative pronouns. Example: 私が行った大学 watashi-ga ittadaigaku 1sg-NOM go-PST college “the college I went to.” NCLT Seminar series

  12. 2. Japanese grammar • Two types of “relative clauses”; “inner” relative clauses (true relative clauses) and “outer” relative clauses (appositions) (Teramura 1991) Example: • 私が答えを見つけた証拠 watashi-ga kotae-wo mitsuketa shoko 1sg-NOM answer-ACC find-PST evidence “The evidence that I found out the answer” (“outer”) • 私が見つけた証拠 watashi-ga mitsuketa shoko 1sg-NOM find-PST evidence “The evidence that I found out ∅” (“inner”: ∅ =evidence) “The evidence that I found out PRO” (“outer”: PRO≠evidence; something evident in the context) • If one of the core arguments is in ellipsis, then it is difficult to distinguish a true relative clause from an apposition. NCLT Seminar series

  13. 2. Japanese grammar • Particles derived from verbs: • Some case particles are derived from verbs; case particles of this type have a fixed meaning (Masuoka and Takubo 1992) Example: ついて tsuite “about” (same form with the adverbial form of the verb つく “attach”) 私は計算言語学について話した。 Watashi-wa keisangengogaku-ni-tsuite hanashi-ta I-TOP computational linguistics-OBL-about talk-PST “I talked about computational linguistics.” NCLT Seminar series

  14. 2. Japanese grammar • Adverbial nouns • They function as the head of an adverbial phrase with a complement (Masuoka and Takubo 1992) Example: ダブリンの大学に通っている時、津波が日本を襲った。 Dabulin-no daigaku-ni kayotteiru toki, tsunami-ga nihon-wo osot-ta. Dublin-in college-OBL go-PROG time, tsunami-NOM Japan-ACC strike-PST “When I was studying at a college in Dublin, a tsunami struck Japan.” • It is also difficult to distinguish the complements in these cases from relative clauses; no syntactic nor morphological clues are available. NCLT Seminar series

  15. 2. Japanese grammar • Coordination • The first coordinated bunsetsu has the particle “to” (but not necessarily), and it is dependent on the next coordinated bunsetsu. Example: ダブリンの大学に通っている時、地震と津波が日本を襲った。 Dabulin-no daigaku-ni kayotteiru toki, jishin-to tsunami-ga nihon-wo osot-ta. Dublin-in college-OBL go-PROG time, jishin-AND tsunami-NOM Japan-ACC strike-PST “When I was studying at a college in Dublin, an earthquake and a tsunami struck Japan.” • Only the last coordinated bunsetsu has the particle which specifies its grammatical function; NCLT Seminar series

  16. 3. Kyoto Text Corpus (KTC) • An automatically parsed text corpus of a newspaper (Mainichi Shimbun) • All articles from the 1st to the 17th of January, 1995 (19,687 sentences, 518,687 tokens) and the editorials from January to December, 1995 (18,708 sentences, 453995 tokens). • Developed by Sadao Kurohashi and Makoto Nagao at the University of Kyoto, using JUMAN and KNP NCLT Seminar series

  17. 3. Kyoto Text Corpus (KTC) • All the texts are automatically annotated with morphological tags by JUMAN (Kurohashi and Nagao 1994) (“juman” means 100,000) • The output of JUMAN are parsed by KNP (Kurohashi and Nagao 1994) based on the dependency among “bunsetsu”, and corrected manually • No syntactic CFG category tags are annotated • Valency of verbal predicate is not annotated NCLT Seminar series

  18. 3. Kyoto Text Corpus (KTC) • JUMAN: morphological analyzer for Japanese based on Bigram information • Least-cost path method (Kurohashi and Kawahara 1992) • Costs are assigned to each morpheme and each pair of two morphemes in a sentence • The lower the morpheme frequency, or the lower the frequency of pairs of morphemes, the higher the cost • If a sentence has several possible analyses, JUMAN sums up the costs, and determines the least-cost analysis as the most plausible analysis for the sentence • Accuracy: around 99.0 % (comparison of automatic analysis and manually corrected analysis of 10,000 sentences) NCLT Seminar series

  19. 3. Kyoto Text Corpus (KTC) The example of the output of JUMAN: 太郎は大学に行った。“Taro went to a college.” Taro wa daigaku ni itta. Taro TOP college OBL went NCLT Seminar series

  20. 3. Kyoto Text Corpus (KTC) The example of the output of JUMAN: 太郎は大学に行った。“Taro went to a college.” Taro wa daigaku ni itta. Taro TOP college OBL went #S-ID: 950101001-001 太郎 tarou * Noun Name * * は wa * Particle AdverbialParticle * * 大学 daigaku * Noun NormalNoun * * に ni * Particle CaseParticle * * 行った itta iku Verb * ConsonantVerb Past 。* mark period * * EOS NCLT Seminar series

  21. 3. Kyoto Text Corpus (KTC) • KNP: dependency structure analyzer based on “bunsetsu” • KNP converts the output of JUMAN into a bunsetsu strings. • Accuracy: 90%(comparison of automatic analysis and manually corrected analysis of 10,000 sentences)(Kurohashi and Nagao 1998) NCLT Seminar series

  22. 3. Kyoto Text Corpus (KTC) 太郎は大学に行った。“Taro went to a college.” Taro-wa daigaku-ni it-ta. Taro TOP college OBL went #S-ID: 950101001-001 太郎 tarou * Noun Name * * は wa * Particle AdverbialParticle * * 大学 daigaku * Noun NormalNoun * * に ni * Particle CaseParticle * * 行った itta iku Verb * ConsonantVerb Past 。* mark period * * EOS NCLT Seminar series

  23. 3. Kyoto Text Corpus (KTC) 太郎は大学に行った。“Taro went to a college.” Taro-wa daigaku-ni it-ta. Taro TOP college OBL went #S-ID: 950101001-001 * 0 2D 太郎 tarou * Noun Name * * は wa * Particle AdverbialParticle * * *1 2D 大学 daigaku * Noun NormalNoun * * に ni * Particle CaseParticle * * *2 -1D 行った itta iku Verb * ConsonantVerb Past 。* mark period * * EOS NCLT Seminar series

  24. 3. Kyoto Text Corpus (KTC) 太郎は大学に行った。“Taro went to a college.” Taro wa daigaku ni itta. Taro TOP college OBL went #S-ID: 950101001-001 * 0 2D 太郎 tarou * Noun Name * * は wa * Particle AdverbialParticle * * *1 2D 大学 daigaku * Noun NormalNoun * * に ni * Particle CaseParticle * * *2 -1D 行った itta iku Verb * ConsonantVerb Past 。* mark period * * EOS Unit 1 Unit 2 Unit 0 NCLT Seminar series

  25. 3. Kyoto Text Corpus (KTC) 大学に太郎は行った。“Taro went to a college.” daigaku ni Taro wa itta. college OBL Taro TOP went #S-ID: 950101001-001 *0 2D 大学 daigaku * Noun NormalNoun * * に ni * Particle CaseParticle * * *1 2D 太郎 tarou * Noun Name * * は wa * Particle AdverbialParticle * * *2 -1D 行った itta iku Verb * ConsonantVerb Past 。* mark period * * EOS Unit 1 Unit 2 Unit 0 NCLT Seminar series

  26. 4. Converting KTC into dependency trees • Motivation: • LFG-based automatic grammar induction for Japanese; GramLab: Treebank based Acquisition of Multilingual Resources (Cahill et al. 2002, etc.) • Related work: • Japanese XLE at Fuji Xerox (Masuichi et al. 2006, etc. ) • PCFG-based Automatic grammar induction from Japanese Corpus (Tokunaga et al. 2005, etc.) • Case frame induction from Japanese Corpus (Kurohashi et al. 2006, etc.) NCLT Seminar series

  27. 4. Converting KTC into dependency trees • Procedure: • At least one syntactic category is annotated on each "bunsetsu" in a sentence. • All “bunsetsu’ in a sentence are integrated into a dependency tree of the sentence. Text corpus Dependency trees F-structures NCLT Seminar series

  28. 4. Converting KTC into dependency trees 太郎は大学に行った。“Taro went to a college.” Taro wa daigaku ni itta. Taro TOP college OBL went #S-ID: 950101001-001 * 0 2D 太郎 tarou * Noun Name * * は wa * Particle AdverbialParticle * * *1 2D 大学 daigaku * Noun NormalNoun * * に ni * Particle CaseParticle * * *2 -1D 行った itta iku Verb * ConsonantVerb Past 。* mark period * * EOS Unit 1 Unit 2 Unit 0 NCLT Seminar series

  29. 4. Converting KTC into dependency trees 太郎は大学に行った。“Taro went to college.” Taro wa daigaku ni itta. Taro TOP college OBL went #S-ID: 950101001-001 * 0 2D 太郎 tarou * Noun Name * * は wa * Particle AdverbialParticle * * *1 2D 大学 daigaku * Noun NormalNoun * * に ni * Particle CaseParticle * * *2 -1D 行った itta iku Verb * ConsonantVerb Past 。* mark period * * EOS Topic: OBL: Unit 1 Unit 2 Unit 0 TopP NP V NCLT Seminar series

  30. 4. Converting KTC into dependency trees 。 ni Taro wa daigaku itta NCLT Seminar series

  31. 5. Converting KTC into f-structures • Motivation: • Are syntactic categories necessary for Japanese? • Word order is (relatively) free. • The type (or absence) of particles in each unit specifies its grammatical function (e.g., if a noun has a particle “wo”, then it is an object) • Verbal morphology specifies the grammatical function of each clause (but not always unambiguous). NCLT Seminar series

  32. 5. Converting KTC into f-structures • Generating f-structure equations directly from the corpus Text corpus Dependency trees F-structures NCLT Seminar series

  33. 5. Converting KTC into f-structures • Generating f-structure equations directly from the corpus • F-structure equations are directly generated from each unit. • All the units are unified into the f-structure of the sentence according to the dependency. Text corpus F-structures NCLT Seminar series

  34. 4. Converting KTC into dependency trees 太郎は大学に行った。“Taro went to a college.” Taro wa daigaku ni itta. Taro TOP college OBL went #S-ID: 950101001-001 * 0 2D 太郎 tarou * Noun Name * * は wa* Particle AdverbialParticle * * *1 2D 大学 daigaku * Noun NormalNoun * * に ni * Particle CaseParticle * * *2 -1D 行った itta iku Verb * ConsonantVerb Past 。* mark period * * EOS Unit 0 Unit 1 Unit 2 NCLT Seminar series

  35. 4. Converting KTC into dependency trees 太郎は大学に行った。“Taro went to college.” Taro wa daigaku ni itta. Taro TOP college OBL went #S-ID: 950101001-001 * 0 2D 太郎 tarou * Noun Name * * は wa * Particle AdverbialParticle * * *1 2D 大学 daigaku * Noun NormalNoun * * に ni * Particle CaseParticle * * *2 -1D 行った itta iku Verb * ConsonantVerb Past 。* mark period * * EOS Topic: OBL: f1 f0 f2 NCLT Seminar series

  36. 4. Converting KTC into dependency trees 太郎は大学に行った。“Taro went to college.” Taro wa daigaku ni itta. Taro TOP college OBL went #S-ID: 950101001-001 * 02D 太郎 tarou * Noun Name * * は wa * Particle AdverbialParticle * * *12D 大学 daigaku * Noun NormalNoun * * に ni * Particle CaseParticle * * *2-1D 行った itta iku Verb * ConsonantVerb Past 。* mark period * * EOS Functional equations from the corpus: F2:pred='行く', F2:tns='pst', F2:stmt='decl', F2:style='plain', F0:pred='太郎', F0:prtav='は', F0 elm F2:topic, F1:pred='大学', F1:case='に', F2:obl=F1. NCLT Seminar series

  37. 4. Converting KTC into dependency trees 太郎は大学に行った。 “Taro went to college.” Taro wa daigaku ni itta. Taro TOP college OBL went F2:pred='行く', F2:tns='pst', F2:stmt='decl', F2:style='plain', F0:pred='太郎', F0:prtav='は', F0 elm F2:topic, F1:pred='大学', F1:case='に', F2:obl=F1. NCLT Seminar series

  38. 4. Converting KTC into dependency trees 太郎は大学に行った。“Taro went to college.” Taro wa daigaku ni itta. Taro TOP college OBL went F2:pred='行く', F2:tns='pst', F2:stmt='decl', F2:style='plain', F0:pred='太郎', F0:prtav='は', F0 elm F2:topic, F1:pred='大学', F1:case='に', F2:obl=F1. F-structure from the functional equations: pred : '行く' tns : pst stmt : decl style : plain topic : 1 : pred : '太郎' prtav : 'は' obl : pred : '大学' case : 'に' NCLT Seminar series

  39. 5. Problems • This “Generating f-structure equations directly from the corpus” method does not always work well. • Core argument ellipses • Two types of relative clauses • Particles derived from verbs • Adverbial nouns • Coordination • The context among units must be taken into consideration to make the generation more accurate. NCLT Seminar series

  40. 5. Problems • Ellipses of core arguments • Contextually-evident units are absent from the sentence • Gender, person and number of the subject are not specified by verbal or adjectival morphology Example: ダブリンの大学に行った。 dabulin-no daigaku-ni it-ta Dublin-in college-OBL go-PST “He/She/They went to a college in Dublin.” NCLT Seminar series

  41. 5. Problems • Core argument ellipses • KTC does not annotate on missing elements. • No equations for missing elements can be generated from KTC. • For the f-structure with ellipses, “PRO” must be added to make the f-structure complete. NCLT Seminar series

  42. 5. Problems • Core argument ellipses • If a predicate has no subject in the clause, then an equation for the subject is added. • If a transitive verb has no object, then an equation for the subject must be added … • However, KTC does not annotate on the valency of verbal predicate, hence it is impossible to tell which verb is transitive only on the basis of annotated information. • Case-frame is required to detect missing objects of transitive verbs. NCLT Seminar series

  43. 5. Problems • Two types of “relative clauses”; “inner” relative clauses (true relative clauses) and “outer” relative clauses (appositions) (Teramura 1991) Example: 私が答えを見つけた証拠 watashi-ga kotae-wo mitsuketa shoko 1sg-NOM answer-ACC find-PST evidence “The evidence that I found out the answer” (“outer”) 私が見つけた証拠 watashi-ga mitsuketa shoko 1sg-NOM find-PST evidence “The evidence that I found out ∅” (“inner”: ∅ =evidence) “The evidence that I found out PRO” (“outer”: PRO≠evidence; something evident in the context) If one of the core arguments is in ellipsis, then it is difficult to distinguish an “outer” relative clause from an “inner” relative clause. NCLT Seminar series

  44. 5. Problems • Two types of relative clause • Features in one bunsetsu are not enough to distinguishthem. • A probabilistic model of analysing them (Abekawa and Okumura 2005) employs the cooccurrence probability of head nouns and verbal predicates in “outer” relative clauses. • This method is expected to be applicable to the present method (in future). NCLT Seminar series

  45. 5. Problems • Case particles derived from verbs • Case particles of this type are analyzed by KNP as verbs, not as case particles. • Bunsetsus with them are analyzed as sentential adjuncts, not as postpositional adjuncts or as complements (in the case of “という”). • The equations must be revised properly. NCLT Seminar series

  46. 5. Problems • Particles derived from verbs: Some case particles are derived from verbs; case particles of this type have a fixed meaning (Masuoka and Takubo 1992) Example: ついて tsuite “about” (same form with the adverbial form of the verb つく “attach”) 私は計算言語学について話した。 Watashi-wa keisangengogaku-ni-tsuite hanashi-ta I-TOP computational linguistics-OBL-about talk-PST “I talked about computational linguistics.” NCLT Seminar series

  47. 5. Problems • Adverbial nouns They function as the head of an adverbial phrase with a complement (Masuoka and Takubo 1992) Example: ダブリンの大学に通っている時、津波が日本を襲った。 Dabulin-no daigaku-ni kayotteiru toki, tsunami-ga nihon-wo osotta. Dublin-in college-OBL go-PROG time, tsunami-NOM Japan-ACC strike-PST “When I was studying at a college in Dublin, a tsunami struck Japan.” • It is also difficult to distinguish the complements in these cases from relative clauses; no syntactic nor morphological clues are available. NCLT Seminar series

  48. 5. Problems • Adverbial nouns • Features in one bunsetsu in not enough to distinguish between them. • If a clause is dependent on an adverbial noun and it is analyzed as a relative clause, then the equation of the clause must be replaced by that of complement. NCLT Seminar series

  49. 5. Problems • Coordination • The first coordinated bunsetsu has the particle “to” (but not necessarily), and it is dependent on the next coordinated bunsetsu. Example: ダブリンの大学に通っている時、地震と津波が日本を襲った。 Dabulin-no daigaku-ni kayotteiru toki, jishin-to tsunami-ga nihon-wo osotta. Dublin-in college-OBL go-PROG time, jishin-AND tsunami-NOM Japan-ACC strike-PST “When I was studying at a college in Dublin, an earthquake and a tsunami struck Japan.” • Only the last coordinated bunsetsu has the particle which specifies its grammatical function; NCLT Seminar series

  50. 5. Problems • Coordination • Only the last coordinated bunsetsu has the particle which specifies its grammatical function; other coordinated bunsetsus cannot be analyzed to have appropriate grammatical functions. • The last coordinated bunsetsu does not have any feature within it as a coordinate; the bunsetsu context must be taken into consideration in order to convert it properly into f-structure equations • Dependency among coordinated bunsetsus must also be reanalyzed; NCLT Seminar series

More Related