Relish Rendering Endangered Languages Lexicons Interoperable through Standards Harmonization

1. RelishRendering Endangered Languages Lexicons Interoperable through Standards Harmonization • SaLTMIL Workshop • Speech and Language Technology for Minority Languages May 23rd2010 LREC Malta Marc Kemps-Snijders Marc.kemps-snijders@mpi.nl Max Planck Institute for Psycholinguistics

2. Increase interoperability between endangered language lexica created on both sides of the Atlantic

3. Background • Lexica constitute important record of endangered languages • Diverging European and American standards for data formatting and markup • LIFT/LLIFT vs. LMF • GOLD vs. ISOcat • Significant effort in tool support by all parties • Structural differences • Differences in terms and abbreviations • Differences in interchange formats

4. European and American Projects and Standards UF ILIT MPI Dobes LEGO Lexicons of endangered languages Intera EMELD DAM-LR Data Driven Ontology ECHO Standards for Terminology GOLD Community CLARIN DCR GOLD Standards for Lexicons ISO IS 12620:2009 DCR SIL LMF ISO FDIS 24613:2008 LMF LIFT

5. MethodologyBottom up approach LLIFT example <entry id="_123"> <!-- here we've inserted the underscore so the id conforms to xml datatype ID, which cannot begin with a number. --> <trait name="original-id" value="123"/> <!-- this is where we'll keep the original id, since we may need it and we have to put an underscore in front of the entry id, so that it conforms to the datatype id format. We considered using <field> as it seemed more semantically appropriate, but <field> would require <form> inside it, which would in turn require a language attribute, and we don't want that. <field> has no appropriate attributes we could use, either. --> <lexical-unit><!-- optional --><!-- the headword --> <form lang="fuh"> <!-- regarding the lang attribute: The format (based on RFC 4646bis or superseding document): ISO language code-script type-ISO country code. Only the ISO language code is really necessary, though. Q: What to do if we need more than one language code to cover a given form, though? For instance, in Tamashek, where what Heath calls 'dialects' have separate ISO codes? A: Use the private use 'x-' format, ie: taq-x-ttq-thz. NB everything following the x- is considered private use, so put anything conforming to the standard first. OR: x-qta (use a temp code, and map it in a URI to all three required codes) Not sure if this would work if we're trying to map individuals to their different possible combinations of dialects, though. --> <text>cow</text> </form> </lexical-unit> <variant> <!-- optional --> <!-- alternate spellings or forms - these can't have any different meaning or grammatical info, as variant can't have <sense> under it. --> <form lang="fuh"> <text>dabere</text> </form> </variant> <variant> <!-- a second variant is possible --> <form lang="fuh"> <text>dabbere</text> </form> Shoebox example \_sh v3.0 400 Iwaidja \_DateStampHasFourDigitYear \lx a \lc Lexical citation ((R) => root) \ps Part of speech \de Definition \ge Gloss-English \re Reversal \xv Example vernacular \xe Example English \rf Reference for example \dt 11/Jul/2007 \lx a- \lc a- \a a- \ps v. prefix \de third person plural intransitive subject prefix \ge 3pl \re they \ng This is the neutral form; the 'towards' form is |fv{ayuwu-}, 'away' form is |fv{ijb-} ~ |fv{ijuwu-} \sd verb prefix \sd inflectional prefix \rf PL93 \xv Amalkban. \xe They move outside. \dt 15/Jul/2007 \lx a- \lc a- \a a- \ps n. pref. \de their (with possessed body parts) \ge 3pl \re their (with possessed body parts) \sd noun prefix \sd inflectional prefix \dt 29/Nov/2006 Lexus example <lexicalEntry> <headword_x0020_group> <date_x0020__x0028_last_x0020_entered_x0029_>11/Jul/2007</date_x0020__x0028_last_x0020_entered_x0029_> <headword>a</headword> <citation_x0020_form>Lexical citation ((R) =&gt; root)</citation_x0020_form> <part_x0020_of_x0020_speech_x0020_group> <part_x0020_of_x0020_speech/> <sense_x0020_number_x0020_group> <contextualized_x0020_example_x0020_group> <example_x0020__x0028_free_x0020_translation_x0029_/> <contextualized_x0020_example/> </contextualized_x0020_example_x0020_group> <definition_x0020_group> <English_x0020_reversal/> <English_x0020_gloss/> <definition/> </definition_x0020_group> <reference_x0020_group> <reference/> </reference_x0020_group> </sense_x0020_number_x0020_group> </part_x0020_of_x0020_speech_x0020_group> </headword_x0020_group> </lexicalEntry> <lexicalEntry> <headword_x0020_group> <date_x0020__x0028_last_x0020_entered_x0029_>12/Jul/2007</date_x0020__x0028_last_x0020_entered_x0029_> <headword>^(d)angkarranaka</headword> <citation_x0020_form>angkarranaka</citation_x0020_form> <part_x0020_of_x0020_speech_x0020_group> <part_x0020_of_x0020_speech>?</part_x0020_of_x0020_speech> <sense_x0020_number_x0020_group> <reference_x0020_group> <reference>IwNo05:19Ap</reference> </reference_x0020_group> <contextualized_x0020_example_x0020_group> ce> </reference_x0020_group> <_x0032_D_x0020_group> <grammatical_x0020_note>The d-initial form is found after prefixes ending in K-; elsewhere the root begins with |fv{a}. The citation form is |fv{dangkarranaka}.</grammatical_x0020_note> </_x0032_D_x0020_group> • Analyze existing lexica to identify commonalities and differences in lexical structure and content Archi Udi Tofa Salar Mocovi Iwaidja Kayardild

6. MethodologyTop down approach • Analyze existing standards for lexical resources (GOLD/LIFT and LMF/DCR) to identify commonalities and differences at the conceptual level. • Harmonize concepts using ISO 12620 Data Category Registry • Harmonize model approaches • Harmonize interchange formats

7. \+DatabaseType MDF 4.0 \ver 5.0 \desc Standard Format markers defined in _Making Dictionaries: A guide to lexicography and the Multi-Dictionary Formatter_. David F. Coward, Charles E. Grimes, and Mark R. Pedrotti. Waxhaw, NC: SIL, 1998. (2nd edition) \+mkrset \lngDefault English \mkrRecord lx \+mkr an \nam Antonym \desc Used to reference an antonym of the lexeme, but using the \lf (lexical function) field for this is better practice. \lng vernacular \mkrOverThissn \CharStyle \-mkr \+mkrbw \nam Borrowed word (loan) \desc Used for denoting the source language of a borrowed word. \lng English \mkrOverThis se \CharStyle \-mkr \+mkrce \nam Cross-ref. gloss (E) \desc Gives the English gloss(es) for the vernacular lexeme referenced by the preceding \cf field. \lng English \mkrOverThiscf \CharStyle \-mkr Harmonizing 12620 data categories • All linguistic concepts will be registered in the ISO 12620 Data Category Registry (ISOcat) • Analysis of existing ISOcat data categories vs. GOLD vs. MDF MDF type file GOLD Comunity ISOcat 12620 Data Category Registry

8. \+mkrps \namPart of speech \descClassifies the part of speech. This must reflect the part of speech of the vernacular lexeme (not the national or English gloss). Consistent labeling is important; use the Range Set feature. Sense numbers are beneath \ps in this hierarchy; don't mark different \ps fields with sense numbers. \lng English \rngsetadj adv …… n num pnpost prtcl v \mkrOverThis se \mkrFollowingThisva \CharStyle \-mkr Determiner Harmonizing 12620 data categories Complex Closed Simple Is a article • Example: part of speech PartOfSpeech Definite article Is a Is a Indefinite article ISOcat: MorhoSyntax Profile ... MDF Multi Dictionary Format GOLD ontology

9. Harmonizing 12620 data categoriesGold example 2 • In some cases GOLD contains additional information • Additional extensions to the conceptual domain • isA relations between GOLD concepts GOLD ontology

10. Harmonizing 12620 data categoriesRelation Registries • Relation Registries describes relations not handled through the ISO 12620 model • Simple relations • e.g MDF /PartOfSpeech/ ‘equals’ MorphoSyntax /PartOfSpeech/ • GOLD relations (GOLD ontology is a Relation Registry) • Compositional Relations (DC is composed of multiple more granular DCs) • e.g. UDI MDF \1d (First dual)  person:firstPerson, grammaticalNumber: dual, value:… • Model specific relations • e.g. TBX model

11. Harmonizing 12620 data categoriesRelation Registries Relation registries Data Category registries resource registries

12. Harmonizing interchange formats Possibility to use TEI? • Can TEI serve as interchange format for LMF and be accepted by CLARIN community? • Decision needs to be made before end 2010 to be useful for RELISH • ODD (One Document does all) • Documentation • Schema information • Schema documents validate xml data structure • In August a workshop is organized to discuss the possibility of using TEI as an interchange format with representatives from ISO, CLARIN, TEI and endangered languages community

13. Adapting the tools • Relish project will result in tool adaptation to support the interoperability aspects and interchange formats

14. Conclusions and remarks • Minority and less resourced languages and tools are • starting to actively participate in the standards discussions • becoming part of the e-infrastructure landscape • have the opportunity to play a mature role in the area of language resources • We need organizations and individuals who are actively involved and represent the position of less resources languages in these discussions • Results from Relish project may be useful for other less resourced language resources as well

15. Thank you for your attention Relish was made possible through the DFG/NEH Bilateral Digital Humanities Program