1 / 17

Search in Transliterated Space

This proposal discusses challenges and methods for enhancing information retrieval (IR) in a transliterated context, particularly focusing on Indian and other non-Roman scripts. It highlights the difficulties encountered with spelling variations, code-mixing, and query resolution across diverse scripts. The shared task involves language identification, transliteration errors, and providing relevant document retrieval in both Roman and native scripts. Additionally, it outlines available data resources and seeks partners for expanding capabilities, especially in Bollywood song lyrics and code-mixed texts.

yovela
Télécharger la présentation

Search in Transliterated Space

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Shared Task Proposal, FIRE 2012 Search in Transliterated Space Monojit Choudhury Microsoft Research Lab India

  2. A Transliterated World Wide Web Song Lyrics

  3. A Transliterated World Wide Web Reviews and Forums

  4. A Transliterated World Wide Web Facebook and Twitter

  5. A Transliterated World Wide Web And lot more

  6. Beyond Indic languages • Many languages that use non-Roman script • Arabic (Saudi Arabia, UAE, Egypt, Morocco,…) • Persian • Indian sub-continental languages (IL & Dzongkha, Nepalese, Sinhala) • Thai, Vietnamese • Cyrillic (Russian, Ukrainian) • Chinese, Japanese, Korean (rare)

  7. Aspects of Transliterated Text • Transliteration • Errors, Contraction • Code Mixing

  8. IR Scenario - I Mono-script Monolingual IR in transliterated space • Query: thandeehavayehchandnisuhanee • Results: Only Roman transliterated documents • Challenge: Spelling variations • tandeehawa ye chandnysoohaany

  9. IR Scenario - II Cross-script and Multi-script Monolingual IR in transliterated space • Query: thandeehavayehchandniOR ठंडी हवा ये चाँदनी • Results: Both Roman transliterated or in native script • Challenge: Transliteration

  10. Scenario - III • Cross-script and Cross-lingual IR • Query: death of mareech and subahoo • Document: Hindi (Transliterated and Devanagari) and English documents

  11. Shared Task on Retrieval Transliterated documents in Roman Mono-script Monolingual IR Transliterated query in Roman Transliterated documents in native script Cross-script Monolingual IR Transliterated query in Roman Multi-script Monolingual IR Documents in Roman and native scripts Query in Roman or native script

  12. Shared Sub-Tasks • Language identification of transliterated queries, documents, code-mixed text koodakazhikkanoruurgan split pea soup undaki ML MLMLMLEN ENENML • Transliteration • Forward: കഴിക്കാന്‍ kazhikkan • Backward: kazhikkan കഴിക്കാന്‍

  13. Available Data • 20000 word pairs each in Bengali, Telugu, and Hindi (labeled with language tags) • 35000 unique Hindi-Roman word pairs obtained from aligning Bollywood song lyrics • More data under preparation from FaceBook on mixture of various languages. • Looking for partners to extend!

  14. Available Data • Currently we have 500 query and url-rel judged pairs for Bollywood song lyrics • Looking for partners to extend it to other (Indian) Languages • Other domains?

  15. Thank you! monojitc@microsoft.com

  16. Other resources • Lexicons • Pronunciation lexicons • G2P for some languages • Stemmers and morphological analyzers • Anything else?

  17. Concluding Remarks • We have built Multi-script Bollywood Song Search and working on transliteration and code-mixing • These are just some initial ideas that came up from our experiences • If you are interested please let me know 

More Related