Unicode Transforms in ICU

1. Unicode Transforms in ICU Mark DavisChief SW Globalization Architect IBM

2. What is ICU? • Internationalization libraries for C, C++, Java* • Open source – non-viral • Sponsored by IBM • Sun’s Java licenses an earlier ICU version; ICU4J updates it. • Unicode standard compliant • full supplementary support • Cross-platform; extensible and customizable • High performance and thread-safe • Multiple locales in same thread – simultaneously • http://oss.software.ibm.com/icu/ 22st International Unicode Conference

3. Unicode text handling Character set conversions (700+) Collation & Searching Locales (170+) Resource Bundles Calendar & Time zones Complex-text layout engine Breaks: character, word, line, & sentence Formatting Date & time Messages Numbers & currencies Transforms Normalization Casing Transliterations ICU Features 22st International Unicode Conference

4. ICU Transforms • Powerful, flexible mechanism • Uppercase, Lowercase, Titlecase, Full/Halfwidth • Normalization • Hex, Character Names • Script to Script conversion… • Supports Styled Text, not just Plain Text • Chaining, Filters, Buffering • Customizable 22st International Unicode Conference

5. Transform Examples • “Any-Uppercase” a → A • “Any-Hex/Java” a → \u0061 • “Greek-Latin” a → α 22st International Unicode Conference

6. Filters • “[aeiou] Latin - Greek” • “Latin” is the source • “[aeiou]” is a filter, restricts the application to only English vowels. Uses UnicodeSet. • “Greek” is the target • “[^\u0000-\u007E] Any - Hex” • “A δ is…” → “A \u03B4 is\u2026” 22st International Unicode Conference

7. UnicodeSet • Ranges [ABC a-z] • Union [[:Lu:] [:P:]] • Intersection [[:Lu:] & [\u0000-\u01FF]] • Set Difference [[:Lu:] - [\u0000-\u01FF]] • Complement [^aeiou] • Properties • Uppercase letters[:Lu:] • Punctuation[:P:] • Script[:Greek:] ICU 2.2: all enumerated Unicode 3.2 properties 22st International Unicode Conference

8. UnicodeSet Property Syntax • Either POSIX or Perl Style • \p{letter} • [:letter:] • Short or long form (UCD Property Aliases) • \p{general_category = uppercase_letter} • \p{gc=Lu} • Case-, Space-, Underbar-Insensitive 22st International Unicode Conference

9. Example Filter • “[:Lu:] Latin-Katakana; Latin-Hiragana” • Converts all uppercase Latin characters to Katakana, • Then converts all other Latin characters to Hiragana. 22st International Unicode Conference

10. Chaining Transforms • “Kana-Latin; Any-Title” • たけだ, まさゆき • takeda, masayuki • Takeda, Masayuki • Any number 22st International Unicode Conference

11. Filtering plus Chaining • “NFD; [:Mark:] Remove; NFC” • Decompose • Remove accents (Marks) • Recompose 22st International Unicode Conference

12. Built-in Transforms • Normalization • Å → Å • Casing • a → A • Full ↔ Halfwidth • カ → ｶ • Character Names • a → {LATIN SMALL LETTER A} • Hex: XML, Java, C++, Perl, … styles • a → \u0061, U+0061, … 22st International Unicode Conference

13. Script ↔ Script Conversions • General conversions, e.g.: Greek-Latin • Source-Target Reversible: φ → ph → φ • Not Target-Source Reversible: f → φ → ph • Variants • By Language: Greek-German • By Standard: Greek-Latin/UNGEGN • Can build your own 22st International Unicode Conference

14. 김, 국삼 김, 명희 정, 병호 たけだ, まさゆき ますだ, よしひこ やまもと, のぼる Ρούτση, Άννα Καλούδης, Χρήστος Θεοδωράτου, Ελένη Gim, Gugsam Gim, Myeonghyi Jeong, Byeongho Takeda, Masayuki Masuda, Yoshihiko Yamamoto, Noboru Roútsē, Ánna Kaloúdēs, Chrḗstos Theodōrátou, Elénē “Any-Latin” Example 22st International Unicode Conference

15. Styled Text • Preserves individual styles on letters, where possible απα → apa 22st International Unicode Conference

16. p? ph? ps? When Buffering • Conversions are not performed if they may extend over boundaries Key Result a α p αp a απα p απαp h απαφ 22st International Unicode Conference

17. Custom Rules • Similar to Regular Expressions • Variables • Property matches • Contextual matches • Rearrangement • $1, $2… • Quantifiers: • *, +, ? 22st International Unicode Conference

18. Differences from Reg. Exp.’s • More Powerful… • Buffered/Keyboard • Styled Text • Ordered Rules • Cursor Backup • Less Powerful… • Only greedy quantifiers • No backup: so no (X | Y) • No “input-side back references” 22st International Unicode Conference

19. Example of Custom Rules • “UnixQuotes-RealQuotes” \`\` > “; # two graves → right-quote \'\' > ” ; # two generics → left-quote • Example (SJ Mercury News online) ``expertise''→“expertise” 22st International Unicode Conference

20. Rule Ordering • Find first rule that matches at start • If no match, or (isBuffered & clipped-Match) • advance start by 1 • Else if match, • Substitute text • Move start as specified • Continue until start reaches limit 22st International Unicode Conference

21. Rule Ordering Example Translit. Reg Exp. xy > c ; s/xy/c/g yx > d ; s/yx/d/g xyx-yxy-xyx cx-dy-cx cx-yc-cx 22st International Unicode Conference

22. Context • Rules: • {γ } [ Γ Κ Χ Ξ γ κ χ ξ ] > n; • γ > g; • Meaning: • Convert gamma into n IF followed byΓ, Κ, Χ, Ξ, γ, κ, χ, or ξ • Otherwise into g 22st International Unicode Conference

23. Cursor Backup |BYO • Allows text to be revisited • Reduces rule-count • Example Rules • BY > ビ | ~Y ; • ~YO > ョ; 1 ビ|~YO 2 ビョ| 22st International Unicode Conference

24. Demonstration • Public Demo • http://oss.software.ibm.com/icu/demo • (local copy, samples) 22st International Unicode Conference

25. More Information http://oss.software.ibm.com/… User Guide /icu/userguide/ C /icu/apiref/utrans_h.html C++ /icu/apiref/ Java API /icu4j/doc/com/ibm/text/ • Latest Version of these slides • http://www.macchiato.com 22st International Unicode Conference

26. ICU Transforms • Powerful, flexible mechanism • Uppercase, Lowercase, Titlecase, Full/Halfwidth • Normalization • Hex, Character Names • Script to Script conversion… • Supports Styled Text, not just Plain Text • Chaining, Filters, Buffering • Customizable 22st International Unicode Conference

27. Q & A 22st International Unicode Conference

28. Backup Slides • Not used in the presentation, except in response to questions 22st International Unicode Conference

29. Buffered Usage • No conversion for clipped match …t…t • Fill buffer • Transliterate • May have left-overs x …τ…t th… • Copy left-overs to start • Fill rest of buffer • Transliterate θ… 22st International Unicode Conference

30. Styled Text Handling • Transforms operate on Replaceable, an interface/abstract class defined by ICU • In ICU4c, UnicodeString is a Replaceable subclass (with no out-of-band data -- no styles) • ICU4j defines ReplaceableString, a Replaceable subclass, also with no styles • Clients must define their own Replaceable subclass that implements their styled text. 22st International Unicode Conference

31. Transliteration Sources • Søren Binks • http://homepage.mac.com/sirbinks/translit.html • UNGEGN • http://www.eki.ee/wgrs/ • … 22st International Unicode Conference

32. API: Information • Like other ICU APIs, can get each of the available Transform IDs: • count =Transliterator:: countAvailableIDs(); • myID = Transliterator::getAvailableID(n); • And get a localizable name for each: • Transliterator::getDisplayName(myID, france, nameForUser); Note: these are C++ APIs; C and Java are also available. 22st International Unicode Conference

33. API: Creation • Use an ID to create: • myTrans = Transliterator::createInstance("Latin-Greek"); 22st International Unicode Conference

34. API: Simple usage • Convert entire string • myTrans.transliterate(myString); 22st International Unicode Conference

35. More Control • Specify Context • Use with Styled Text abcdefghijklmnopqrstuvwxyz contextStart contextLimit start limit 22st International Unicode Conference