SlideServe
Télécharger Obtenir Pro
Menu
Accueil Pro
Préférences
ApparenceLight Aide et commentaires
© SlideServe · v2026

Unicode Normalization

ethel avatar
ethel
23 diapositives · 481 Views · Oct 21, 2014

Unicode Normalization. Mark Davis www.macchiato.com. Normalization. Uniqueness two equivalent strings have precisely the same normalized form Fast binary comparison, accurate digital signatures Recommended for XML, JavaScript and other standards. Canonical Equivalence.

Unicode Normalization - slide 1
01 / 23Unicode Normalization
1 / 17

Télécharger la présentation

Unicode Normalization
Une image / un lien ci-dessous est fourni (tel quel) pour télécharger la présentation

Politique de téléchargement : le contenu du site vous est fourni TEL QUEL pour votre information et votre usage personnel ; il ne peut être vendu, concédé sous licence ni partagé sur d’autres sites sans l’accord de son auteur.Le contenu vous est fourni TEL QUEL, pour votre information et votre usage personnel uniquement. Téléchargez la présentation en cliquant sur ce lien. Lors du téléchargement, si vous ne parvenez pas à télécharger une présentation, l’éditeur a peut-être supprimé le fichier de son serveur.Pendant le téléchargement, si vous ne parvenez pas à obtenir une présentation, le fichier a peut-être été supprimé par l’éditeur.

F I N

Plus de contenu associé

Transcription de la présentation

  1. Unicode Normalization Mark Davis www.macchiato.com

  2. Normalization • Uniqueness • two equivalent strings have precisely the same normalized form • Fast binary comparison,accurate digital signatures • Recommended for XML, JavaScript and other standards

  3. Canonical Equivalence • Fundamental equivalence • Indistinguishable to users, when correctly rendered • Includes • Combining sequences • Hangul • Singletons Ç C ¸ 가 ㄱ ㅏ Ω Ω

  4. Compatibility Equivalence • Formatting differences • Font variants (ℌ) • Breaking differences (-) • Cursive forms (ﻦ ﻨ ﻧ ﻥ) • Circled (⑪) • Width, size, rotated (ｶ ﹠ ︷) • Super/subscripts (₉ ⁹) • Squared characters (㌀) • Fractions (⅚) • Others (dž) ｶ カ ㎏ k g fi f i

  5. UTR #15:Unicode Normalization Forms

  6. Normalization Requirement • Uniqueness: two equivalent strings will have precisely the same normalized form • If two strings x and y are canonical equivalents, then C(x) = C(y) D(x) = D(y) • If two strings are compatibility equivalents, then KC(x) = KC(y) KD(x) = KD(y)

  7. Affected Characters • None of the forms affect text with only ASCII characters (U+0000 to U+007F) • None of the forms generate compability characters that were not in the source text. • Both KD and KC replace compatibility characters. • Both D and C maintain compatibility characters.

  8. Cautions: Decomposition • Requires decomposition mappings from the Unicode Character Database • Those decomposition mappings must be applied recursively • The string must be put into canonical order • Either Canonical or Compatibility

  9. Cautions: Composition • Decomposition required first! • Then canonical composition • Composition data: fixed at Unicode 3.0.0 • Some characters are excluded from composition • Form C and Form KC can still have combining characters! • Required for Indic, Arabic, Hebrew, &c.

  10. Caution: Both C & D • All normalization forms are not closed under string concatenation. Example: • NFC/D "…a◌̰" + "◌̀…" • Not Norm."…a◌̰◌̀…" • NFC "…à◌̰…" • NFD "…a◌̀◌̰…" • Exceptions easy to test for

  11. Composition Process • Decompose (D or KD) • Combine unblocked characters with the previous starter, if possible*

  12. Composition Exclusions • Script Specifics क + ◌̣ ⇏ क़ • Futures: G + ◌̣ ⇏ G̣ • Singletons* Ω ⇏ Ω • Non-starter sequences* ◌̈ + ◌́ ⇏ ◌̈́

  13. Legacy Encoding • Legacy text is ‘normalized’ if it maps 1:1 to normalized Unicode text • Legacy sets: • Prenormalized: e.g. ISO 8859-1 • Normalizable: e.g. ISO 2022 (ISO 5426/ISO 8859-1/…) • Unnormalizable: e.g. ISO 5426

  14. Programming Identifiers • Closed under all Normalization Forms, if minor changes incorporated • Modified syntax: • identifier := start ( start | extend )* • start := [{Lu}{Ll}{Lt}{Lm}{Lo}{Nl}]- irregulars – combining_like • extend := [{Mn}{Mc}{Nd}{Pc}{Cf}]- irregulars + combining_like + mid_dot • (Almost) closed under Case Mappings • see SpecialCasing.txt

  15. Resources • Reference version on Unicode Site • Production Version • http://oss.software.ibm.com/icu • ICU: C/C++ and Java Versions • Open Source, with IBM Public License • Free commercial use and distribution: Not Viral! • Panel Later today • Other companies also providing: ask!

  16. Normalization • Uniqueness: two equivalent strings have precisely the same normalized form • Fast binary comparison, accurate digital signatures • Recommended for XML, JavaScript and other standards

  17. Q & A

  18. Backup Slides

  19. Definition: Starter • S is a starter = • Canonical class of zero in the Unicode Character Database • Can start a composition • Examples: Starters: Spacing marks, some non-spacing ‘a’, ‘ق’ ‘Θ’ ‘क’ ‘ी’ ‘◌ै’ Non-starters: most non-spacing marks ‘◌̀’, ‘◌̊’ ‘◌̽’ ‘◌̥’

  20. Definition: Blocked • C is blocked from S • There is some character B between S and C, and either • B is a starter or • B has the same canonical class as C • Examples • “ABC” – B blocks C from A • “A◌̀◌̊” – ◌̀ blocks ◌̊ from A • “A◌̥◌̊” – ◌̥doesn’t block ◌̊ from A

  21. Testing Conformance: Canonical

  22. Unicode Normalization • Introduction • Normalization forms • Design goals • Specification • Excluded characters • Versions • Legacy encodings • Applications

  23. Characters and Encoding Forms Abstract Encoded Serialized UTF-16BE UTF-8 C5 00 C5 C3 85 212B 21 2B E2 84 AB Å F0000 DB 80 DC 00 F3 B0 80 80 00 61 03 0A 61 CC 8A A 61 30A °