1 / 34

exercise in the previous class

prob. 0.363 0.174 0.143 0.098 0.087 0.069 0.045 0.021. 0. A B C D E F G H. 1. exercise in the previous class. binary Huffman code? average codeword length?. 0.363 ×1+0.174 ×3+...+0.021×5= 2.660. 1.000. 0.637. 0.359. 0.278. 0.185. 0.135. 0.066. 0.363 A. 0.174 B.

halden
Télécharger la présentation

exercise in the previous class

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. prob. 0.363 0.174 0.143 0.098 0.087 0.069 0.045 0.021 0 A B C D E F G H 1 exercise in the previous class • binary Huffman code? • average codeword length? 0.363×1+0.174×3+...+0.021×5=2.660 1.000 0.637 0.359 0.278 0.185 0.135 0.066 0.363 A 0.174 B 0.143 C 0.098 D 0.087 E 0.069 F 0.045 G 0.021 H 0 100 110 1010 1011 1110 11110 11111

  2. prob. 0.363 0.174 0.143 0.098 0.087 0.069 0.045 0.021 A B C D E F G H exercise in the previous class • 4-ary Huffman code? [basic idea] join four trees • we may have #trees < 4 in the final round. • with one “join”, 4 – 1 = 3 trees disappear. • add dummy nodes, start with 3k+1 nodes. ? dummy 1.000 d 0.320 a 0.066 b c 0.363 A 0.174 B 0.143 C 0.098 D 0.087 E 0.069 F 0.045 G 0.021 H 0 * 0 * a b c da db dc dda ddb

  3. today’s class • basic properties needed for source coding • uniquely decodable • immediately decodable • Huffman code • construction of Huffman code • extensions of Huffman code • theoretical limit of the “compression” • related topics today

  4. today’s class (detail) • Huffman codes are good, but how good are they? • Huffman codes for extended information sources • possible means (手段) to improve the efficiency • Shannon’s source coding theorem • the theoretical limit of efficiency • some more variations of Huffman codes • blocks of symbols with variable block length math. algorithm math. algorithm

  5. 0 0 0 1 1 1 how should we evaluate Huffman codes? good code • immediately decodable...“use code trees” • small average codewordlength (ACL)  It seems that Huffman’s algorithm gives a good solution. To see that Huffman codes are really good, we discuss a mathematical limit of the ACL • ...under a certain assumption (up to the slide 11) • ...in the general case (Shannon’s theorem)

  6. theoretical limit under an assumption assumption • the encoding is done in a symbol-by-symbol manner • define one codeword for each symbol of the source S • S produces M symbols with probabilities p1, ..., pM Lemma (restricted Shannon’s theorem): • for any code, the ACL ≥ H1(S) • a code with ACL ≤H1(S)+1 is constructible H1(S) is the borderline of “possible” and “impossible”.

  7. Shannon’s lemma (bad naming...) To prove the restricted Shannon’s theorem a small technical lemma (Shannon’s lemma) is needed. Shannon’s lemma (シャノンの補助定理） For any non-negative numbers q1, ..., qM with q1 +...+ qM≤ 1, with the equation holds if and only if pi = qi. remind: p1, ..., pM are symbol probabilities and p1 +...+ pM= 1

  8. proof (sketch) left hand side – right hand side= y = – logex 1 O y = 1 – x the equation holds iffqi/pi = 1

  9. proof of the restricted Shannon’s theorem: 1 for any code, the average codeword length ≥ H1(S) Let l1, ..., lM be the length of codewords, and define . • Kraft: • Shannon’s Lemma: • the ACL • We have shown that L ≥ H1(S).

  10. proof of the restricted Shannon’s theorem: 2 a code with average codeword length ≤H1(S)+1 is constructible Choose integers l1, ..., lM so that . • The choice makes , and ... Kraft’s inequality • We can construct a code with codeword length l1, ..., lM, whose ACL is

  11. the lemma and the Huffman code Lemma (restricted Shannon’s theorem): • for any code, the ACL ≥ H1(S) • a code with ACL ≤H1(S)+1 is constructible We can show that, for a Huffman code, • L≤H1(S) + 1 • there is no symbol-by-symbol code whose ACL is smaller than L. proof ... by recursion on the size of code trees • A Huffman code is said to be a compact code.

  12. symbol A B average prob. 0.8 0.2 C1 0 1 1.0 C2 1 0 1.0 A A A B C C 10 11 11 0 0 0 coding for extended information sources The Huffman code is the best symbol-by-symbol code, but... • the ACL  1 • not good for encoding binary information sources If we encode severalsymbols in a block, then... • the ACL per symbolcan be < 1 • good for binary sources also A B A C C A 10 110 01

  13. block Huffman coding message ABCBCBBCAA... • fixed-length (equal-, constant-) • variable-length (unequal-) • block partition • run-length “block” operation blocked message ABCBCBBCAA... Huffman encoding codewords 01 10 001 1101...

  14. fixed-length block Huffman coding • ACL: 0.6×1+0.3×2+ 0.1×2 = 1.4 bit for one symbol prob. 0.6 0.3 0.1 codeword 0 10 11 A B C blocks with two symbols • ACL: 0.36×1+ ... + 0.01×6 = 2.67 bit, but this is for two symbols • 2.67 / 2 = 1.335 bit for one symbol prob. 0.36 0.18 0.06 0.18 0.09 0.03 0.06 0.03 0.01 codeword 0 100 1100 101 1110 11110 1101 111110 111111 AA AB AC BA BB BC CA CB CC improved!

  15. block coding for binary sources • ACL: 0.8×1+ 0.2×1 = 1.0 bit for one symbol prob. 0.8 0.2 codeword 0 1 A B blocks with two symbols • ACL: 0.64×1+ ... + 0.04×3 = 1.56 bit for two symbols • 1.56 / 2 = 0.78 bit for one symbol prob. 0.64 0.16 0.16 0.04 codeword 0 10 110 111 AA AB BA BB improved!

  16. block size 1 2 3 : ACL per symbol 1.0 0.78 0.728 : the block length prob. 0.512 0.128 0.128 0.032 0.128 0.032 0.032 0.008 codeword 0 100 101 11100 110 11101 11110 11111 blocks with three symbols • ACL: 0.512×1+ ... + 0.008×5 = 2.184 bit for three symbols • 2.184 / 3 = 0.728 bit for one symbol AAA AAB ABA ABB BAA BAB BBA BBB larger block size  more compact

  17. block code and extension of information source What happens if we increase the block length further? Observe that... • a block code defines a codewordfor each block pattern. • one block = a sequence of n symbols of S = one symbol of Sn, the n-th order extension of S  restricted Shannon’s theorem is applicable: H1(Sn) ≤ Ln < H1(Sn) + 1 • Ln = the ACL for n symbols • for one symbol of S,

  18. Shannon’s source coding theorem • H1(Sn) / n ... the n-th order entropy of S (→ Apr. 12) • If n goes to the infinity... Shannon’s source coding theorem: for any code, the ACL ≥ H(S) a code with ACL ≤H(S) + εis constructible

  19. what the theorem means • Shannon’s source coding theorem: • for any code, the ACL ≥ H(S) • a code with ACL ≤H(S) + ε is constructible • Use block Huffman codes, and you can approach to the limit. • You never overcome the limit. however. prob. 0.8 0.2 block size 1 2 3 : ACLper symbol 1.0 0.78 0.728 : 0.723 + ε A B H(S) = 0.723

  20. remark 1 Why block codes give smaller ACL? • fact 1: the ACL is minimized by a real-numbersolution • if P(A) = 0.8, P(B) = 0.2, then we want l1 and l2 with... s.t. • fact 2: the length of a codeword must be an integer ...loss! ...gain! s.t. and integers frequent loss, seldom gain...

  21. remark 1 (cnt’d) • the gap between the ideal and the realcodeword lengths: ... is an integer approximation of • the gap is weighted by the probability… the weighted gap long block  many symbols  small probabilities  small weighted gaps  close to the ideal ACL p

  22. today’s class (detail) • Huffman codes are good, but how good are they? • Huffman codes for extended information sources • possible means (手段) to improve the efficiency • Shannon’s source coding theorem • the theoretical limit of efficiency • some more variations of Huffman codes • blocks of symbols with variable block length math. algorithm math. algorithm

  23. practical issues (問題) of block coding • Theoretically saying, the block Huffman codes are the best. • From practical viewpoint, there are several problems: • We need to know the probability distribution in advance. (this will be discussed in the next class) • We need a large table for the encoding/decoding. • if one byte is needed to record one entry of the table... • 256 byte table, if block length = 8 • 64 Kbyte table, if block length = 16 • 4 Gbytetable, if block length = 32

  24. use blocks with variable-length prob. 0.512 0.128 0.128 0.032 0.128 0.032 0.032 0.008 codeword 0 100 101 11100 110 11101 11110 11111 If we define blocks so that they have the same length, then ... • some blocks have small probabilities • those blocks also need codewords If we define blocks so that they have similar probabilities, then ... • length differ from block by block • the table has little useless blocks AAA AAB ABA ABB BAA BAB BBA BBB prob. 0.512 0.128 0.16 0.2 codeword 0 100 101 11 AAA AAB AB B

  25. definition of block patterns Block patterns must be defined so that... the patterns can represent (almost) all symbol sequences. • bad example: block pattern = {AAA, AAB, AB} AABABAAB AAB AB AAB AABBBAAB AAB ? two different approaches are well-known; • block partition approach • run-length approach

  26. AA AB B 0.64 0.16 0.2 A B 0.8 0.2 define patterns with block partition approach • prepare all blocks with length one • partition the block with the largest probability by appending one more symbol • go to 2 Example: P(A) = 0.8, P(B) = 0.2 codewords AAA AAB AB B 0.512 0.128 0.16 0.2 0 100 101 11

  27. how good is this? to determine the average codeword length, assume that n blocks are produced from S: AAA AAB AB B 0.512 0.128 0.16 0.2 0 100 101 11 S AAA AB AAA B AB ... 0 101 0 11 101 ... encode • 0.512n×1 + 0.128n×3 ... • = 1.776n bits • 0.512n×3 + 0.128n×3 ... • = 2.44n symbols 2.44n symbols are encoded to 1.776n bits  the average codewordlength is 1.776n / 2.44n = 0.728 bit (almost the same as the block length = 8, p. 16, but small table)

  28. define patterns with run-length approach run = a sequence of consecutive (連続の) identical symbol Example: divide a message into runs of “A”: A B B A A A A A B A A A B run of length = 3 run of length = 1 run of length = 0 run of length =5 The message is constructible if the lengths of runs are given.  define blocks as runs of various length

  29. upper-bound the run-length small problem? ... there can be very long run  put an upper-bound limit : run-length limited (RLL) coding upper-bound = 3 • ABBAAAAABAAAB is represented as • one “A” followed by B • zero“A” followed by B • three or more “A”s followed by B • two“A”s followed by B • three or more “A”s followed by B • zero “A” followed by B run length 0 1 2 3 4 5 6 7 : representation 0 1 2 3+0 3+1 3+1 3+3+0 3+3+1 :

  30. run-length Huffman code • Huffman code defined to encode the length or runs • effective when there is strong bias on the symbol probabilities p(A) = 0.9, p(B) = 0.1 run length 0 1 2 3 or more block pattern B AB AAB AAA prob. 0.1 0.09 0.081 0.729 codeword 10 110 111 0 • ABBAAAAABAAAB: 1, 0, 3+, 2, 3+, 0 ⇒ 110 10 0 111 0 10 • AAAABAAAAABAAB: 3+, 1, 3+, 2, 2 ⇒ 0 110 0 111 111 • AAABAAAAAAAAB: 3+, 0, 3+, 3+, 2 ⇒ 0 10 0 0 111

  31. example of various block coding • S: memoryless & stationary. P(A) = 0.9, p(B) = 0.1 • the entropy of S is H(S) = –0.9log20.9 – 0.1log20.1=0.469 bit symbol A B prob. 0.9 0.1 codeword 0 1 • code 1: a naive Huffman code average codeword length = 1 • code 2: fixed-length (3bit) average codeword length = 1.661/3symbols = 0.55/symbol AAA AAB ABA ABB 0.729 0.081 0.081 0.009 0 100 110 1010 BAA BAB BBA BBB 0.081 0.009 0.009 0.009 1110 1011 11110 11111

  32. example of various block coding (cnt’d) with n blocks... • 0.1n×1 + ... + 0.478n×7 = 5.215n symbols • 0.1n×3 + ... + 0.478n×1 = 2.466n bits the average codeword length per symbol = 2.466 / 5.215 = 0.47 • code 3: run-length Huffman (upper-bound = 8) length 0 1 2 3 prob. 0.1 0.09 0.081 0.073 codeword 110 1000 1001 1010 length 4 5 6 7+ prob. 0.066 0.059 0.053 0.478 codeword 1011 1110 1111 0

  33. summary of today’s class • Huffman codes are good, but how good are they? • Huffman codes for extended information sources • possible means (手段) to improve the efficiency • Shannon’s source coding theorem • the theoretical limit of efficiency • some more variations of Huffman codes • blocks of symbols with variable block length

  34. exercise • Write a computer program to construct a Huffman code for a given probability distribution. • Modify the above program so that it can handle fixed-length block coding. • Give distribution, change the block length, and observe how the average codeword length changes according to the change.

More Related