Uniquely decodable and parseable codes Instanteneous and prefix codes Kraft-Mc-Millan inequality Huffman coding: a proce

1. Uniquely decodable and parseable codes • Instanteneous and prefix codes • Kraft-Mc-Millan inequality • Huffman coding: a procedure for designing an optimum code • Improving bit rate beyond the entropy limit Dr.E.Regentova

2. Average code length Let N is a no raining and R is a raining event in Las Vegas. Let p(N) calculated over a long period is 0.8 and p(R)=0.2. To decode such a source, we need = -0.8 log2(0.8)-0.2log20.2=0.7458 bits Dr.E.Regentova

3. Can we improve rates by joint coding? Average length is 1.2 b/input sequence , that is, 0.4 b/pixels, and the codewords are unique. Problem : Suppose 0101 is received. It can be decoded as NNNNNRNNNNR or NRNNRN, or else. Dr.E.Regentova

4. Parseable codes • Before a codeword can be decoded, it must be parsed. • Parsing describes the activity of breaking the message string into its component codewords. • After parsing, each codeword can be decoded into its symbol sequence. Dr.E.Regentova

5. Example of Parseable Code 1.6 bits per sequence, or 0.53 bits/symbol. This is 47% improvement if compare to 1.2 b/s. Suppose, 101100011010 is received. The only decoded sequence output is NNNNNRRRNNRN. Dr.E.Regentova

6. Compare three codes Dr.E.Regentova

7. Code 1 Binary tree Code is not parseable If 00 is received, there is no way to recover either a1a1 or a3 only is sent. a3 is a descendent of a1, and a4 is a descendent of a2. a1 is a prefix for a3 and a2 is a prefix of a4 Dr.E.Regentova

8. Code 2 Binary tree Code is parseable It terminates at 0, or at 111 None of codewords serve as prefix for any other. Dr.E.Regentova

9. Instantaneous codes An instantaneously parseable code is one that can be parsed as soon as the last bit of a codeword is received. An instantaneous code must satisfy the prefix condition. That is, no codeword may be a prefix of any other codeword. Dr.E.Regentova

10. Binary tree: Code 3 Code is parseable, but not instantaneous Accumulate at 0. The bit before 0 is the last bit of a previous word. Thus, we have to wait until the next symbol is received Dr.E.Regentova

11. Prefix code: external nodes only are codewords Internal nodes: give rise to other nodes External nodes: Do not give rise to other nodes Dr.E.Regentova

12. Code 2: Prefix code, i.e. none of nodes is a prefix to other nodes Dr.E.Regentova

13. Kraft-Mc-Millan inequality: part a) Let C be a code with N codewords with lengths l1,l2,…lN, and l1≤l2≤…≤lN. If C is uniquely decodable, then : * Dr.E.Regentova

14. Kraft-Mc-Millan inequality- part b) If we have a sequence of positive integers l1,l2,…lN, and l1 ≤ l2 ≤…≤ lN that satisfy inequality (*), then there exists a uniquely decodable code whose codewords lengths are given by the sequence l1,l2,…lN, Dr.E.Regentova

15. Efficiency and optimality A measure of the efficiency of the code is its redundancy- the difference between the entropy and the average length. The optimum code is one with a minimum redundancy Desired property: minimum variance Dr.E.Regentova