The Role of Specialization in LDPC Codes

1. The Role of Specialization in LDPC Codes Jeremy Thorpe Pizza Meeting Talk 2/12/03

2. Talk Overview • LDPC Codes • Message Passing Decoding • Analysis of Message Passing Decoding (Density Evolution) • Approximations to Density Evolution • Design of LDPC Codes using D.E.

3. The Channel Coding Strategy • Encoder chooses the mth codeword in codebook C and transmits it across the channel • Decoder observes the channel output y and generates m’ based on the knowledge of the codebook C and the channel statistics. Encoder Channel Decoder

4. Linear Codes • A linear code C (over a finite field) can be defined in terms of either a generator matrix or parity-check matrix. • Generator matrix G (k×n) • Parity-check matrix H (n-k×n)

5. LDPC Codes • LDPC Codes -- linear codes defined in terms of H. • H has a small average number of non-zero elements per row

6. H is represented by a bipartite graph. There is an edge from v to c if and only if: A codeword is an assignment of v's s.t.: Graph Representation of LDPC Codes Variable nodes . . . . . . Check nodes

7. Every variable node has degree λ, every check node has degree ρ. Ensemble is defined by matching left edge "stubs" with right edge "stubs via a random permutation Regular (λ,ρ) LDPC codes Variable nodes π . . . . . . Check nodes

8. Message-Passing Decoding of LDPC Codes • Message Passing (or Belief Propagation) decoding is a low-complexity algorithm which approximately answers the question “what is the most likely x given y?” • MP recursively defines messages mv,c(i) and mc,v(i) from each node variable node v to each adjacent check node c, for iteration i=0,1,...

9. Liklihood Ratio For y1,...yn independent conditionally on x: Probability Difference For x1,...xn independent: Two Types of Messages...

10. Definition: Properties: ...Related by the Biliniear Transform

11. Message Domains Probability Difference Likelihood Ratio Log Prob. Difference Log Likelihood Ratio

12. On any iteration i, the message from v to c is: It is computed like: Variable to Check Messages v c . . . . . .

13. Check to Variable Messages • On any iteration, the message from c to v is: • It is computed like: • Assumption: Incoming messages are indep. v c . . . . . .

14. Decision Rule • After sufficiently many iterations, return the likelihood ratio:

15. Theorem about MP Algorithm • If the algorithm stops after r iterations, then the algorithm returns the maximum a posteriori probability estimate of xvgiven y within radius r of v. • However, the variables within a radius r of v must be dependent only by the equations within radius r of v, r ... v ... ...

16. Analysis of Message Passing Decoding (Density Evolution) • in Density Evolution we keep track of message densities, rather than the densities themselves. • At each iteration, we average over all of the edges which are connected by a permutation. • We assume that the all-zeros codeword was transmitted (which requires that the channel be symmetric).

17. D.E. Update Rule • The update rule for Density Evolution is defined in the additive domain of each type of node. • Whereas in B.P, we add (log) messages: • In D.E, we convolve message densities:

18. Familiar Example: • If one die has density function given by: • The density function for the sum of two dice is given by the convolution: 1 2 3 4 5 6 2 3 4 5 6 7 8 9 10 11 12

19. D.E. Threshold • Fixing the channel message densities, the message densities will either "converge" to minus infinity, or they won't. • For the gaussian channel, the smallest (infimum) SNR for which the densities converge is called the density evolution threshold.

20. Regular (λ,ρ) LDPC codes • Every variable node has degree λ, every check node has degree ρ. • Best rate 1/2 code is (3,6), with threshold 1.09 dB. • This code had been invented by1962 by Robert Gallager.

21. D.E. Simulation of (3,6) codes • Set SNR to 1.12 dB (.03 above threshold) • Watch fraction of "erroneous messages" from check to variable. • (note that this fraction does not characterize the distribution fully)

22. Improvement vs. current error fraction for Regular (3,6) • Improvement per iteration is plotted against current error fraction. • Note there is a single bottleneck which took most of the decoding iterations.

23. a fraction λi of variable nodes have degree i. ρi of check nodes have degree i. Edges are connected by a single random permutation. Nodes have become specialized. Irregular (λ, ρ) LDPC codes Variable nodes π λ2 ρ4 λ3 . . . . . . ρm λn Check nodes

24. D.E. Simulation of Irregular Codes (Maximum degree 10) • Set SNR to 0.42 dB (~.03 above threshold) • Watch fraction of erroneous check to variable messages. • This Code was designed by Richardson et. al.

25. Comparison of Regular and Irregular codes • Notice that the Irregular graph is much flatter. • Note: Capacity achieving LDPC codes for the erasure channel were designed by making this line exactly flat.

26. Multi-edge-type construction • Edges of a particular "color" are connected through a permutation. • Edges become specialized. Each edge type has a different message distribution each iteration.

27. D.E. of MET codes. • For Multi-edge-type codes, Density evolution tracks the density of each type of message separately. • Comparison was made to real decoding, next slide (courtesy of Ken Andrews).

28. MET D.E. vs. decoder simulation

29. Regular, Irregular, MET comparison • Multi-edge-type LDPC codes improve gradually through most of the decoding. • MET codes have a threshold below the more "complex" irregular codes.

30. Design of Codes using D.E. • Density evolution provides a moderately fast way to evaluate single- and multi- edge type degree distributions (hypothesis-testing). • Much faster approximations to Density Evolution can easily be put into an outer loop which performs function minimization. • Semi-Analytic techniques exist as well.

31. Review • Iterative Message Passing can be used to decode LDPC codes. • Density Evolution can be used to predict the performance of MP for infinitely large codes. • More sophisticated classes of codes can be designed to close the gap to the Shannon limit.

32. Beyond MET Codes (future work) • Traditional LDPC codes are designed in two stages: design of the degree distribution and design of the specific graph. • Using very fast and accurate approximations to density evolution, we can evaluate the effect of placing or removing a single edge in the graph. • Using an evolutionary algorithm like Simulated Annealing, we can optimize the graph directly, changing the degree profile as needed.