1 / 9

Understanding Channel Capacity and Mutual Information in Information Theory

This chapter explores the concepts of channel capacity and mutual information in the context of information theory. It discusses how the conditional probabilities of symbols received from a noisy channel influence the information conveyed. Key topics include Bayes' Theorem, the characterization of stationary channels, entropy measures, and the implications of noise on information gain. The chapter also delves into the relationship between input and output entropies, shared mutual information, and the significance of independence between source symbols. Through mathematical models and theorems, it elucidates the fundamentals of error-correcting codes and information loss.

slade
Télécharger la présentation

Understanding Channel Capacity and Mutual Information in Information Theory

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Chapter 7 The Channel and Mutual Information

  2. symbols can’t be swallowed a1:: aq b1:: bs A B alphabet of symbols sent alphabet of symbols received P(bj|ai) or randomly generated Information Through a Channel For example, in an error correcting code over a noisy channel, s≥ q. If two symbols sent are indistinguishable when received, s < q. Characterize a stationary channel by a matrix of conditional probabilities: received row column sent s Pi,j P = Pi,j = P(bj | ai) q 7.1, 7.2, 7.3

  3. For p(ai) = probability of source symbols, let p(bj) = probability of being received [p(a1) … p(aq)]P = [p(b1) … p(bs)] no noise: Pi,j = I; p(bj) = p(aj) all noise: Pi,j = 1/s; p(bj) = 1/s The probability that ai was sent and bj was received is: Baye’s Theorem P(ai, bj) = p(ai) ∙ P(bj | ai) = p(bj) ∙ P(ai | bj). [coincidental probability] So if p(bj) ≠ 0, the backwards conditional probabilities are: 7.1, 7.2, 7.3

  4. Binary symmetric Channel P0,0 a = 0 b = 0 p(a = 0) = p p(a = 1) = 1 − p P0,0 = P1,1 = P P0,1 = P1,0 = Q P0,1 P1,0 P1,1 a = 1 b = 1 a = 0 a = 1 p(b = 0) p(b = 1) (p∙ P + (1 − p) ∙ Qp ∙ Q + (1 − p) ∙ P) P Q Q P (p 1−p) = 7.4

  5. P = 1 Q = 0 P = Q = ½ If p = 1 − p = ½ (equiprobable) then: P(a = 1 | b = 0) = P(a = 0 | b = 1) = Q P(a = 0 | b = 0) = P(a = 1 | b = 1) = P P Q Q P 7.4

  6. H(A| B) H(B| A) H(A) Input entropy H(B) Output entropy System Entropies condition on bj : average over all bj : This is the information loss in the channel, called “equivocation” (also called “noise entropy”) Similarly : 7.5

  7. H(A, B) Joint Entropy Intuition: taking snapshots A B Define : H(A| B) H(B| A) H(B | A) H(A) H(A | B) H(A, B) = H(B) 7.5

  8. H(A, B) A priori A posteriori joint H(A| B) H(B| A) P(ai | bj) p(ai) I(A; B) The amount of information they are sharing corresponds to Information gain upon receiving bj : I(ai) − I(ai | bj) . shared Mutual Information By symmetry: If ai and bj are independent (all noise), then P(ai ; bj) = p(ai) ∙ p(bj) and hence P(ai | bj) = p(ai)  I(ai ; bj) = 0. No information gained in channel. 7.6

  9. from symmetry Similarly: Average over all ai: By Gibbs I(A; B) ≥ 0. Equality only if P(ai, bj) = p(ai)∙p(bj) [independence]. = H(A) + H(B) − H(A, B) ≥ 0  H(A, B)  H(A) + H(B) We know H(A, B) = H(A) + H(B | A) = H(B) + H(A | B).  I(A ; B) = H(A) − H(A | B) = H(B) − H(B | A) ≥ 0  H(A | B)  H(A) and H(B | A)  H(B). 7.6

More Related