Information Content

1. Information Content Tristan L’Ecuyer

2. Claude Shannon (1948), “A Mathematical Theory of Communication”, Bell System Technical Journal 27, pp. 379-423 and 623-656. Historical Perspective • Information theory has its roots in telecommunications and specifically in addressing the engineering problem of transmitting signals over noisy channels. • Papers in 1924 and 1928 by Harry Nyquist and Ralph Hartley, respectively introduce the notion of information as a measurable quantity representing the ability of a receiver to distinguish different sequences of symbols. • The formal theory begins with Shannon (1948), the first to establish the connection between information content and entropy. • Since this seminal work, information theory has grown into a broad and deep mathematical field with applications in data communication, data compression, error-correction, and cryptographic algorithms (codes and ciphers).

3. Link to Remote Sensing • Shannon (1948): “The fundamental problem of communication is that of reproducing at one point, either exactly or approximately, a message selected at another point.” • Similarly, the fundamental goal of remote sensing is to use measurements to reproduce a set of geophysical parameters, the “message”, that are defined or “selected” in the atmosphere at the remote point of observation (eg. satellite). • Information theory makes it possible examine the capacity of transmission channels (usually in bits) accounting for noise, signal gaps, and other forms of signal degradation. • Likewise in remote sensing we can use information theory to examine the “capacity” of a combination of measurements to convey information about the geophysical parameters of interest accounting for “noise” due to measurement error and model error.

4. Unwanted Solutions ∆y ∆x ∆x <∆x ∆x <∆x <∆x ∆x Corrupting the Message:Noise and Non-uniqueness • Measurement and model error as well as the character of the forward model all introduce non-uniqueness in the solution. Linear Model Quadratic Model Cubic Model

5. Forward Model Errors (∆y) • Uncertainty due to unknown “influence parameters” that impact forward model calculations but are not directly retrieved often represents the largest source of retrieval error • Errors in these parameters introduce non-uniqueness in the solution space by broadening the effective measurement PDF Forward Problem Inverse Problem Errors in Inversion “Influence” parameters Uncertainty in “influence” parameters Measurement error Forward model errors

6. Error in product from width of posterior distribution from application of Bayes theorem. σReff στ Reff R2.13μm σ∆TB Obs. Soln σTB R0.64μm τ Error Propagation in Inversion Bi-variate PDF of (sim. – obs.) measurements. Width dictated by measurement error and uncertainty in forward model assumptions.

7. τ = 45±5; Re = 11±2 τ = 18±2; Re = 19±2 τ = 10 20 30 50 8 μm 12 μm 2.13 μm Reflectance 24 μm 48 μm τ = 2 τ = 10 20 30 50 τ = 2 0.66 μm Reflectance Visible Ice Cloud Retreivals • Nakajima and King (1990) technique based on a conservative scattering visible channel for optical depth and an absorbing near- IR channel for reff • Influence parameters are crystal habit, particle size distribution, and surface albedo. Due to assumptions: τ = 16-50; Re = 9-21

8. CloudSat Snowfall Retrievals • Snowfall retrievals relate reflectivity, Z, to snowfall rate, S • This relationship depends on snow crystal shape, density, size distribution, and fall speed • Since few, if any of these factors can be retrieved from reflectivity alone, they all broaden the Z-S relationship and lead to uncertainty in the retrieved snowfall rate

9. Hex Columns 4-arm Rosettes 6-arm Rosettes 8-arm Rosettes Reflectivity (dBZe) Snowfall Rate (mm h-1) Impacts of Crystal Shape (2-7 dBZ)

10. Sensitivity to ν Sensitivity to PSD Shape ν = 0 ν = 1 ν = 2 Sekhon/Srivastava a & b = -10% a & b = +10% Reflectivity (dBZe) Reflectivity (dBZe) Snowfall Rate (mm h-1) Snowfall Rate (mm h-1) Impacts of PSD (3-6 dBZ)

11. Ideal Case “Reality” Reflectivity Reflectivity Snowfall Rate (mm h-1) Snowfall Rate (mm h-1) • PSD and snow crystal shape, however, spread the range of allowable solutions in the absence of additional constraint Implications for Retrieval • Given a “perfect” forward model, 1 dB measurement errors lead to errors in retrieved snowfall rate of less than 10 %

12. Quantitative Retrieval Metrics • Four useful metrics for assessing how well formulated a retrieval problem: • Sx – the error covariance matrix provides a useful diagnostic of retrieval performance measuring the uncertainty in the products • A – the averaging kernel describes, among other things, the amount of information that comes from the measurements as opposed to a priori information • Degrees of freedom • Information content • All require accurate specification of uncertainties in all inputs including errors due to forward model assumptions, measurements, and any mathematical approximations required to map geophysical parameters into measurement space.

13. Clive Rogers (2000), “Inverse Methods for Atmospheric Sounding: Theory and Practice”, World Scientific, 238 pp. Degrees of Freedom • The cost function can be used to define two very useful measures of the quality of a retrieval: the number of degrees of freedom for signal and noise denoted ds and dn, respectively where Sais the covariance matrix describing the prior state space and K represents the Jacobian of the measurements with respect to the parameters of interest. • ds specifies the number of observations that are actually used to constrain retrieval parameters while the dn is the corresponding number that are lost due to noise ds dn

14. Degrees of Freedom • Using the expression for the state vector that minimizes the cost function it is relatively straight-forward to show that where Im is the m x m identity matrix and A is the averaging kernel. • NOTE: Even if the number of retrieval parameters is equal to or less than the number of measurements, a retrieval can still be under-constrained if noise and redundancy are such that the number of degrees of freedom for signal is less than the number of parameters to be retrieved.

15. Entropy-based Information Content • The Gibbs entropy is the logarithm of the number of discrete internal states of a thermodynamic system where pi is the probability of the system being in state i and k is the Boltzmann constant. • The information theory analogue has k=1 and the pi representing the probabilities of all possible combinations of retrieval parameters. • More generally, for a continuous distribution (eg. Gaussian):

16. Entropy of a Gaussian Distribution • For the Gaussian distributions typically used in optimal estimation we have: • For an m-variable Gaussian dist.:

17. Information Content of a Retrieval • The information content of an observing system is defined as the difference in entropy between an a priori set of possible solutions, S(P1), and the subset of these solutions that also satisfy the measurements, S(P2): • If Gaussian distributions are assumed for the prior and posterior state spaces as in the O. E. approach, this can be written: since, after minimizing the cost function, the covariance of the posterior state space is:

18. Interpretation • Qualitatively, information content describes the factor by which knowledge of a quantity is improved by making a measurement. • Using Gaussian statistics we see that the information content provides a measure of how much the ‘volume of uncertainty’ represented by the a priori state space is reduced after measurements are made. • Essentially this is a generalization of the scalar concept of ‘signal-to-noise’ ratio.

19. B: Next finer scale = 4 divisions  H = 2 • C: Finer still = 8 divisions  H = 3 • D: Finest scale = 16 divisions  H = 4 D C B Measuring Stick Analogy • Information content measures the resolution of the observing system for resolving solution space. • Analogous to the divisions on a measuring stick: the higher the information content, the finer the scale that can be resolved. • A: Biggest scale = 2 divisions  H = 1 A Full range of a priori solutions

20. Prior State Space 0.64 μm (H=1.20) LWP (gm-3) 0.64 & 2.13 μm (H=2.51) 17 Channels (H=3.53) LWP (gm-3) Re (μm) Re (μm) Liquid Cloud Retrievals • Blue  a priori state space • Green  state space that also matches MODIS visible channel (0.64 μm) • Red  state space that matches both 0.64 and 2.13 μm channels • Yellow  state space that matches all 17 MODIS channels

21. Radar Only Radar + Radiometer Reflectivity Reflectivity Snowfall Rate (mm h-1) Snowfall Rate (mm h-1) Snowfall Retrieval Revisited • With a 140 GHz brightness temperature accurate to ±5 K as a constraint, the range of solutions is significantly narrowed by up to a factor of 4 implying an information content of ~2.

22. 2 5 1 Return to Polynomial Functions X1 = X2 = 2; X1a = X2a = 1 σy = 10% σa = 100% σy = 25% σa = 100% σy = 10% σa = 10%

23. L’Ecuyer et al. (2006), J. Appl. Meteor. 45, 20-41. Cooper et al. (2006), J. Appl. Meteor. 45, 42-62. Application: MODIS Cloud Retrievals • The concept of information content provides a useful tool for analyzing the properties of observing systems within the constraints of realistic error assumptions. • As an example, consider the problem of assessing the information content of the channels on the MODIS instrument for retrieving cloud microphysical properties. • Application of information theory requires: • Characterize the expected uncertainty in modeled radiances due to assumed temperature, humidity, ice crystal shape/density, particle size distribution, etc. (i.e. evaluate Sy); • Determine the sensitivity of each radiance to the microphysical properties of interest (i.e. compute K); • Establish error bounds provided by any available a priori information (eg. cloud height from CloudSat); • Evaluate diagnostics such as Sx, A, ds, and H

24. Error Analyses • Fractional errors reveal a strong scene-dependence that varies from channel to channel. • LW channels are typically better at lower optical depths while SW channels improve at higher values.

25. Sensitivity Analyses • The sensitivity matrices also illustrate a strong scene dependence that varies from channel to channel. • The SW channels have the best sensitivity to number concentration in optically thick clouds and effective radius in thin clouds. • LW channels exhibit the most sensitivity to cloud height for thick clouds and to number concentration for clouds with optical depths between 0.5-4. 0.646 μm 2.130 μm 11.00 μm

26. 9 km 11 km 14 km H 9 km 11 km 14 km ds Information Content • Information content is related to the ratio of the sensitivity to the uncertainty – i.e. the signal-to-noise.

27. Uniform 10% Errors Rigorous Errors 11 km 11 km 11 km 11 km The Importance of Uncertainties • Rigorous specification of forward model uncertainties is critical for an accurate assessment of the information content of any set of measurements.

28. With CloudSat Without CloudSat 11 km 11 km 11 km 11 km The Role of A Priori • Information content measures the amount state space is reduced relative to prior information. • As prior information improves, the information content of the measurements decreases. • The presence of cloud height information from CloudSat, for example, constrains the a priori state space and reduces the information content of the MODIS observations.