Scientific Decision Making in the Face of Uncertainty Tips, tricks and pitfalls in the

Scientific Decision Making in the Face of Uncertainty Tips, tricks and pitfalls in the optimum estimation of physical quantities Douglas Muir Kittery, Maine

Scientific Decision Makingin the Face of UncertaintyTips, tricks and pitfalls in the optimum estimation of physical quantities In the development of new technologies, the role formerly played by the construction and testing of engineering mockups is increasingly being played by computer simulations. The predictive power of computer simulation relies, in turn, on the availability of numerical databases containing accurate and complete scientific information describing the underlying physical processes. In meeting this need, data analysts routinely assemble the results of hundreds, or even thousands, of individual measurements and then systematically distill from this body of information best estimates of thetrue values of the required quantities. Here, we describe the work involved in performing this type of large-scale evaluation of scientific data. We also discuss the dissemination of evaluated data by centers such as the IAEA Nuclear Data Center.

Providing Answers to Questions of Great Social SignificanceExamples: How long can one prudently operate a fission reactor without replacing the pressure vessel? What is the rate of plasma cooling due to line radiation from ions of high-Z plasma impurities? What is the impact of human activity on global climate change, and what can be done to mitigate these impacts? What is the ultimate temperature limit of high Tc superconductors? Does country N have a clandestine nuclear weapons program? Can a sensor be developed to efficiently locate explosives and illicit drugs in airport luggage without physically opening the bags?

Answering complex questions like these requires expertise from many fields: chemistry, physics, electrical engineering, software engineering, etc.. Contributions are needed from theorists and experimentalists, and, especially, from "data evaluators" • "Data evaluation" refers to the relatively new profession of bulding databases of recommended numerical values of constants that describe natural processes. Data evaluators draw simultaneously upon their skills as a scientist, a statistician and a computer scientist. • "Data measurement" can be thought of as a special case of "data evaluation," the main difference being that a smaller set of measurements is considered, usually the author's own measurements. • Examples of the constants of interest include nuclear and atomic reaction data, nuclear and atomic spectrocopic data, chemical reaction data, strength and ductility of metal alloys, and many other kinds of material property data. • The arrival at a set of recommended data can be a lengthy process.

Specific Examples of DataBelow is a list of topics where better nuclear data is needed, according to a specialists' meeting convened in November 2000 by the International Atomic Energy Agency (IAEA) in Vienna, Austria • Medical Applications ... production cross sections of diagnostic and therapeutic radionuclides, data for internal radiation dosimetry, transport data for fast neutron and proton therapy • Ion Beam Analysis ... reaction and scattering cross sections of few-MeV charged particles for material surface analysis • Nuclear Astrophysics ... data for "exotic" charged particle reactions • Nuclear Safeguards and Inspection ... data for gamma spectroscopy of irradiated fuel, neutron interrogation, spontaneous neutron emission, non-intrusive inspection of airline luggage • Critical Reactors, including Closed Fuel Cycles ... Pb-cooled fast reactors, thorium utilization in heavy water reactors • Accelerator Driven Systems ... driven subcritical reactors, accelerator based transmutation of nuclear waste, spallation neutron sources

Disseminating the Data to the User CommunityA key player in the dissemination of nuclear data is the IAEA Nuclear Data Center in Vienna. It offers cost-free access, via the web and CD-ROM, to experimental and evaluated nuclear and atomic data, in close coordination with the US data center at Brookhaven and the Nuclear Energy Agency in Paris.

IAEA Nuclear Data Serviceswww-nds.iaea.org

IAEA Nuclear Data Services (continued)www-nds.iaea.org

The "Eight-Fold Way" to Better Data • Step 1. Study the relevant physics. • Step 2. Review the body of available experimental data. • Step 3. Fit the available measured data with a physically correct theory, or at least a physically reasonable phenomenological model. • Step 4. Test the data set and selected model(s) for internal consistency. Remove, where possible, the sources of inconsistency. • Step 5. Use statistical techniques, such as minimum variance estimation, to obtain recommended values for all needed quantities. • Step 6. Store the results in an agreed data-exchange format. • Step 7. Disseminate the results to the user community. • Step 8. Plan and perform experiments to expand the body of available measurements.

Experimental vs. Evaluated DataMeasured and evaluated data for nuclear reaction 27Al(n,a)24Na. The reaction product decays by b-decay with half-life of 15.02 hours and emits gamma rays of energy 1.369 and 2.754 MeV.

In Data Evaluation, Doing Good Physics is not a Luxury. It's a Necessity. In addition to satisfying our scientific curiosity, there are a number of important applied reasons for supplementing direct data measurements with information from physically correct theory. This is true even in fields such as nuclear physics, where the available theoretical models are inherently phenomenological. Such models provide valuable shape information, and usually good absolute accuracy can be achieved by adjusting certain free parameters to bring predictions of the model into agreement with accurate measurements. The primary benefit that results from the use of theoretical models is that they help in enforcing conservation laws and improving the completeness of coverage. In addition, when contradictory measurements are encountered, the model can often guide the evaluator in choosing between them. Finally, fitted model parameters often become "proxies" for more detailed microscopic data, and thereby reduce the size of the database.

Data Evaluation 101: Two Measurements, no Correlations Data evaluation is all about that finding optimum estimates. To illustrate what we mean by an "optimum estimate," suppose that two uncorrelated measurements of a given physical quantity have produced results m1 and m2with variances (mean squared errors) of v1 and v2. We construct an estimate of the true value of the quantity by taking a linear average of m1 and m2. m(f) = f m1 + (1 -f) m2 v(f) = f2 v1 + (1 -f)2 v2 We now determine the value of f that yields an estimate m(f) having minimum variance by differentiating v(f) with respect to f, setting this derivative to zero, and solving for fmv. This yields our desired estimate: mmv /vmv = m1 /v1 + m2 /v2 1/vmv = 1/v1 + 1/v2 Note the gain in "information" (reciprocal variance). Minimum variance estimates extract the maximum possible information from the input data.

Real World Data Evaluation: Very Large Data Sets and General Correlations The simplicity of the mathematics in our previous example was a consequence of the fact that there were only two measurements and they were assumed uncorrelated. Unfortunately, real life is rarely that simple. In most situations, the specific method used to measure physical quantities introduces significant measurement correlations. For example, if two different neutron cross sections are measured relative to the same standard, such as the n-p elastic scattering cross section, the uncertainty in the hydrogen cross section employed in the data reduction process will affect both measured values in a correlated way. Another way that correlations enter the picture is when the best available information is obtained from measurements of summed quantities. For example, total cross sections can usually be measured much more accurately than the constituent partial cross sections. This introduces large negative correlations in the uncertainty of the partials.

Mathematical Treatment of Uncertainty: Bringing Order out of Chaos To explore the issues that arise in the evaluation and application of data when correlations are present, it is helpful to begin with some basic definitions. We first introduce the notation (xi, i = 1, k) to indicate the results of the measurement (or evaluation) of k different physical quantities. In each case, the data are assumed to be perturbed from the true values of the physical quantities ci by random errors ei, xi = ci + ei . One doesn't actually know either the true values ci or the value of the errors ei, but the method used to perform the measurement or evaluation determines the expected size of the errors and information about the strength and sign of correlations.

Random Vectors and their Expectation Values It is convenient to consider the errors ei as elements of a random vectore, and the data values xi as elements of a random vectorx. The true values ci likewise form the elements of a constant vector c. We assume that the measurements are unbiased, so that the errors have zero mean, < ei > = 0 . Hence, < xi > = ci . The symbol < > denotes the "expectation" operator, which performs an average of each element of the bracketed quantity (which may be a scalar, a vector or a matrix) over the joint probability distribution of the errors. The terms "average value", "mean value", "expected value" and "true value" are used more-or-less interchangeably.

Data Covariances In vector notation, then, x = c + e , < e> = 0 , < x> = c . The features of the joint probability distribution that are of primary interest to us here are the second moments of this distribution. These are just the expectation values of each of the possible pair-wise products ei ej that can be formed from the elements of the error vector e. One normally refers to these second moments as data covariances. It is often convenient to consider the moments ei ej to be the elements of a data covariance matrix D(x), D(x) = < (x - < x >) (x - < x >)* > =< e e* > , where the * symbol indicates the matrix transpose.

Data Covariances (continued) Since e is a column vector of dimension k and e* is a row vector with the same elements, D(x) is a square matrix of dimension k. (For a helpful review of matrix multiplication and related operations, see Reference 2.) The diagonal elements [D(x)]iiof the covariance matrix are just the variances of the xi, equal to the square of the standard errorsDxi.The off-diagonal elements [D(x)]ij = < ei ej > of the covariance matrix quantifies the extent to which xi and xj are subject to a common source (or sources) of error, that is, the sign and strength of the correlations. We will also be interested cases where there is a second data vector y, not necessarily of the same length as x. We can define a rectangular "cross-covariance matrix" C(x, y) containing the uncertainties in x and y and their correlations as follows: C(x, y) = < (x - < x >) (y - < y >)* >

Uncertainties in Quantities of Applied Interest We consider now a set of quantities of applied interest, which we represent by the vector t. For the sake of simplicity, we shall limit the scope of our discussions to cases where the elements of t can be calculated from different linear combinations of the physical constants x. Thus there is a rectangular matrix of constants H, such that t = H x . Since x is a random vector, so is t. It is easy to show that < t > = H < x > . The covariance matrix of t can be obtained from the basic definition, D(t) = < (H x - < H x >) (H x - < H x >)* > = H < (x - < x >) (x - < x >)* > H* = H D(x) H*.

Assessing the Adequacy of Existing Data The so-called "sandwich rule," D(t) = H D(x) H*, is extremely useful in various types of data analysis. D(t) records both the uncertainties of the set of calculated quantities t and the correlations among these uncertainties. Probably the most important use of the covariances D(t) of applied quantities is to determine whether or not the existing data evaluation x meets the accuracy requirements of a particular simulation application. Such determinations are important, for example, in justifying new research to improve the data. As we shall see in the following slides, the availability of valid data covariances also provides the basis for an objective and efficient updating of the evaluation when the results of new measurements become available.

Introducing New Experimental Information There is an extensive literature on the subject of techniques for updating an existing data evaluation x. These updating techniques, variously described as "Bayesian", "Maximum Likelihood", "Maximum Entropy" or "Kalman Filter" methods, are operationally equivalent to minimum variance estimation. We label the new measurement vector as y. Let the number of new measurements be m, not necessarily equal to the number of evaluated data. We restrict ourselves to the case where the following linear model applies, < y > = R < x > . We note that the vector quantity R x contains values of the measured data quantities, as calculated or predicted from the previous evaluation of the data. Unlike the calculated quantity t discussed above, y is a measured quantity, so that, in general, y is not equal to R x . In fact, it is precisely the non-zero difference p = y - R x that contains the information we need to improve our knowledge of x.

Pitfall 1. Inconsistencies in the Network of Input Information Before actually using the new data y to improve the existing evaluation x, it is essential to test the network of available information for consistency. By "network", we mean the prior data x, the new data y, all of their respective covariances, and the model R. A distinguishing feature of the difference vector p is that (in sharp contrast to y and x) the true value of p is a known quantity. < p > = < y - R x > = < y > - R < x > = R < x > - R < x > = 0 This means we can compare the actual error in p, p - < p > with the expected error, as summarized in the covariance matrix D(p), which we will calculate shortly. The most useful measure of consistency is the specific matrix product c2 = p* Gp , where G-1 = D(p) . Note that c2 is a scalar quantity, a single number.

The Chi-Square Test Just why the expected value of c2 should be equal to the number of degrees of freedom is not totally obvious. Following Reference 1, we begin by recalling an interesting property of the trace (sum of diagonal elements) of a square matrix: If r and s are two arbitrary column vectors of equal length, then the scalar quantity s* r is equal to the trace of the square matrix rs*. This theorem facilitates a computation of the expected value of c2: < c2 > = < p* Gp > = < trace [pp* G] > = trace [< pp* G >] = trace [< pp* > G] = trace [D(p) G] = trace I(m,m) = m . Therefore, if c2 greatly exceeds m, the network of input information is shown to be inconsistent. Repairs should be performed before proceeding.

The Bottom Line: Better Data The cross covariance matrix relating p and x is C(x, p) = C(x, y) - C(x, Rx) = C(x, y) - D(x) R* . With a modest amount of additional matrix algebra, the covariance matrix of p can be obtained from other defined quantities as follows: D(p) = D(y) - R C(x, y) - [ R C(x, y) ]* + R D(x) R* . We recall that G denotes the inverse of the covariance matrix of p. G-1 = D(p) . Let x' indicate the improved estimate of the true value of x, taking into account both the previous evaluation of x and the information content of the new measurement y. As shown in Reference 3, the new estimate x' and the associated covariance matrix D(x') can be written very compactly in terms of the quantities just defined: x' = x - C(x, p) G p D(x') = D(x) - D[ C(x, p) Gp] = D(x) - C(x, p) G [C(x, p)]* .

In Summary ...To summarize, minimum variance estimation provides an objective and efficient way to take maximum advantage of new experimental information. It yields a new evaluation of both the data x and their covariances D(x). We can also list the following desirable features: The largest covariance matrix that needs to be inverted D(p) is of order of the number of new measurements y, independent of the number of data quantities x, so it is practical to project the effect of a small number of new measurements onto a very large existing database. No restrictions whatever are placed on the extent of measurement correlations. No assumptions are made regarding the shapes of the probability distributions of the errors. One can iterate the processs to produce a whole sequence of updates.

Pitfall 2. Distorting the evaluated result by making physically incorrect assumptions, e.g., "the shape of curve is well known."Pitfall 3. Ignoring correlations in measurement data.

Back to the Future: The Use of Data Covariances to Plan Future Measurements We recall the result for the effect of performing a new experiment on the uncertainty of the existing evaluation. D(x') = D(x) - C(x, p) G [C(x, p)]* . The final uncertainties D(x') do not depend on the actual experimental result p = y - R x, but rather they depend only on the model R and the data covariances D(x) and C(x, y). The covariances, in turn, are determined by the design of the measurement (and not the results of the experiment), so they can be estimated in advance. This means that value of a proposed experiment, in terms of the potential reduction of uncertainty, can be estimated before the experiment is actually conducted. This suggests an objective approach to the planning of measurement campaigns aimed at improving the accuracy of a major database. To see a recent example where this approach is applied to the updating of a data set containing 91,000 parameters, see Reference 6.

Recommended Reading 1. An excellent text on physical statistics is W. C. Hamilton,"Statistics in Physical Science: Estimation, Hypothesis Testing, and Least Squares," Ronald Press, New York (1964). 2. A useful primer on matrix operations is Frank Ayers, Jr., "Theory and Problems of Matrices," Schaum Publishing, New York (1962). 3. D.W. Muir, "Evaluation of Correlated Data using Partitioned Least Squares," Nuclear Science and Engineering, v. 101, pp. 88-93 (1989). 4. D.W. Muir, "ZOTT99, Data Evaluation using Partitioned Least Squares, Code package IAEA1371/01, Computer Program Service, Nuclear Energy Agency, Paris (1999). 5. D. W. Muir, "Treatment of Discrepant Data using the Method of Least Distortion," CODATA 2000, Baveno, Italy (October 2000). 6. D.W. Muir, "GANDR: Tool For Global Assessment of Nuclear Data Requirements," consultant's report, IAEA Nuclear Data Section, Vienna.

Any questions?

Scientific Decision Making in the Face of Uncertainty Tips, tricks and pitfalls in the