Wolfgang Garche Saxony-Anhalt Environmental Protection Agency

1. Estimation of Measurement Uncertainties Wolfgang Garche Saxony-Anhalt Environmental Protection Agency Department Air Quality Monitoring, Information, Assessment wolfgang.garche@lau.mlu.sachsen-anhalt.de Saxony-Anhalt Wolfgang Garche State Environmental Protection Agency

2. Measurement error Systematic error Random error Known systematic error Unknown systematic error Correction Residual error Measurement uncertainty Measurement result Types of measurement error and their consideration in determining the result of a measurement and the associated uncertainty (Source: EUROLAB Technical Report 1/2006 „Guide to the Evaluation of Measurement Uncertanty for Qualitative Test Results“) Saxony-Anhalt Wolfgang Garche State Environmental Protection Agency

3. Measured values for simultaneous occurring of random and systematic errors (Source: EUROLAB Technical Report 1/2006 „Guide to the Evaluation of Measurement Uncertanty for Qualitative Test Results“) Saxony-Anhalt Wolfgang Garche State Environmental Protection Agency

4. Definitions • Uncertainty of measurement is a parameter, associated with the result of a measurement, that characterises the dispersion of the values that could reasonably be attributed to the measurand. Uncertainty of the result Estimated quantity intended to characterise a range of values which contains the reference value. Standard uncertainty (u) Uncertainty of the result of a measurement expressed as a standard deviation. Combined standard uncertainty (uc ) Standard uncertainty of the result of a measurement when that result is obtained from the values of a number of other quantities, equal to the positive square root of a sum of terms, the terms being the variances or covariances of these other quantities weighted according to how the measurement result varies with changing these quantities. Saxony-Anhalt Wolfgang Garche State Environmental Protection Agency

5. Expanded uncertainty (Up) Quantity defining an interval about the result of a measurement that may be expected to encompass a large fraction of the distribution of values that could reasonably be attributed to the measurand. Coverage factor (k) Numerical factor used as a multiplier of the (combined) standard uncertainty in order to obtain an expanded uncertainty. Accuracy The closeness of agreement between a test result and the accepted reference value. Trueness The closeness of agreement between the average value obtained from a large series of test results and an accepted reference value. Precision The closeness of agreement between independent test results obtained under stipulated conditions. Saxony-Anhalt Wolfgang Garche State Environmental Protection Agency

6. EN ISO 20988:2007 „Air quality – Guidelines for estimating measurement uncertainty“ • A five-step procedure for uncertainty estimation is described: • Problem specification • Statistical analysis • Estimation of variances and covariances • Evaluation of uncertainty statements • Reporting Saxony-Anhalt Wolfgang Garche State Environmental Protection Agency

7. Problem specification • Objective is to specify • the measurement • the wanted uncertainty statement • the experimental data • effects not described by experimental data Saxony-Anhalt Wolfgang Garche State Environmental Protection Agency

8. Statistical analysis A statistical model equation shall be established to describe the relationship between the statistical population of possible results of measurement Y and the input data. Saxony-Anhalt Wolfgang Garche State Environmental Protection Agency

9. Directapproach • A direct approach provides a series of results of measurement observed in a specified experimental design and, if appropriate, expert judgement of additional deviations dYi caused by effects not described by the series of observations statistical model equation: where: Y a possible result of measurement y a realized result of measurement (input data) dYi an additional deviation of result of measurement y not described by the experimental data (can be neglected, if the corresponding variance contributes less than 5 % to the variance estimate var(Y) used in the uncertainty estimation) variance budget equation: where: var(Y) estimate of the variance of possible results of measurement Y var(y) estimate of the variance of a series of results of measurement y var(dYi) estimate of the variance of additional deviation dYi , obtained by a type B evaluation Saxony-Anhalt Wolfgang Garche State Environmental Protection Agency

10. Indirect approach • provides for each input quantity xiof a known method model equation • y = f(x1,..,xK) • either a series of experimental data xi(j) collected in a specified experimental design, or a variance estimate var(xi). If appropriate, additional deviations δYj , which are not described appropriately by the experimental data to be evaluated, are assessed by expert judgement. statistical model equation: where Ypossible result of measurement; xi input quantity of the method model equation y = f (x1,.., xK) dYi additional deviation of result of measurement y not described by the experimental data variance budget equation: where ci sensitivity coefficient with respect to variations of input quantity xi cov(xi, xj) estimate of covariance between input quantities xiand xj Saxony-Anhalt Wolfgang Garche State Environmental Protection Agency

11. Sensitivity coefficients • The sensitivity coefficient ciis the partial derivative of the method model function • y = f (x1,.., xK ) ci - can be calculated numerical from the partial derivative Example: y= mass concentration of particles ci - as mean value of the ratio of observed changes of the results of measurement Dy(j) to the the changes of the input data Dxi(j) Saxony-Anhalt Wolfgang Garche State Environmental Protection Agency

12. Estimation of variances and covariances • Type A: Evaluation using statistical analysis of measurement series • The calculation method is depending on the experimental design used to collect the input data. • Type B: Evaluation using means other than statistical analysis of measurement series (expert judgement) • If information are available • on the expected range of variation [min(δYj) < δYj< max(δYj)] of the deviation δYj • on the expected type of the statistical population of δYj Saxony-Anhalt Wolfgang Garche State Environmental Protection Agency

13. Covariances • The covariance associated with the values xi and xkassigned to two input quantities of the applicable method model equation shall be zero if: • xiand xkhave not been observed repeatedly in the same experimental design • either xior xkwas kept constant, when providing repeated observations of the other quantity repeatedly. Calculation of covariances can be avoided often by appropriate choice of experimental designs! Saxony-Anhalt Wolfgang Garche State Environmental Protection Agency

14. Evaluation of uncertainty statements Combined standard uncertainty: where var(Y) is the estimate of the variance of the population of possible results of measurement Y Relative standard uncertainty: Expanded Uncertainty: Saxony-Anhalt Wolfgang Garche State Environmental Protection Agency

15. Expanded uncertainty – coverage faktor k coverage factor kand coverage probability pshall be stated when an expanded uncertainty Up(y) is reported • relationship between k and p: • y is a mean value (N>1) of independent observations with the same measuring system • y is obtained by single application of a measuring method, distribution of possible results approximately is Gaussian • y is obtained by single application of a measuring method, distribution of possible results are not described by a Gaussian distribution Saxony-Anhalt Wolfgang Garche State Environmental Protection Agency

16. Case 1. and 2.:k = t(p,n) (direct approach) where: t(p,n)is the (1-p)-quantile of Students t- distribution of ν degrees of freedom p is the coverage probability of interval [–t(p,ν); +t(p,ν)] by Students t-distribution with ν degrees of freedom n is the number of degrees of freedom; n = N – 1 Uncertainty interval for a coverage probability: Indirect approach: Welch-Satterthwaite-equation: Saxony-Anhalt Wolfgang Garche State Environmental Protection Agency

17. Reporting • A report on execution of a specified task of uncertainty estimation shall include (at least) the following items: • Problem specificationincluding • method of measurement • wanted uncertainty statement • statistical population of possible results of measurement considered • experimental data and experimental design • effects not described by experimental data • statistical analysis, describing the applied statistical model equation and the variance (budget) equation • evaluation methods, describing the applied evaluation methods • numerical value of the wanted uncertainty statement and its range of application Saxony-Anhalt Wolfgang Garche State Environmental Protection Agency

18. Uncertainty estimation in practise • What is required ?? • Analysers have to fulfil the data quality objectives of the Directive 2008/50/EC “on ambient air quality and cleaner air for Europe”. • (Uncertainty and minimum data capture) • Analysers have to fulfil the relevant performance characteristics and criteria of the EN standards. • (Type approval test) Test reports of the type approval should contain all needed uncertainty information for a specific type of analyser. But the user has to show that an analyser fulfil the requirements on measurement uncertainty also under the specific conditions on the monitoring site.  suitability evaluation Saxony-Anhalt Wolfgang Garche State Environmental Protection Agency

19. Type approval of analysers • the value of each individual performance characteristic tested in the laboratory shall fulfil the requirements • the expanded uncertainty calculated from the standard uncertainties due to the values of the specific performance characteristics obtained in the laboratory tests fulfils the requirements • the value of each of the individual performance characteristics tested in the field shall fulfil the requirements • the expanded uncertainty calculated from the standard uncertainties due to the values of the specific performance characteristics obtained in the laboratory and field tests fulfils the requirements Saxony-Anhalt Wolfgang Garche State Environmental Protection Agency

20. General Requirements: • When a type approved analyser has been chosen for a particular measuring task, then the suitability of this analyser shall be evaluated at a specific measuring location. • The analyser at the specific site has been judged to conform with the EU data quality objectives • An expanded uncertainty calculation for the type approved analyser shall be made according to the specific circumstances at the monitoring station or site. • If the site-specific conditions are outside the conditions for which the analyser is type approved, then the analyser shall be retested under these site-specific conditions and a revised type approval will be issued. • If the analyser complies with the requirements, then that particular analyser may be installed and used at that monitoring station. Saxony-Anhalt Wolfgang Garche State Environmental Protection Agency

21. Saxony-Anhalt Wolfgang Garche State Environmental Protection Agency

22. Operating Sequence: Choosing of the needed performance characteristics from the type approval test report Evaluation or estimation of the site-specific conditions and of the uncertainty of the span gas used for calibration Calculation of the combined expanded uncertainty with inclusion of the site-specific conditions Comparison with the uncertainty requirements Saxony-Anhalt Wolfgang Garche State Environmental Protection Agency

23. Example: Uncertainty of the certification of a transfer standards Measurand y: NO concentration of the test gas Method: analyzer according to EN 14211 Input data: daily measurements of a certified NO test gas uncertainty of the test gas concentration with Model equation: Variance equation: Residual standard deviation: Standard uncertainty: Saxony-Anhalt Wolfgang Garche State Environmental Protection Agency

24. certified NO concentration yR = 132,8 ppb stated expanded uncertainty U(yR) = 2% with k=2  standard uncertainty u(yR)=1,328 ppb Calculated: mean value 132,65 ppb residual standard dev. u(e)= 0,538 ppb Expanded uncertainty: Saxony-Anhalt Wolfgang Garche State Environmental Protection Agency

25. Ozone measurements: (example C.2 from EN ISO 20988) Method: automatically measurements according EN 14625 (UV-absorption) function control every 25 h with zero and span gas daily correction of zero offsets Measurand: 1-h mean value of the ozone concentration in ambient air Described effects: variationsof surrounding air temperature and pressure Method model equation: y = x – e(j) x is the observed value without correction e(j) is the offset for day j Wanted uncertainty statement: standard uncertainty u(y), expanded uncertainty U0,95(y) Saxony-Anhalt Wolfgang Garche State Environmental Protection Agency

26. Experiment: 1. every 25 hours providing of zero gas and determination of the zero offset e(j) = x0(j) 2. every 25 hours providing of span gas and evaluation of the span factor ß(j) = xs(j)/ys Input data: series of offset corrections (N=20) series of span factors (N=20) Reference values: Zero gasy0= 0 µg/m³ Span gas ys = 280 µg/m³ u(ys) = 2,8 µg/m³ Effects not described: Influence of sampling device (Influence of humidity and other compounds in the ambient air) Variance budget equation: var(y) = u²(x) + u²(e) + 2cov(x,e) cov(x,e) = 0 Saxony-Anhalt Wolfgang Garche State Environmental Protection Agency

27. Standard uncertainty of zero offset e: (includes the bias uB(e)) Model equation for x(j): Variance equation: (cov(ß,yS) = 0) Standard uncertainty of ß: (includes the bias uB(ß) ) Saxony-Anhalt Wolfgang Garche State Environmental Protection Agency

28. Standard uncertainty of y: Uncertainty depends on the corrected measured value! Degrees of freedom: n = 20  coverage factor: k = t(0,05,20) = 2,1 Now it is possible to calculate an uncertainty statement for the 1-hour mean values measured in the observed period depending on the corrected measured value. Saxony-Anhalt Wolfgang Garche State Environmental Protection Agency

29. Helpful documents and programs: EN ISO 20988 “Air quality – Guidelines for estimating measurement uncertainty” Eurolab Technical Report 1/2006 “Guide to the Evaluation of Measurement Uncertainty for Quantitative Test Results” Eurolab Technical Report 1/2007 “Measurement Uncertainty Revisited: Alternative Approaches to Uncertainty Evaluation” Excel-Program Nordtest TR 537 “Measurement Uncertainty Estimation” Saxony-Anhalt Wolfgang Garche State Environmental Protection Agency

30. Many Thanks for your Attention! Saxony-Anhalt Wolfgang Garche State Environmental Protection Agency