Rank Histograms – measuring the reliability of an ensemble forecast - PowerPoint PPT Presentation

slide1 n.
Download
Skip this Video
Loading SlideShow in 5 Seconds..
Rank Histograms – measuring the reliability of an ensemble forecast PowerPoint Presentation
Download Presentation
Rank Histograms – measuring the reliability of an ensemble forecast

play fullscreen
1 / 32
Rank Histograms – measuring the reliability of an ensemble forecast
148 Views
Download Presentation
barth
Download Presentation

Rank Histograms – measuring the reliability of an ensemble forecast

- - - - - - - - - - - - - - - - - - - - - - - - - - - E N D - - - - - - - - - - - - - - - - - - - - - - - - - - -
Presentation Transcript

  1. Rank Histograms – measuring the reliability of an ensemble forecast • You cannot verify an ensemble forecast with a single observation. • The more data you have for verification, (as is true in general for other statistical measures) the more certain you are. • Rare events (low probability) require more data to verify => as do systems with many ensemble members. From Barb Brown

  2. From Tom Hamill

  3. Troubled Rank Histograms Counts 0 10 20 30 Counts 0 10 20 30 1 2 3 4 5 6 7 8 9 10 Ensemble # 1 2 3 4 5 6 7 8 9 10 Ensemble # Slide from Matt Pocernic

  4. From Tom Hamill

  5. From Tom Hamill

  6. From Tom Hamill

  7. From Tom Hamill

  8. From Tom Hamill

  9. Example of Quantile Regression (QR) Our application Fitting T quantiles using QR conditioned on: Ranked forecast ens ensemble mean ensemble median 4) ensemble stdev 5) Persistence R package: quantreg

  10. Step 2: For each quan, use “forward step-wise cross-validation” to iteratively select best subset Selection requirements: a) QR cost function minimum, b) Satisfy binomial distribution at 95% confidence If requirements not met, retain climatological “prior” Step I: Determine climatological quantiles Probability/°K climatological PDF 1. Regressor set: 1. reforecast ens 2. ens mean 3. ens stdev 4. persistence 5. LR quantile (not shown) 3. T [K] 2. 4. Temperature [K] observed forecasts Time Step 3: segregate forecasts into differing ranges of ensemble dispersion and refit models (Step 2) uniquely for each range Final result: “sharper” posterior PDF represented by interpolated quans forecasts Forecast PDF posterior I. II. III. II. I. Probability/°K prior T [K] Temperature [K] Time

  11. Rank Probability Score for multi-categorical or continuous variables

  12. Scatter-plot and Contingency Table Brier Score Does the forecast detect correctly temperatures above 18 degrees ? y = forecasted event occurence o = observed occurrence (0 or 1) i = sample # of total n samples => Note similarity to MSE Slide from Barbara Casati

  13. Other post-processing approaches … 1) Bayesian Model Averaging (BMA) – Raftery et al (1997) 2) Analogue approaches – Hopson and Webster, J. Hydromet (2010) 3) Kalman Filter with analogues – DelleMonache et al (2010) 4) Quantile regression – Hopson and Hacker, MWR (under review) 5) quantile-to-quantile (quantile matching) approach – Hopson and Webster J. Hydromet (2010) … many others

  14. Quantile Matching: another approach when matched forecasts-observation pairs are not available => useful for climate change studies ECMWF 51-member Ensemble Precipitation Forecasts compared To observations • 2004 Brahmaputra Catchment-averaged Forecasts • black line satellite observations • colored lines ensemble forecasts • -Basic structure of catchment rainfall similar for both forecasts and observations • -But large relative over-bias in forecasts

  15. Forecast Bias Adjustment • done independently for each forecast grid • (bias-correct the whole PDF, not just the median) Model Climatology CDF “Observed” Climatology CDF Pmax Pmax Precipitation Pfcst Padj 25th 50th 75th 100th 25th 50th 75th 100th Quantile Quantile In practical terms … ranked forecasts ranked observations 0 1m 0 1m Precipitation Precipitation Hopson and Webster (2010)

  16. Bias-corrected Precipitation Forecasts Original Forecast Brahmaputra Corrected Forecasts Corrected Forecast => Now observed precipitation within the “ensemble bundle”