Pertti Nurmi Juha Kilpinen Annakaisa Sarkanen ( Finnish Meteorological Institute )

1. Pertti Nurmi Juha Kilpinen Annakaisa Sarkanen ( Finnish Meteorological Institute ) Probabilistic Forecasts of Near-Gale Force Winds in the Baltic Applying ECMWF, EPS and Other Methods ECMWF Forecast Products User Meeting 15 – 17 June 2005 ECMWF User Meeting

2. Introduction: A study with 2 frameworks: • Develop warning criteria / Guidance methods to forecast probability of near-gale force winds in the Baltic  Joint Scandinavian research undertaking • e.g. Finland and Sweden issue near-gale & storm force wind warnings for same areas using different criteria => homogenise ! • Evaluation of ECMWF products • Deterministic and probabilistic forecasts • Two (maybethree) calibration methods • Here, only ECMWF data applied  Later, HIRLAM, too • Here, 6 Finnish coastal stations  Later, c. 15-20 stations from Sweden, Denmark, Norway • Goal: Common Scandinavian operational practice (?) ECMWF User Meeting

3. ECMWF MARS u & v components at 10 m => wind speed at 10m Forecast lead times: +12 hr to +144 hr Data retrieval: 0.5 * 0.5 degree resolution Operational, Control, EPS data (interpolated to 0.5o * 0.5o) Nearest grid point used Forecasts / observations valid: 00, 06, 12, 18 utc Observations: 10 minute mean wind speed Data coverage: 1/10/04 – 30/4/05  212 days Data: ECMWF User Meeting

4. with height of instrumentation ? with observing site surroundings and obstacles ? with the coast ? with nearby islands ? with barriers ? with installations ? with low-level stability ? NE We may have problems: ECMWF User Meeting

5. Observing stations( 6 out of 39 ) 02_873- Hailuoto 02_910- Valassaaret 02_980- Nyhamn 02_979 - Bogskär 02_981 - Utö 02_987- Kalbådagrund ECMWF User Meeting

6. Heights of the instrumentation ( in red, the 6 out of 39 ) (m) 55 50 45 40 35 30 25 20 15 10 5 ECMWF User Meeting

7. 873-Hailuoto 46 / 8 m ECMWF User Meeting

8. 910 - Valassaaret 22 / 18 m ECMWF User Meeting

9. 979 - Bogskär 32 / 4 m ECMWF User Meeting

10. 980 - Nyhamn 25 / 8 m ECMWF User Meeting

11. 981 - Utö 31 / 9 m ECMWF User Meeting

12. 987 - Kalbådagrund 32 / 23 m ECMWF User Meeting

13. 979 - Bogskär Unstable 32 m Neutral Stable 10 m Wind speed dependence: Logarithmic wind profile 14 m/s 15 15,5 m/s threshold ECMWF User Meeting

14. Methods for producing probabilistic forecasts: • EPS (51 members): Probability of wind speed > 14 m/s • Kalman filtering • Various approaches  No details given here • Deterministic forecasts, adjusted by “a posteriori” estimate of the observed error distribution of the dependent sample Probability distribution of near-gale • Gives an estimate of the upper limit of the probabilistic predictability. Deterministic forecasts: • Error distribution of original sample (212 cases) • Approximation of the error distribution with a Gaussian fit (m, s): • ”sample error” method ECMWF User Meeting

15. Methods for producing probabilistic forecasts: • Deterministic forecasts, adjusted with a Gaussian distribution fitted to model forecasted stability(temperature forecasts at 2 adjacent model levels)  Probability distribution of near-gale, “stability” method • Scheme used at SMHI (H. Hultberg) • “Neighbourhood” method • Both spatial (right) and temporal “neighbours” • c. 25-75 “members” • Applicable primarily for hi-res models ? ECMWF User Meeting

16. Calibration of EPS forecasts: • Traditionally, calibration of the EPS is done by re-labeling the probabilities using the information of the reliability diagram (large sample of past forecasts and observations is needed) • Here, Kalman filtering is used to calibrate the EPS mean (as well as operational and control forecasts). Then each EPS member is transformed with the same relationship (state vector). • This will calibrate the “mean” of the distribution, hopefully also the “spread”. • Kalman filtering is also used in the traditional way to correct the deterministic forecasts and then to estimate the probabilities using the observed error distribution. ECMWF User Meeting

17. 32 m 46 m 25 m 22 m 31 m 32 m Sample climatologic characteristics ECMWF User Meeting

18. 32 m 46 m 25 m 22 m 31 m 32 m Sample climatologic characteristics, ref. ECMWF ECMWF User Meeting

19. 32 m 46 m 25 m 22 m 31 m 32 m Sample climatologic characteristics, ref. ECMWF ECMWF User Meeting

20. Sample climatologic characteristics, ref. ECMWF ECMWF User Meeting

21. Deterministic FCs: Bias - RMSE - 981_Utö w.r.t to FC lead time ME (Bias) RMSE “Ensemble spread” ECMWF User Meeting

22. Deterministic FCs: Bias - RMSE - 987_Kalbåda w.r.t to FC lead time ME (Bias) RMSE ECMWF User Meeting

23. Brier Score: • BS = ( 1/n ) Σ ( p i – o i ) 2 • Common accuracy measure of prob fcs • o iis binary (0 or 1) • Analogous to MSE in probability space • A quadratic scoring rule • Sensitive to large forecast errors ! • Careful with limited datasets ! • Influenced by climatologic frequency of the sample • Different samples not to be compared • Brier Skill Score: • BSS = [ 1 – BS / BS ref ] *100 Range: 0 to 1 Perfect score = 0 Range: - oo to 100 Perfect score = 100 Probabilistic FCs: BrierSkill w.r.t to FC lead time 987_Kalbåda ECMWF User Meeting

24. Relative Operating Characteristic Probabilistic FCs: ROC • To determine the ability of a forecasting system to discriminate between situations when a signal is present (here, occurrence of gale) from no-signal cases (“noise”) • To test model performance relative to a specific threshold • Applicable for probability forecasts and also for categorical deterministic forecasts • Allows for their comparison • Gained popularity in forecast verification in recent years ECMWF User Meeting

25. Probabilistic FCs: ROC Curve • Graphical representation in a square box of the Hit rate (H) (y-axis) against the False Alarm Rate (F) (x-axis) for different potential decision thresholds • Curve is plotted from a “binned” set of probability forecasts by stepping (or sliding) a decision threshold (e.g. 10% probability intervals) through the forecasts, each probability decision threshold generating a separate 2*2 contingency table • The probability forecast is transformed into a set of categorical “yes/no” forecasts • A set of value pairs of H and F is obtained, forming the curve • It is desirable that H be high and F be low, i.e. the closer the point is to the upper left-hand corner, the better the forecast • A perfect forecast system, with only correct forecasts & no false alarms, (regardless of the threshold chosen) has a “curve” that rises from (0,0) (H=F=0) along the y-axis to (0,1) (upper left-hand corner; H=1, F=0) and then straight to (1,1) (H=F=1) H = a / ( a + c ) F = b / ( b + d ) ECMWF User Meeting

26. Probabilistic FCs: ROC Curve generation ( S a ) ( S b ) b+d =5351 a+c =1920 Example H = a / ( a + c ) F = b / ( b + d ) To learn more about ROC and Signal Detection Theory, check: http://wise.cgu.edu/ ECMWF User Meeting

27. Probabilistic FCs: ROCA Area Range: 0 to 1 Perfect system = 1 • Area under the ROC curve • Decreases from 1 when curve moves downward from the ideal top-left corner • A useless forecast system is along the diagonal, when H=F and the area is = 0.5; Such system cannot discriminate between occurrences and non-occurrences of the event ROCA based skill score: ROC_SS = 2 * ROCA - 1 • Negative below the diagonal • At it’s minimum: ROC_SS = - 1, when ROCA = 0 • ROC is applicable for deterministic categorical forecasts • ROC_SS translates into KSS  TSS (= H – F ) • Only one single decision threshold - only a single ROC point results Typically, this is “inside“ the ROC area, i.e. indicating worse quality • ROC, ROCA and ROC_SS are directly related to a decision-theoretic approach • Can be related to the economic value of probability forecasts to end users • Allowing for the assessment of the costs of false alarms Range: -1 to 1 Perfect score = 1 ECMWF User Meeting

28. Probabilistic FCs: ROC curve/area; T +48 hr 987_Kalbåda ROC_EPS ROC_Kalman (EPS) ROCA = 0.85 ROCA = 0.73 ECMWF User Meeting

29. Probabilistic FCs: ROC curve/area; T + 24 hr 981_Utö ROC_”stability” ROC_”neighbour” ROCA = 0.88 ROCA = 0.96 ECMWF User Meeting

30. EPS Probabilistic FCs: ROC Area w.r.t to FC lead time 873_Hailuoto 910_Valassaaret ECMWF User Meeting

31. Probabilistic FCs: ROC Area w.r.t to FC lead time 981_Utö 987_Kalbåda ECMWF User Meeting

32. Conclusions  Future: • So far we’ve just scratched the (sea) surface • Need much more experimentation with various methods • Different methods for different time/space scales • e.g. very-short vs. medium-range ? • Biases and other scores depend on station (e.g. observation height) • (Statistical) adjustment of original observations required ? • Finland has an operational scheme for this ! • EPS forecasts are slightly underdispersive • Kalman filtering reduces the biases and produces better prob. forecasts for most stations in terms of the ROC curve/area • Apply to data from other counterparts • Reach the goal… !!! ECMWF User Meeting

33. Forecast Quality Project 2005 The Royal Meteorological Society at the behest of the UK weather forecasting industry and their customers, has undertaken a project to establish methodologies and metrics by which the quality of weather forecast services can be assessed from a user perspective on a basis that is clear, scientifically well founded, relevant to the users’ needs and easily applied and understood. UK forecast user and provider input is NOW needed!www.rmets.org/survey Almost finnish(ed), but one advertisement… ECMWF User Meeting