130 likes | 289 Vues
A Superior Alternative to the Modified Heidke Skill Score for Verification of Categorical Versions of CPC Outlooks. Bob Livezey Climate Services Division/OCWWS/NWS 28 th Climate Diagnostics and Prediction Workshop Reno, October 20, 2003. Outline. Introduction
E N D
A Superior Alternative to the Modified Heidke Skill Score for Verification of Categorical Versions of CPC Outlooks Bob Livezey Climate Services Division/OCWWS/NWS 28th Climate Diagnostics and Prediction Workshop Reno, October 20, 2003
Outline • Introduction 2. Contingency Tables & Notation • Common Scores & Score Attributes • Gandin & Murphy Equitable Scores • Gerrity Scores • Recommendations
Contingency Tables and Notation Table 4.2. Contingency table giving pij in percent (total sample size n=788) for U.S. mean temperature forecasts for June through August 1983-90. pij: Joint relative frequencies pi:: Observed relative frequencies qi: Forecast relative frequencies pi*: Prescribed relative observed frequencies (climatology)
Some Simple Categorical Skill Scores: Heidke, CPC Heidke, and Pierce
Some Simple Categorical Skill Scores: Heidke, CPC Heidke, and Pierce Skill scores for U.S. mean temperature forecasts in three categories for February through April and June through August 1983-90.
Desirable Attributes of Scores • Equitable; • Equitable without dependence on the forecast distribution; • Rewards for correct forecasts inversely proportional to their event frequencies; • Penalties for incorrect forecasts directly proportional to their event frequencies; • Penalties for incorrect ordinal forecasts with equal event frequencies proportional to degree of miss; • Consistent with an underlying linear association and insensitive to type or number of categories used. Note: 2 &3 imply that all information in the contingency table is taken into account.
Recommendations • CPC use the Gerrity score for ordinal multi-categorical verification • Forecast history is digitized so skill history can be constructed • Clueless audience remains clueless • Score now equitably accounts for all facets of forecast performance • CPC use actual frequencies • CPC routinely determine confidence limits of scores • Reference Jolliffe and Stephenson (2003; Wiley)