June 8, 2005

1. DSST Ensemble VerificationCall 4 SlidesKevin WarnerHank HerrWon Kwock/Edwin WellesJulie Demargne June 8, 2005

2. CBRFC MRF Project DSST Verification Call June 8, 2005 Kevin Werner NWS/WRH/SSD

3. Outline • MRF / ESP hindcasting experiment • Pseudo reforecasting code • Verification of hindcast experiment • Future Work

4. CBRFC AHPS PROJECT A cooperative effort between:

5. Schematic of Using Ensembles from MRF(day 1-14) As Input to ESP Blending/Attachment Days 1-14 Days 15-365 Ensembles From MRF Ensembles From Climatology

6. ESP Reforecast Probabilistic forecast (or model) verification requires a large dataset. This was accomplished through reforecasting. Reforecasts done for every basin for every day between 1979 – 1999. Reforecasts made with both reforecasted MRF and historical MAT/MAPs.

7. ESP Reforecast • For EACH reforecast day (i.e. 1/1/79, 1/2/79, … , 7/1/99)… • NWSRFS carryover states from historical simulated saved. • MAT and MAP ensembles created from MRF reforecast. • Two sets of flow ensembles (i.e. *.CS time series) created from ESP: • Control case: uses historically observed forcings • MRF case: uses 14 day forcings from MRF model

8. ESP Reforecast Schematic: State variables derived from historical simulation and used to initialize reforecast conditional simulations.

9. ESP Reforecast Example:ESP reforecasted flow ensembles (colored lines) produced from MAT/MAP ensembles

10. ESP Reforecast Scripts • Reforecasting done with two TCL scripts that run ESP iteratively: • gen_fg_co.tcl – Runs ESP historical simulation based on existing carryover in fs5files. Carryover iteratively saved for each day. • gen_fg_ts.tcl – Runs ESP simulations based on carryover created from first script. Resulting conditional time series (*.CS) are archived for further analysis.

11. gen_fg_co.tcl #Created 9/03 Kevin Werner#Generates ESP carryover for each day in specified range.#Sequentially creates input file for fcst and calls fcst.###Define Parameters#set env(calb_data_dir) /local/lx/rfc/nwsrfs/calb/data/area_tsset env(calb_area_ts_dir) /local/lx/rfc/nwsrfs/calb/data/area_tsset startsec [clock scan 12/29/2005]set endsec [clock scan 1/03/2006]set fcstindir "/local/lx/rfc/nwsrfs/ofs/input/oper/fcst/"set fcstinfile "co_gen"set ofsoutdir "/local/lx/rfc/nwsrfs/ofs/output/kvw/"#set fcstindir "/fs1/home/oper/reforecast/"##Loop over each day, basin#for {set csec $startsec} {$csec <= $endsec} {incr csec 86400} {##Create fcst input file# set of [open "[subst $fcstindir][subst $fcstinfile]" w] puts $of "@SETOPT" puts $of "STARTESP 0606/2005/" puts $of "WINDOWS(1) 1030/2005/ 0601/2006/" puts $of "NUMCOSAV(1) [clock format $csec -format "%m%d/%Y/"]" puts $of "HISTSIM(1)" puts $of "REGULATE(0)" puts $of "TSUNITS(1) 91 92 93 94 95" puts $of "HISTWYRS 1980 2002" puts $of "FGROUP SPSV" puts $of "@COMP ESP" puts $of "@STOP" close $of if {[catch {exec /local/lx/rfc/nwsrfs/ofs/scripts/ofs -p fcst -i co_gen -o co_gen} msg]} {puts $msg}} Define range of days to create carryover Loop over each day required Save Carryover Run ESP

12. gen_fg_ts.tcl ##Created 10/03 Kevin Werner#Generates ESP time series from carryover generated from gen_fg_co.tcl.###Define Parameters#set env(calb_data_dir) /local/lx/rfc/nwsrfs/calb/data/area_tsset env(calb_area_ts_dir) /local/lx/rfc/nwsrfs/calb/data/area_tsset startyear 1981set endyear 2002set startmonth 1set startday 1set endmonth 1set endday 2set fcstindir "/local/lx/rfc/nwsrfs/ofs/input/oper/fcst/"set fcstinfile "ts_gen_rls"set basinfile "spsv.basins"set basins [read_file -nonewline $basinfile]set basins [split $basins \n]set fgroup "SPSV"set esptsdir "/local/lx/rfc/nwsrfs/ens/files/kvw/espts/spsv/"set ofsoutdir "/local/lx/rfc/nwsrfs/ofs/output/kvw/"#set fcstindir "/fs1/home/kvw/reforecast/"##Loop over each year#for {set year $startyear} {$year <= $endyear} {incr year} { set startsec [clock scan [subst $startmonth]/[subst $startday]/$year] set endsec [clock scan [subst $endmonth]/[subst $endday]/$year]#Loop over each day for {set csec $startsec} {$csec <= $endsec} {incr csec 86400} {##Create fcst input file# set of [open "[subst $fcstindir][subst $fcstinfile]" w] puts $of "@SETOPT" puts $of "STARTESP [clock format $csec -format "%m%d/%Y/"]24MST" puts $of "WINDOWS(1) 0401/[clock format $endsec -format "%Y/"] 0801/[clock format $endsec -format "%Y/"]" puts $of "ESPOTDIR OTDIR=spsv" puts $of "ESPINDIR INDIR=spsv" puts $of "HISTWYRS $startyear $endyear" puts $of "FGROUP $fgroup" puts $of "@COMP ESP" puts $of "@STOP" close $of if {[catch {exec /local/lx/rfc/nwsrfs/ofs/scripts/ofs -p fcst -i $fcstinfile -o $fcstinfile} msg]} {puts $msg} } Define range of days to perform reforecast over Define directory structure Time loops over each day required Write fcst input file Iterate over required days Run ESP

13. gen_fg_ts.tcl Loop over each segment (“basin”) continued##Loop over each basin# foreach basin $basins { set tsfname "[subst $basin].SIM24.SQME.24.CS" file rename -force [subst $esptsdir]$tsfname [subst $esptsdir][subst $tsfname]\ [clock format $csec -format ".%m.%d.%Y"] set tsfname "[subst $basin].ADJ1.QINE.01.CS" file rename -force [subst $esptsdir]$tsfname \ [subst $esptsdir][subst $tsfname][clock format $csec -format .%m.%d.%Y"]#end basin loop }#end day loop }#end year loop } Rename and archive ESP conditional simulations (i.e. *.CS) files for analysis End time and basin loops

14. Reforecasting analysis • Analysis and verification done with combination of TCL and R scripts • TCL scripts for extracting required time series from archived *.CS files (i.e. flows, seasonal volumes, etc.) • R scripts for verifying, analyzing, and plotting • Examples available upon request.

15. ESP Reforecast Example Following example from Granby, CO (GBYC2) reforecast for May 1, 1985. This is a snowmelt situation.

16. Temperature Input into ESP MRF derived MATs and MAPs (not shown) are attached to historical years (“ensembles”) and ‘fed’ to ESP. Note MRF is warmer in first week

17. Input into ESP MRF derived MATs related to the entire year of historical ensembles.

18. ESP flow time series Hourly instantaneous flow ensembles are created by ESP and saved. MRF shows higher flows than historical when it is warmer (during the first week). These may be converted into probabilistic forecasts…

19. ESP peak flow Peak flow forecasts shown as Probability Density Functions (PDFs). MRF shows higher probabilities in higher flows for two weeks.

20. ESP Forecast Verification Mean hydrograph and RPSS values… Good forecast skill improvements during rising limb of hydrograph. Forecast lead time Forecast valid date

21. ESP Forecast Verification

22. Future Reforecasting Work • Currently exploring use of the Climate Forecasting System (CFS) run by CPC/EMC using similar framework • CFS output typically in monthly averages • Reforecast archive available from 1981 • Shows some promise during ENSO events particularly for forecasts made during the autumn months • May be useful for early season water supply forecasts

23. Ensemble Hindcast Time Series Generator(ETSGEN)

24. Overview • Etsgen gui is a graphical user interface (GUI) which enables a user to construct HCL input files for the OFS FCST ESP function and run those HCL input files to generate ESP hindcasts to be used for verification studies.

26. Hindcast Procedure • ESP is run in historical simulation mode to acquire past carryover • One set of carryover is saved for each year of historical simulation run • ESP is executed using each of the past saved carryover as the state of system at forecast time

27. ETSGEN • Input • Hindcast parameters • Carryover group, forecast group(s), segment(s) • Carryover save date • Hindcast start and end dates • Hindcast start and end years • Output • OFS input decks that can be used to generate carryover for hindcasts and the ESP time series files that comprise the hindcasts • Script espvs_generate.sh executes OFS jobs • ETSGEN can be used to run espvs_generate.sh

28. JProbVS Wen Kwock / Edwin Welles May 24, 2005

29. Purpose of This Presentation • Introduce functions of JProbVS • General description of computational process

30. JProbVS Diagram JProbVS Batch mode espadpIteration_DMI Discrimination RPS Reliability RPSS ROC

31. JProbVS Process espadpIteration_DMI DataCard/ ESP TS Tables espadp RPS RPSS Discrimination Reliability ROC Stat Result Stat Result Stat Result Stat Result Stat Result Process Data/File

32. JProbVS Input/Output • Input files: • Input deck: probVSBatch • ESP time series files • Datacard files • Observe file name must in *.OBS format • Forecast file name must in *.VS.CCYYMMMDD format • BAYI4.BAYI4.SQME.24.VS.1981Feb01 • Output files: • .Data files • BAYI4.BAYI4.SQME.24.Feb01.QCVFUZZ.Dis_stat.data • .IWK files • BAYI4.BAYI4.SQME.24.Feb01.QCVFUZZ.Dis_stat.iwk • .Results files • BAYI4.BAYI4.SQME.24.Feb01.QCVFUZZ.Dis_stat.results • .XML files • BAYI4.BAYI4.SQME.24.Feb01.QCVFUZZ.Dis_stat.XML • .Table files • BAYI4.BAYI4.SQME.24.Feb01.QCVFUZZ.Dis_stat.table • Readable by matlab

33. RPS 1) RPS Results ----------- Year | RPS --------+--------- 1950 | 1.0640051509569648 1951 | 1.2982671726400459 1952 | 0.13029338333090265 1953 | 1.5452837682028684 … 2001 | 1.78505677402705 --------+--------- Average | 0.5358592525789612

34. RPS %Formatted for Matlab and Spreadsheets; <tab> delimited columns. % % ========================== % Ranked Probability Score % ========================== % Columns are Year, RPS, with the first row the average (and therefore it has a year of Nan) % NEED to include in here the probability thresholds used in the calculations Nan 0.5359 1950 1.0640 1951 1.2983 1952 0.1303 1953 1.5453 … 2001 1.7851

35. RPS

36. RPSS 1) RPSS Results ----------- Year | RPSc RPSf | RPSS --------+------------------+--------- 1950 | 1.625 1.064 | 34.523 1951 | 1.625 1.298 | 20.107 1952 | 0.125 0.13 | -4.235 1953 | 1.625 1.545 | 4.906 … 2001 | 1.625 1.785 | -9.85 -------------+--------- Average | 0.625 0.0 | 0.0

37. RPSS % Formatted for Matlab and Spreadsheets; <tab> delimited columns. % % ========================== % Ranked Probability Skill Score. % ========================== % Columns are Year, climate RPS, Forecast RPS, and RPSS % with the first row the average (and therefore it has a year of Nan) % the Skill score is reported as a % so divide by 100 in plts % NEED to include in here the probability thresholds used in the calculations. NaN 0.62 -999.0 -999.0 1950 1.62 1.06 34.52 1951 1.62 1.30 20.11 1952 0.12 0.13 -4.23 1953 1.62 1.55 4.91 … 2001 1.62 1.79 -9.85

38. RPSS

39. Discrimination 1) Discrimination Results ---------------------- 3.2) Discrimination | For all forecasts where the obs occurred from C1 to <C2, the probability of the forecast was: | Observation | 0% to <25%| 25% to <50%| 50% to <75%| 75% to <100% From C1 to < C2|------------------|------------------|------------------|--------------- % # Obs| F0 F1 F2 | F0 F1 F2 | F0 F1 F2 | F0 F1 F2 --------------------|------------------|------------------|------------------|------------------- 694.0 |0.0 0.692 1.0 |0.154 0.231 0.0 |0.308 0.077 0.0 |0.538 0.0 0.0 694.0 1500.0 |0.577 0.154 0.538 |0.077 0.577 0.154 |0.346 0.231 0.192 |0.0 0.038 0.115 1500.0 |0.923 0.077 0.154 |0.077 0.385 0.462 |0.0 0.462 0.308 |0.0 0.077 0.077 Where the following applies | From to < -----+-------------------- F0 | 694.0 F1 | 694.0 1500.0 F2 | 1500.0

40. Discrimination % Formatted for Matlab and Spreadsheets; <tab> delimited columns. % % ========================== % Discrimination Diagram Points. % ========================== % First row is lower category bound. Second row is upper category bound. % Starts third row, columns are lower probability bound, upper probability % bound, probabilities for category one, probabilities for category two, ... NaN NaN -inf 694.00 1500.00 -inf 694.00 1500.00 -inf 694.00 1500.00 Nan NaN 694.00 1500.00 inf 694.00 1500.00 inf 694.00 1500.00 inf 0 25 0 69 100 58 15 54 92 8 15 25 50 15 23 0 8 58 15 8 38 46 50 75 31 8 0 35 23 19 0 46 31 75 100 54 0 0 0 4 12 0 8 8

41. Discrimination

42. Reliability 1) Reliability Results ------------------- Forecast Categories | Probability ranges from #1% to less than #2% ( or to 100% ) From to < | 0 25 25 50 50 75 75 100 -----------------------+--------------------------------------------- 698.68 | 0.00 0.40 0.36 1.00 698.68 1,540.48 | 0.25 0.64 0.46 0.50 1,540.48 | 0.07 0.60 0.45 0.00

43. Reliability % Formatted for Matlab and Spreadsheets; <tab> delimited columns. % % ========================== % Reliability Diagram Points. % ========================== % First line provides forecast category max values corresponding to data % in that column. % For every row after, the first column is lower bound of the forecast % probability bin, the second column is the upper bound of the bin, % and then the reliability diagram point is provided for each category. NaN NaN 698.68 1540.48 INF 0.00 0.25 0.00 0.25 0.07 0.25 0.50 0.40 0.64 0.60 0.50 0.75 0.36 0.46 0.45 0.75 1.00 1.00 0.50 0.00

44. Reliability

45. ROC 1) ROC Results ------------------- | Forecast Categories Probability | 1,192.32 | 1,268.65 | 684.52 | Thresholds | POFD POD | POFD POD | POFD POD | ------------+--------------+--------------+--------------+- 0.25 | 0.09 0.26 | 0.09 0.24 | 0.00 0.68 | 0.75 | 0.45 0.79 | 0.40 0.88 | 0.58 0.98 |

46. ROC % Formatted for Matlab and Spreadsheets; <tab> delimited columns. % % ================== % ROC Diagram Points. % ================== % First line provides forecast thresholds corresponding to data % in that column. % For every row after, first column is probability threshold used to % calculate the POFD and POD points. Then, POFD and POD points are % provided for each of the forecast thresholds. NaN 1192.32 1192.32 1268.65 1268.65 684.52 684.52 0.25 0.09 0.26 0.09 0.24 0.00 0.68 0.75 0.45 0.79 0.40 0.88 0.58 0.98

47. ROC

48. For JProbVS documentation, go to http://hsp.nws.noaa.gov/oh/hrl/nwsrfs/esp/jprobvs.pdf

49. Ensemble Hindcaster for Precipitation, Temperature, & Streamflow Probabilistic Verification Hydrologic Ensemble Prediction Team Hydrology Laboratory Office of Hydrologic Development NOAA/National Weather Service June 8, 2005 1

50. Ensemble Hindcaster • Goal: capability for systematic and rigorous hindcasting to validate ensemble science and assess benefit-cost effectiveness of newly proposed solutions • Statement of Need: • Why: validate ensemble science and evaluate probabilistic forecast performance • Originator: AB-, CN-, and MARFC currently generating short-term hydrologic ensembles • Existing capabilities: hindcasting capability at CBRFC; limited capability for short-term precipitation and temperature ensembles • Benefits: improve predictions and validate improvements relative to forecast reliability and accuracy; serve RFC’s operational need for ensemble system calibration and forecast validation 2