Inter-comparison and Validation Task Team (IV-TT)

Inter-comparison and Validation Task Team (IV-TT) Report for GOVST Matt Martin, Fabrice Hernandez

Contents • Introduction • Objectives of IV-TT • Membership • Main elements of work-plan • Santa Cruz workshop outcomes for IV-TT. • Recent progress: • Routine class 4 inter-comparison project. • GOV multi-model ensemble project. • Links to GSOP inter-comparison of climate indices. • Other validation work. • Summary and future plans.

IntroductionObjectives of IV-TT • To coordinate and promote the development of scientific validation and inter-comparison of operational oceanography systems. • Activities to include: • The definition of metrics to assess the quality of analyses and forecasts (e.g. forecast skill), for physical and biogeochemical parameters. • Setting up specific global and regional inter-comparison experiments. • Define metrics related to specific applications. • Liaise with the JCOMM ET-OOFS team for operational implementation. • Develop cooperation with CLIVAR/GSOP for climate issues

IntroductionMembership Existing members of IV-TT: • Fabrice Hernandez (France) • Matt Martin (UK) • Alistair Sellar (UK) • Hal Ritchie (Canada) • Greg Smith (Canada) • Gary Brassington (Australia) • Jim Cummings (US) • Clemente Tanajura (Brazil) • Peter Oke (Australia) • PavelSakov (Norway) • Marina Tonani (Italy) • AvichalMehra (US) Other members welcome.

IntroductionIV-TT work plan • Provide a demonstration of inter-comparison and validation in an operational framework: • write proposal for production of routine class 4 metrics. • implement at participating OOFS. • routinely upload data from OOFS to the US GODAE server. • generate statistical inter-comparisons on a web-page. • once mature, report to ET-OOFS for implementation operationally. • continue to test new metrics etc in GOV and regularly update ET-OOFS. • Improve inter-comparison and validation methodology for operational oceanography: • Obtain feedback from GOVST members participating in dedicated meetings organized by specific communities • New metrics outlined during workshops organized by the task team • Limited inter-comparison projects between some OOFS, in order to test new metrics.

Santa Cruz workshop IV-TT objectives GODAE OceanView – CLIVAR-GSOP Workshop on Observing system evaluations and Inter-comparison 13 – 17 June 2011, UCSC, Santa Cruz, CA, USA Specific goals of the workshop were to: Demonstrate the value of in situ and satellite observations to short-term, seasonal, and decadal forecast systems; Move towards routine monitoring of the global ocean observing system (explore possibilities for “Observation Impact Statements”); Review inter-comparison of class 4 metrics from operational short-range forecast systems and seasonal prediction systems; and Linking inter-comparison and observing system evaluation monitoring activities within GODAE OceanView to those of CLIVAR.

Santa Cruz workshop Inter-comparison and validation presentations Fabrice Hernandez GODAE OceanView IV Work Plan Yan Xue* Comparative Analysis of Upper Heat Content Variability from Ensemble Operational Ocean Analyses AlexandreKurapov* The data assimilation component of the real-time coastal ocean circulation forecast system off Oregon Tong Lee Inferring meridional heat transport through meridional volume transport of the AMOC Clemente Tanajura A Sensitivity Study of short-range Forecasts over the Atlantic METAREA V with HYCOM and the Cooper Haines Scheme using different sea surface heights Mario Adani Quality Assessment of a 1985 - 2007 Mediterranean Sea Reanalysis Stephanie Guinehut Combination of in-situ and satellite observations to monitor the Global Ocean State Gary BrassingtonOceanMAPS version 2 - what did we get from the upgrade? Andrea Storto The CMCC experience in the context of Global Ocean reanalyses Oscar Alves An ensemble ocean data assimilation system for seasonal prediction and its comparison with other state-of-the-art ocean reanalysis Balakrishnan Nair Real-time and delayed mode validation for the Indian Ocean Forecasting System (INDOFOS) Gregory Smith Evaluation of CONCEPTS Ice-Ocean Forecasting System SanthaAkella The GMAO Ocean Retrospective Analysis

Santa Cruz workshop Validation techniques Some commonly used techniques from GOV and coastal groups: Innovation statistics (bias, RMS, stdev, ACC), for common variables (T, S, SSH, SST) Time average increments for diagnosing model biases. Assessing forecast skill using observations and analyses. SSH variability and position of western boundary currents. Some commonly used techniques from CLIVAR/GSOP groups: Assessing meridional overturning circulation (AMOC); Heat transport diagnostics; Comparisons of the depth of the 20C isotherm; Upper ocean heat content for various depth ranges.

Santa Cruz workshop Validation techniques Less commonly used or new assessment techniques (for ocean forecasting systems) include: Assessment of tides in a global system using tide gauges as reference; Altimetry-based analysis of the internal tide (sampling the model in the same way as the data); Comparisons with HF radar data as an independent reference (especially useful when the available velocity data is assimilated); Forecast skill assessments in ensemble forecasting systems, e.g. by comparing spread in the ensemble with the RMSE of the mean; Calculating EOFs of assimilation increments to ascribe various dynamical processes to the model errors; Using Lyapunov exponents to show the scales which are reconstructed by the analysis; Statistical verification of the location of features (e.g. eddies); Verification of Arctic Ocean circulation and investigation of Arctic heat budget; Assessment of volume-integrated heat content in a particular 3D box and calculation of various contributing terms (advective and surface fluxes, data assimilation impact); Validation of geostrophiccurrents at 1000m depth using the ANDRO data-base (based on Argo trajectories); Comparison of currents with ADCP data from ships and from moored buoys; Comparison of modeledice thickness with IceSat thickness data..

Santa Cruz workshop Inter-comparison talks Hernandez presented an overview of the IV-TT work plan. Xue presented a detailed comparison of upper ocean heat content (over the top 300m, HC300) from various ocean reanalyses. The questions addressed by the study included: How well is the mean HC300 analysed? What are the impacts of changes in the observing system? How well do the reanalyses capture inter-annual and multi-decadal variability? What climate indices should be monitored? What is the role of HC300 in predictability of ENSO, PDO, NAO, …? Also inter-comparisons carried out by the PEODAS group for assessing their system.

Yan Xue Climate Prediction Center /NCEP Mean Heat Content in 1985-2009 obs 1985 obs 2008

Santa Cruz workshop IV-TT breakout Discussions of the routine class 4 inter-comparison project => changes to the proposed method to ensure consistency of observations and format. Initial discussion of a proposed project to produce a GOV multi-model ensemble, which we decided to pursue. Discussions of ways in which the GOV groups could get involved in the CLIVAR/GSOP inter-comparison of climate indices. These three initiatives were discussed further yesterday…

1. Class 4 inter-comparisonOverview Aim is to develop a system for routine inter-comparison of the operational ocean forecasts contributing to GOV. This will be done in order to facilitate improvements to the systems and to help understand where our systems should focus development. Comparisons of forecasts against observations was chosen for this so that we don’t have to deal with excessive data volumes. These data will be made available for inter-comparison: in common formats; for a core sub-set of the various data types which the systems assimilate; to enable assessment of forecasts.

1. Class 4 inter-comparisonOverview The core data types agreed on for use in the inter-comparison, which are now being implemented are: SST from in situ drifting buoys, using the US GODAE data-set. SST from L3 AATSR data-set. In situ T & S profile data from MyOcean (produced by Coriolis). Altimeter SLA data from MyOcean (CLS/Aviso). Model data exchange will be done through the US GODAE server. All files will be encrypted. The project proposal document has been updated to include a step-by-step guide of how groups can contribute to the inter-comparison. The document was distributed to the IV-TT and put on the GOV web-site.

1. Class 4 inter-comparisonOverview Produce a core set of plots for the GOV web-site (password protected). Any participating group can download files and inter-compare the results. Observation inputs US GODAE server UKMO download obs files and produce model analysis, forecast, persistence and climatological information at the obs locations in netCDF format. Each OOFS downloads the UKMO files and replaces the model data with those from their system. ~5 days behind real time

1. Class 4 inter-comparisonCurrent status Production of SST files at UKMO now being done routinely and uploaded onto the US GODAE server – set up by Andy Ryan and Alistair Sellar. Other data types: code written and tested for T & S profiles, and altimeter SLA data. Will get T & S profiles, and SLA transferred to US GODAE server soon (profiles in next week or so, SLA before end of 2011). Canadians already started downloading data and plan to start producing the files soon. Mercator, Bluelink, NRL, NCEP and REMO also plan to participate soon. Now setting up web-pages (currently internally) to display information from these class 4 files. These could be used as a basis for an external web-page to display inter-comparison information. If so this would be password protected.

1. Class 4 inter-comparison Example results from FOAM In Situ SST RMSE oC Profile Temperature RMSE oC

1. Class 4 inter-comparison Prototype web pages

2. GOV multi-model ensembleOverview In the past, for particular incidents (e.g. oil spills), groups have requested sharing of ocean forecasts from GOV partners in order to provide ensemble information. A proposal was made at the Santa Cruz workshop to begin generation of a GODAE OceanView multi-model ensemble. There was a general consensus that this should be done and some practical aspects were discussed at the workshop. Yesterday we had further discussions. A number of the OOFS now have model data available externally and routinely on OpenDAP or ftp servers.

2. GOV multi-model ensembleProposal In order to make the exchange of the data an easier process in the case of emergencies, and to perform some scientific studies on the use of multi-model ensembles, it was agreed that some pilot studies will be carried out: In the Australian region by CSIRO (Peter Oke). Globally by NCEP (HendrikTolman). It was agreed to use analyses and daily mean forecasts out to 6-days of SST, SSS, SSH, surface u and v, and sea-ice concentration and thickness. Also proposed to include mixed layer depth, although difficulties with common definition. First step is to collect information on access for this data and distribute to the participating groups.

3. GOV/GSOP climate indices inter-comparison The production of climate indices from GODAE OceanView systems was discussed at the Santa Cruz workshop. Mercator are involved in the GSOP inter-comparisons and produce various climate indices from their 1/4- and 1/12-resolution models, which has proved useful. For a number of the GODAE OceanView systems, the priorities are the class 4 and ensemble inter-comparison projects discussed before. However, it was agreed that we should produce the surface climate indices as part of the multi-model ensemble as a minimum (and interested groups could provide other information if desired). In addition, it was agreed that some of the systems would produce upper ocean heat content estimates on a monthly basis on a 1 degree grid, and provide them to Yan Xue.

Other validation issues discussed yesterday The XBT community would like to know the impact of their data on the GOV systems. Some groups have agreed to examine particular repeat lines (e.g. near Brazil, near Australia) and to inter-compare model data with the XBTs, e.g.: REMO will compare results to XBT lines from Brazil. CSIRO will compare results to XBT lines near Australia. Tide gauges. Nicolas Ferry presented interesting results from a validation exercise with Mercator. It was agreed that this would be useful to include in the framework of the class 4 inter-comparison project. But on a slower time-scale (once per year). The data-set used so far can be distributed to interested groups. Will need to distribute a list of the location of the tide gauges so that the model information can be calculated routinely.

Summary and future work • Workshop at Santa Cruz helped to: • Provide overview of validation done at the various centres. • Refine plans for class 4 inter-comparison. • Propose and discuss GOV multi-model ensemble. • Make stronger links with GSOP groups. • Class 4 inter-comparison is progressing. • General agreement on the GOV multi-model ensemble for surface fields, and a way forward with some pilot studies (globally and around Australia). • Agreement from some systems to begin contributing to the inter-comparison of upper ocean heat content organised within GSOP.

Inter-comparison and Validation Task Team (IV-TT)