1 / 18

Ed Armstrong Jet Propulsion Lab/CalTech Physical Oceanography DAAC Pasadena, California

Data and Metadata Standards for GHRSST: present to future ESIP Federation Meeting 15-18 July 2008 Univ. of New Hampshire, Durham, NH. Ed Armstrong Jet Propulsion Lab/CalTech Physical Oceanography DAAC Pasadena, California. Data throughput…. Gb Day -1. User Base. Power Users: NAVO

mio
Télécharger la présentation

Ed Armstrong Jet Propulsion Lab/CalTech Physical Oceanography DAAC Pasadena, California

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Data and Metadata Standards for GHRSST: present to futureESIP Federation Meeting 15-18 July 2008Univ. of New Hampshire, Durham, NH Ed Armstrong Jet Propulsion Lab/CalTech Physical Oceanography DAAC Pasadena, California ESIP Federation Meeting 15-18 July 2008

  2. Data throughput… Gb Day-1 ESIP Federation Meeting 15-18 July 2008

  3. User Base • Power Users: • NAVO • UK Met Office • NASA JPL • Canada Meteorological Agency • Danish Meteorological Institute ESIP Federation Meeting 15-18 July 2008

  4. Product Specifications - 2003 netCDF 3.6 Self describing format Popular in the ocean modeling and NWP community Direct access Improves usability: L2P products can have over 20 arrays Portable GHRSST is an international project with data produced on a variety of platforms “bundled” with CF/COARDS compliant metadata Limitations: No internal compression. Externally compressed (bzip2 or gzip) No chunking ESIP Federation Meeting 15-18 July 2008

  5. Metadata Specifications - 2003 NASA Directory Interchange (DIF) format Compact, well defined user history, simple to implement A granule level XML metadata (File Record [FR]) for each file made by producer typical entries are dynamic such as time and position e.g., <Southernmost_Latitude></Southermost_Latitude> A product level XML metadata (Dataset Description [DSD]) typical entries are more static contacts, document links, and summary statements e.g., <Summary></Summary> Some “custom” tags added. For example, <file_compression> These are stored as separate databases at the JPL GDAC Master Metadata Repository (MMR) mySQL relational database for search, query, and assembly ESIP Federation Meeting 15-18 July 2008

  6. Metadata Specifications - 2003 FGDC conversion All GHRSST products are transferred to NOAA Longterm Stewardship and Reanalysis Facility (LTSRF) after a period of 30 day NOAA has requirement for FGDC compliant metadata for its products. No data ingested without metadata. A conversion of the FR and DSD records to FGDC is performed on file-by-file basis Mappings from DIF to FGDC done using XSLT style sheets For L2P data multiple files of same product type in same FGDC record. Up to 288 files per FGDC record. ESIP Federation Meeting 15-18 July 2008

  7. Mapping Anomaly Some mappings are not clean. FR record: <Personnel> <Role>Technical Contact</Role> <First_Name>Ed</First_Name> <Last_Name>Armstrong</Last_Name> <Email>ghrsst@podaac.jpl.nasa.gov</Email> <Address> Jet Propulsion Laboratory, 4800 Oak Grove Dr, Pasadena, CA 91109 USA </Address> </Personnel> mapped FGDC record: <address> Jet Propulsion Laboratory, 4800 Oak Grove Dr, Pasadena, CA 91109 USA </address> <city> Jet Propulsion Laboratory, 4800 Oak Grove Dr, Pasadena, CA 91109 USA </city> <state> Jet Propulsion Laboratory, 4800 Oak Grove Dr, Pasadena, CA 91109 USA </state> <postal> Jet Propulsion Laboratory, 4800 Oak Grove Dr, Pasadena, CA 91109 USA </postal> <country> Jet Propulsion Laboratory, 4800 Oak Grove Dr, Pasadena, CA 91109 USA </country> ESIP Federation Meeting 15-18 July 2008

  8. sea_water_temperature (Attribute:depth) Measurements of SST (netCDF CF1.3) SSTskin – SST10m (K) surface_temperature -0.17 0 0.5 1.0 1.5 2.0 2.5 3.0 sea_surface_skin_temperature 10mm 1 mm sea_surface_subskin_temperature 1 m Night-time (or strong winds) profile in red Depth Sea_surface_foundation_temperature 10 m 1. Night-time (or strong winds) profile in red 2. Day time situation, strong solar radiation and light winds ESIP Federation Meeting 15-18 July 2008

  9. CF 1.3 Specification for SST standard names • surface_temperature (SSTint) • sea_water_temperature (SSTdepth) • sea_surface_skin_temperature; K • sea_surface_subskin_temperature; K • sea_surface_foundation_temperature: K ESIP Federation Meeting 15-18 July 2008

  10. Future product development • NetCDF version 4 • Benefits • Internal compression and chunking (from HDF storage model) • Support for Parallel I/O • Backward compatibility with v3.6 • The lure of “new stuff” • Advantages must be weighted against a format that has proven metrics and use • Operationally it is difficult to implement a new file format • The user may (will) complain. New software and libraries to install etc. • Evaluated this summer, with a decision by next GHRSST Science Team Meeting June 2009 for adoption ESIP Federation Meeting 15-18 July 2008

  11. Future metadata requirements • European Space Agency will require metadata for next generation Sentinel satellite derived products to be ISO 19115-2 compliant. This directly impacts GHRSST. • Also required for using GHRSST products in WMO Integrated Global Observing Systems (WIGOS) systems (a GEOSS component) • What to change in our metadata production system ? • We have an investment in something that works. We can’t break everything and start from scratch. • The way forward is likely to generate mappings to ISO 19115 similar to FGDC conversion. • Currently reviewing our metadata architecture (FR and DSD XML records) to see what needs to be added/modified • Review must be completed in the next months. A written protocol by the end of the year. • What part of ISO 19115 to conform to?? North American profile…Remote Sensing extensions. • ISO metadata embedded in netCDF v4 GHRSST file ?? ESIP Federation Meeting 15-18 July 2008

  12. Summary • GHRSST has demonstrated significant achievements with regard to data and metadata standards on a relatively modest budget • Although focused on SST discipline, is serves as a model for a working GEOSS • Supporting Science and Applications through well thought out and implemented best practices • One of the challenges will be to evolve with new technologies wrt to format and metadata standards • ISO 19115 • Relatively straightforward once the mappings are determined • netCDF v4 • More challenging. More investigation needed. • Others: Search, Discovery, Imaging • THREEDS: data cataloguing • JPL Datacasting: RSS-like data subscription • Swath granule subset • ECHO metadata inventory • OPeNDAP • WMS POET ESIP Federation Meeting 15-18 July 2008

  13. Backups ESIP Federation Meeting 15-18 July 2008

  14. (A)ATSR reduces uncertainty in global average SST (John Kennedy, Nick Rayner) ESIP Federation Meeting 15-18 July 2008

  15. GDAC overview ESIP Federation Meeting 15-18 July 2008

  16. GHRSST-PP L4 products (A. Harris) • Polar Orbiting infrared has high accuracy & spatial resolution • Geostationary infrared has high temporal resolution • Microwave Polar orbiting has all-weather capability ESIP Federation Meeting 15-18 July 2008

  17. AnalysisProducts Datamerging and analysis Add ancillary QC SSES& dynamic flags netCDF format netCDF format SST Obs. Products Input SST data of Varied format& with no errors L3 SSTObservations Applications The GHRSST-PP Strategy SST system quality control and uncertainty estimation L1P Radiance data ESIP Federation Meeting 15-18 July 2008

  18. Main data flow HR-DDS data flow MDB/SSES data flow RAN data flow GMPE data flow L2P/L2Pc SST data products In situ data Satellite SST data Auxiliary data GDS-v2.0 Rev 0.1 June 2008 MDB/SSES system Derive Single Sensor error statistics (SSES) Ingest & QC input data streams Satellite and in situ MDB Single Sensor Error Statistics Generate L2P data products (L2Pc -> L2P) Reference data fields MDB records Generate L3Pdata products Generate MDB records L3P SST data products Reference data fields HR-DDS system Reference data fields Extract HR-DDS L2P granules HR-DDS archive GDAC &RDAC L2P HR-DDS data granules Generate L4 analysed data products (e.g., SSTfnd) based on L2P data input. Rolling and long-term archive system User data access through LAS/DODS HR-DDS server L4 SST HR-DDS data granules Extract HR-DDS L4 SST granules Reference data fields Auxiliary data Re-Analysis f L2P/L3P/L4 Analysed L4 SST data products Diagnostics, metrics and GDS data product validation GMPE SST ensemble products Generate GMPE products CDR SST data products Metricsdefinitions Reference data fields RAN system GMPE system ESIP Federation Meeting 15-18 July 2008

More Related