430 likes | 446 Vues
Focus on data integration for modelers, partnership for better management, use of metadata standards, and enhancing data accessibility. Explore the importance and challenges of data integration in environmental science modeling.<br>
E N D
Integrated data management in the ESMF (ESME) Steve Hankin(NOAA/PMEL & IOOS/DMAC) ESMF Team meeting July 2004
The growing importance of data integration to modelers: We no longer fund “modeling”. Today we fund “climate prediction” or “coastal processes” … the science topics. Modeling is just a component. a program manager (anonymous)
Model outputs need to be made useful to many classes of users education research community modeling community project model run
Data and products need to be made more usable for modelers Real time and delayed-mode observations assimilation data products modeling boundary & initial conditions validation comparison
An ESME must include a plan for data management. But, how ? Funds are limited …
Partnership: A community of data managers has formed … GO-ESSP – Global Organization of Earth System Science Portals(http://esportal.gfdl.noaa.gov) • Unidata • ESG (NCAR, LLNL) • OPeNDAP (a.k.a. ‘DODS’) • COLA • NOMADS (GFDL, PMEL, NCDC, NCEP) • NASA/GCMD • BADC, BODC • WMO • …
Ocean data systems following similar approaches … • National Virtual Ocean Data System (NVODS) • US Integrated Ocean Observing System • GODAE (US and International) • OCMIP, AOMIP, …
Workflow Data portal components: • Data discovery • Data access/transport • On-line browse and comparison • (Segue to analysis)
Data discovery == Metadata + search Mature standards do not exist today. A task for ESMF: define and utilize metadata standards for modelers
Metadata for modelers • Reviewed & “published” (a “standard”) • Structured (XML) • Generated automatically in conjunction with setting up model runs • Standardized parameter names (“controlled vocabularies”) • Hierarchical • components, grids, fields and attributes
Data discovery == Metadata + search Others are working hard on search: • Traditional metadata partners(e.g. GCMD) • Semantic Web (Google on steroids)(3 years off?)
Workflow Data portal components: • Data discovery • Data access/transport • On-line browse and comparison • (Segue to analysis)
Data transport OPeNDAP (a.k.a. “DODS”) • Network data access • Format-independence • Subsetting • Aggregation (GDS, Unidata) • Compression • Security: Grid-enabled OPeNDAPg
CF (“climate and forecast”) CF 1.0 is now a “standard” • “use” metadata – e.g. units, coords. • curvilinear, hybrid-Z, time-dependent • great applicability beyond modeling, too Discussion question: As the use of the CF standard widens how should the community support it?Not enough to “endorse” it. Need a partnership.
Workflow Data portal components: • Data discovery • Data access/transport • On-line browse and comparison • (Segue to analysis)
LAS -- an “Information Product Server” Ferret, CDAT or other • Metadata (XML) contains the “intelligence” • Back end applications do the real work • OPeNDAP provides remote data access
Uniform data access Informationaccess
Workflow Data portal components: • Data discovery • Data access/transport • On-line browse and comparison • (Segue to analysis)
discover browse access Metadata Standards NASA Global Change Master Directory (GCMD)
Collaborating groups of modelers LAS “sisters” share metadata to form a unified (virtual) site.OPeNDAP allows LAS to difference distributed fields.
A wealth of data products are available through the National Virtual Ocean Data System (NVODS)
Access to observations (WODB -- 9 million ocean profiles)
“Batch” access to products Query available data sets: Query variables in data set “model_1”: Query space-time domain: Request a subset of data as a file: (“asc” for ASCII format) >lasls http://cpu/LAS >lasls http://cpu/LAS model_1 >lasls http://cpu/LAS model_1 sst >lasget -x 20:60 -y 20:60 -t 11-Dec-2000 -f asc http://cpu/LAS model_1 sst
IOOSUS Integrated Ocean Observing System • Detect and Predict Change • Mitigate natural hazards • Improve safety and efficiency of marine ops • Ensure national security • Reduce public health risks • Protect and restore marine ecosystems • Sustain marine resources
RegionalData ManagementSystems Products InternationalData ManagementSystems Users Maps Forecasts Terrestrial and AtmosphericData ManagementSystems On-line Browse Archive Centers Modeling Ships Hand Measurements Satellites Floats Primary DataAssembly & QC Moorings IOOS Data Management andCommunications Subsystem Metadata, Data Discoveryand Data TransportStandards and Protocols
Ships Hand Measurements Satellites Floats Primary DataAssembly & QC Moorings IOOS Data Management andCommunications Subsystem RegionalData ManagementSystems Products Users InternationalData ManagementSystems Maps Forecasts Terrestrial and AtmosphericData ManagementSystems Metadata, Data Discoveryand Data TransportStandards and Protocols On-line Browse Archive Centers Modeling
Recommendations for ESMF • Define ESMF metadata standard and use it • Consider a GCMD modelers’ portal, too • Endorse partnership with GO-ESSP and the emerging tools • Data available through OPeNDAP(g) • Live Access Servers for on-line collaborations
Questions? NVODS LAS:http://www.ferret.noaa.gov/nvods GCMD “DODS” Portal:http://gcmd.gsfc.nasa.gov/Data/portals/dods