1 / 27

JCOMM/IODE ETDMP Pilot Project The Technology Prototype for

INTERGOVERNMENTAL OCEANOGRAPHIC COMMISSION (OF UNESCO). WORLD METEOROLOGICAL ORGANIZATION. JCOMM/IODE ETDMP Pilot Project The Technology Prototype for the End-To-End Marine Data Management Basic Solution, Development Status and Use for Supporting the Marine Activity

alvaro
Télécharger la présentation

JCOMM/IODE ETDMP Pilot Project The Technology Prototype for

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. INTERGOVERNMENTAL OCEANOGRAPHIC COMMISSION (OF UNESCO) WORLD METEOROLOGICAL ORGANIZATION JCOMM/IODE ETDMP Pilot Project The Technology Prototype for the End-To-End Marine Data Management Basic Solution, Development Status and Use for Supporting the Marine Activity (Nikolay Mikhailov and at all, RIHMI-WDC/Russian NODC, Russian Federation) JCOMM-II, Halifax, Canada, 2005

  2. Contents of Presentation 1. JCOMM/IODE ETDMP & E2EDM 2. E2EDM Pilot Project 3. E2EDM Technology Concept 4. E2EDM Data Model and Components 5. Current Status 6. The Future Use of the E2EDM

  3. JCOMM/IODE ETDMP Expert Team for Data Management Practices • ETDMP was organized by IOC/WMO JCOMM (JCOMM-1, 2001) • IODE GTADE have been merged with JCOMM ETDMP (JCOMM MAN, 2002; IODE-XVII, 2003) The Term of Reference • recommend and implement of end-to-end data management system for JCOMM/IODE • review and assess the effectiveness of data management practices, recommend best practices • The ETDMP Pilot Projects for 2003-2005: • Metadata Management • Data Assembly, Quality Control and Quality Assurance • End-to-End Data Management Prototype

  4. Distributed and discionnected Because of Incomplete Semantically Inconsistent Syntactically Incompartible Because of Because of Because of Large number of collections/flows of different organizations Lack of standards for specific domains Different world perceptions Variety of software and formats Why we need to use E2EDM ? Difficult to find needed data in time Because of Integration Issues

  5. User DecisionMaking Processing/product generation I n t e g r a t i o n Archiving DM Problem Definition Assembling DM Covering DM Scheme Collection E2EDM Functions Definition DM Properties End-to-End Data Management Strategy • The E2EDM system is the combination: • the best data management practices • the integration technology as an “umbrella” which provides “transparent” interaction between data sources resulted from these DM practices

  6. E2EDM PP Goal The goal of E2EDM PP – to develop a prototype of a web-oriented technology (E2EDM prototype) to: • provide real-time data fusion from distributed local data systems as an aggregated “virtual” data set • provide access and value-added products to JCOMM/IODE users • demonstrate "end-to-end" data management opportunities.

  7. E2EDM PP Scope E2EDM Prototype provides functionality for: • local data systems “wrapping” by E2EDM services • data extraction from distributed sources on user requests • on-line aggregation and visualization • creation of new products based on the obtained data E2EDM Prototype integrates the data: • at operational and delay-mode time scale • across multidisciplinary ocean and marine meteo datasets/flows • from geographically distributed data sources existing in different formats (codes, dictionaries and etc.)

  8. E2EDM Project participants • RIHMI-WDC/RNODC, Russia • VLIZ, Belgium • IFREMER, France • Met Office, UK • MEDS, Canada • NODC, USA

  9. E2EDM PP Parameters • Marine meteo: air temperature, sea surface temperature, pressure, wave height and wave direction, wind speed and wind direction • Ocean:temperature, salinity, oxygen, nitrite E2EDM PP data sources • historical marine meteorological data (MCSS) – Met Office • historical ocean cruise data – RIHMI-WDC, VLIZ, NODC • real-time GTS SHIP data –RIHMI-WDC • real-time GTS BATHY and TESAC data –RIHMI-WDC, MEDS • Delay-mode GTS TESAC/ARGO data - IFREMER • monthly climatic fields of ocean parameters – USA • analysis data from GTS (sea surface temperature or wave) - RIHMI-WDC (GTS) • ocean SST satellite data (imageries) – RIHMI-WDC (Internet)

  10. Key Definitions E2EDM Object - any entity, the data on which are used for implementation of E2EDM functions Local Data Systems – DBMS, structured data files, non-structured data files of an organization (centre). Data Source - an abstraction providing the interface between a local data system and the E2EDM system. Data source is implemented by Data Provider. Information Resource – a data(metadata) set generated by a data source from the local data system and available for the E2EDM services Services - software implementing a wide range of data handling, processing and representation functions E2EDM Soft Components – Data Provider and Integration Server to provide system services Data Exchange Protocol - specifications regulating exchange between local data systems and E2EDM services, and between services. It consists of data request and response messages, transport data file

  11. USER Integration Server Data Provider Data Provider Integration Server Data Provider Data Provider Local System Local System Local System Local System E2EDM Technology Concept

  12. Existing systems analysis The tools was considered that: • provide operations for marine data • have properties of scalability and extension both in data domain and in functionality • have practically tested components for data integration from distributed data sources • involves common metadata standards • The following technologies/systems/standards were analyzed: • Grid technology • DiGIR (Distributed Generic Information Retrieval) • DODS/OPeNDAP • ESIMO (Russian Distributed Data System) • Sea-Search metadata • WMO Core Metadata • NDG (NERC Data Grid) • ISO 19115 standard

  13. Systems/standards used in E2EDM Prototype • Grid technology – metadata management concepts • DiGIR – request/response protocol, database access service, basis for the E2EDM Concept XML Schema • NetCDF – format for “transport” data files • ESIMO –navigation services, data search mechanism and visualization tools, data file access service • NDG – object segmentation ideology, distributed concepts • Sea-Search CDI (Common Data Index), WMO Core Metadata and ISO 19115 as the basic standards for the E2EDM Global XML Schema

  14. E2EDM Prototype Specifications E2EDM namespace is handled by the E2EDM global XML Schema, which is the information core of the E2EDM technology • E2EDM data model. • Objects, classes of element, metadata and data records for • discovery • encoding • delivery • presentation • E2EDM Namespace • Global XML Schema • Concept XML Schema • Data Exchange Protocol • Request/response messages • Data transport format The E2EDM concept (federal) XML Schema includes the list of semantic elements of metadata and data which are used in the exchange between E2EDM services

  15. E2EDM Prototype – Architecture: Integration Server level

  16. E2EDM Prototype – Architecture: Data Provider level

  17. 1. Goal&scope Virtual Centre Virtual Centre Comparison of E2EDM and WMO GISC Prototypes Features E2EDM GISC 2. Middleware EJB EJB 3. Metadata 7 Metadata records Metadata pool 4. Data 3 Data records, Transport File ? 5. Management Integration Server, Data Provider Portal-Server, Data polls

  18. Current Development Status E2EDM data model:Objects, Classes, Records, Data Sets and their interconnection Specifications: 24classes of elements, 7 metadata and 3 data records Namespace: Global XML Scheme and Concept XML-scheme Transport data file: Specifications for DataPoint, DataProfile and DataGrid oriented exchange Software. E2EDM Data Provider (DAS@FAS), E2EDM Integration Server, End-User Application E2EDM Web-site: http://data.meteo.ru/e2edm/

  19. Current Operational Status Ocean WA2001 data Data Provider (VLIZ) Data Provider (MEDS) RIHMI-WDC mirror Integration server Integration server Data Provider (NODC, USA) IODE Project Office Data Provider (RIHMI-WDC) • Ocean cruise data • GTS SHIP data • GTS BATHY/TESAC data • Climate Ta/Tw fields • Satellite imageries Data Provider (IFREMER) Data Provider (MetOffice) operational Tesac/Argo data planned Marine meteo data

  20. E2EDM Operation Examples/1 Data Source List E2EUserD E2ESearchMD Parameters E2ELinkMD Agregation/presentation – Integration Server Request Message Response Message Config_File Transport File Wrapping/selection – Data Provider

  21. E2EDM Operation Examples/2 • Search criteria • Data Source • Parameter • Geoarea • Date/time • Request realization • Data Source’s selection • Connection with Data Providers • Local data processing • Transport file forming • Response and transport data file transportation Request status control

  22. E2EDM Operation Examples/3 – Data View • E2EDM Web site http://data.meteo.ru/e2edm/ • E2EDM Integration Server - IODE Project Office (VLIZ) http://e2edm.vliz.be - RIHMI-WDC (mirror) http://data.meteo.ru:8080/iserv/

  23. E2EDM Current Problems • Data duplication and UDI • Improving (simplifying) the installation procedures • Data Transportation Time – 1 000 records ore 2 minutes: caching, DiGIR new version, etc. • Openness – expanding the Concept XML-schema • Cross-Mapping – optimizing scheme for local data encoding to E2EDM Codes by Config_File modification • Resource Instances – linkage of the search metadata to data and thematic metadata on low granularity levels (observation and profile series) • End – User Interfaces – typical forms, GIS, plot-table viewing • ….. and ……………and………….and………..

  24. I can note that even in binary file case the time of connections between Data Providers and Integration Server is very big (see table) and we now combat this issue. E2EDM Operation Time

  25. E2EDM in Future Integration Server Integration Server Integration Server Establishing • Installations of E2EDM Integration Server – IODE Project Office • Installations of E2EDM Data Provider software by IODE/JCOMM centres • Configuration file preparing and registration of data sources • Management of system by Integration Server Adjustment and use • User applications for a various applied tasks • Search and involving the new data sources • Data sources federations (E2EDM releases)

  26. E2EDM use for marine activities by implementing the applied services • IODE/JCOMM Actual Data sets/Flow monitoring: • Check the actuality and content of the IODE Data Centres resources • Preparing the current map (reports) of IODE data sources. • Providing GOOS/GCOS/COOP users (modelers): • Establishing the data sources (centres) groups on basis on E2EDM release concept • Delivery the required data sets for Modeling Centre(s) on constant basis. • Providing end-users: • complex information – from different disciplines, observation/ climate/forecasts, data+metadata, • tools for integrated data use – current ocean state – climate; checking phenomena using historical data and etc. • Regular and standard marine environment assessments –GMA, GEOSS

  27. THANK YOU

More Related