The US Integrated Ocean Observing System (IOOS) Plan for Data Management and Communications (DMAC)

1. The US Integrated Ocean Observing System (IOOS)Plan forData Management and Communications (DMAC) IOOS DMAC Steering Committee Integrated Ocean Observing System Data Management and Communications

2. RegionalData ManagementSystems Products InternationalData ManagementSystems Users Maps Forecasts Terrestrial and AtmosphericData ManagementSystems On-line Browse Archive Centers Modeling Ships Hand Measurements Satellites Floats IOOS Data Management andCommunications Subsystem Primary DataAssembly & QC Moorings Metadata, Data Discoveryand Data TransportStandards and Protocols

3. Ships Hand Measurements Satellites Floats IOOS Data Management andCommunications Subsystem Primary DataAssembly & QC Moorings RegionalData ManagementSystems Products Users InternationalData ManagementSystems Maps Forecasts Terrestrial and AtmosphericData ManagementSystems Metadata, Data Discoveryand Data TransportStandards and Protocols On-line Browse Archive Centers Modeling

4. DMAC Timeline – Creating the Plan Requirements, schedules, & costs publicreview mtg DMAC Steering Committee Convened “Expert”review June Sep Dec Feb May Sept Mar June 2002 2003 1st Draft Plan Airlie workshop • -Metadata • -Transport • -Archive • -Products Expert team papers Publish DMAC Plan Integrated Ocean Observing System Data Management and Communications

5. What makes this a difficult problem? Like the classic “trilemma” … We can’t have all three Integrated Ocean Observing System Data Management and Communications

6. …competingcharacteristics rule out the most straightforward solutions Integrated Ocean Observing System Data Management and Communications

7. The DMAC Plan is a framework for interoperability among independent, heterogeneous systems, large and small • Participating organizations will manage their data independently Integrated Ocean Observing System Data Management and Communications

8. The methods utilized by data suppliers to deliver data to their “customers” need not change(FTP, subscription services, etc.) Guiding Principles Integrated Ocean Observing System Data Management and Communications

9. However, a data provider may choose to utilize DMAC middleware as a cost effective way to reach a wide range of users. Guiding Principles Integrated Ocean Observing System Data Management and Communications

10. Why doesn’t DMAC exist today? The greatest challenges are in the area of community outreach and organizational behavior rather than in technology! DMAC requires participation from • All levels of management • Scientists • Technical staff Integrated Ocean Observing System Data Management and Communications

11. The DMAC Plan • An evolutionary path • A software engineering strategy • An inter-agency/international outlook • Three parts • Part I – overview for general readers • Part II – formal requirements, milestones, tasks for technical planners • Part III – in-depth appendices on special topics Integrated Ocean Observing System Data Management and Communications

12. The DMAC Plan • Technology components • Data Discovery (metadata) • Data Transport • On-line Browse • Archive • Metrics, feedback and fault correction Integrated Ocean Observing System Data Management and Communications

13. The DMAC Plan • Data Discovery (metadata) • Utilize FGDC (& future ISO) • A partnership of current and new metadata repositories • Metadata WG to design the strategy for distributed search • A Web Portal at which users can search • Embed in other Web pages • Machine-to-machine accessible • Seamless segue to DMAC On-Line Browse and Data Transport Integrated Ocean Observing System Data Management and Communications

14. The DMAC Plan • Data Transport (access) • “Web Services”  standardize the interface to the data • Adopt current successes • OPeNDAP (a.k.a. “DODS” or “NVODS”) • Discipline neutral • Potentially interoperable with GIS components • Adapts to current and future computer applications • WG must define comprehensive semantic data model(s) Integrated Ocean Observing System Data Management and Communications

15. The DMAC Plan • On-line Browse • Uniform access to all IOOS data from Web browsers • maps (“GIS”) & basic scientific graphics • readable tables • Target audience is IOOS scientific managers(also useful to many others) • Utilize Data Transport • Live Access Server (LAS) is an initial “pre-operational” component Integrated Ocean Observing System Data Management and Communications

16. The DMAC Plan • Archive • Build upon partnership of National and Regional archive centers • Archive WG to be formed • establish DMAC archive policies • ensure that all data have an archive plan • Utilize DMAC Metadata and Transport standards • (new) Operational data plus QC to be archived(in coordination with data assembly centers such as US GODAE server and NOAA/NDBC) Integrated Ocean Observing System Data Management and Communications

17. The DMAC Plan • Data products(*)(IOOS Modeling and Analysis Subsystem) • State estimation • Forecasts • Information products(*) • Value-added products for user groups provided by the private sector, academe, government * Not part of the DMAC Plan Integrated Ocean Observing System Data Management and Communications

18. The DMAC Plan • Feedback and Fault Correction • DMAC Web portal will invite user feedback • DMAC to maintain a “Help desk” • Performance metrics gathered automatically • Oversight by the DMAC Steering Committee • Continual, active outreach to users Integrated Ocean Observing System Data Management and Communications

19. Implementation of IOOS has already begun • New funding possible by FY06 • Key pieces of DMAC must be in place by that time • Interim metadata standards (incl. semantic data model) • Distributed data discovery architecture • Voids to be filled in the data transport toolkit • Initial responsibilities clear among Archive centers • Must fund these developments using current mechanisms! (See details in Part I, Section 4 of DMAC Plan) Integrated Ocean Observing System Data Management and Communications

20. DMAC Phased Implementation • Immediate Priorities • Metadata Standards (W.G.) • GIS&OBIS Data Transport integration • Archive strategy (W.G.) • Software engineering oversight • Five year implementation Plan • Standards and Tool Development ($30M) • Hardware & Software for networking, data assemb. & archive ($37M) • Pilot projects ($14M) Sept 2003 $2.3M $81M ‘04 ‘05 ‘06 ‘07 ‘08 ‘09 ‘10 DMAC Planpublished minimally functioning initial system IOC Integrated Ocean Observing System Data Management and Communications

21. NVODS and DMAC • Use of OPeNDAP (NVODS) was not a guaranteed outcome of the DMAC planning process • It was selected for DMAC Data Transport because the NVODS project anticipated the DMAC analysis of the IOOS “trilemma” – the need for a “framework for interoperability”. Integrated Ocean Observing System Data Management and Communications

22. NVODS and DMAC OPeNDAP must be made ready for the FY06 rapid expansion of IOOS. • GIS compatibility • OBIS (biological) compatibility • delimited files (ASCII) • OPeNDAP Development Kit (ODK) – an easily adaptable server core Integrated Ocean Observing System Data Management and Communications

23. NVODS and DMAC • A semantic data model • need leadership from NVODS • A stable, published spec (v4 -- XML) • Expanded resource control framework • Restrict data volumes, cpu cycles, privs, … • Expanded “web service” infrastructure • Query resource limits Integrated Ocean Observing System Data Management and Communications

24. NVODS and DMAC • Improved Sequence data support“pilot”  “pre-op” • additional client options • perl, python, “translation”, jdbc (?), … • usability conventions (metadata) • min/max ranges, size guidelines, … • much more hands-on experience • further forms of aggregation • in-situ collections of profiles and time series, 5D and 6D forecast ensembles Integrated Ocean Observing System Data Management and Communications

25. NVODS and DMAC Hard work remains to be done, but with a sustained effort NVODS/OPeNDAP can succeed Integrated Ocean Observing System Data Management and Communications

26. Questions? Click for immediate actions recommended for IOOS data providers Integrated Ocean Observing System Data Management and Communications

27. Concrete actions to contribute data into the IOOS/DMAC framework today • Data discovery • Produce FGDC metadata • Submit it to national search sites(NASA/GCMD, NOAA/NCDDC & CSC) • Create a home page and a search portal to your data holdings • Interoperable data access • On-line browse Integrated Ocean Observing System Data Management and Communications

28. Concrete actions … • Data discovery • Interoperable data access • Gridded data: • Install OPeNDAP(*) servers • Time series: • Consider partnership with NOAA/NDBC(**) • In-situ data: • Use RDBMS or standard formats • Contribute to development of pilot IOOS/DMAC components – “DRDS”, “A.S.” • On-line browse (*) http://www.unidata.ucar.edu/packages/dods/ (**) Dave.Gilhousen@noaa.gov , (228)-688-2840 Integrated Ocean Observing System Data Management and Communications

29. Concrete actions … • Data discovery • Interoperable data access • On-line browse • Install Live Access Server(*) utilizing DMAC/OPeNDAP data transport • Create custom Web sites that demonstrate regional data integration (*) http://www.ferret.noaa.gov/LAS/ Integrated Ocean Observing System Data Management and Communications

30. Questions? Integrated Ocean Observing System Data Management and Communications