NASA CEOP Final Summary CEOS WGISS-26 Boulder, Colorado September 23, 2008

1. NASA CEOP Final SummaryCEOS WGISS-26Boulder, ColoradoSeptember 23, 2008 Yonsook Enloe yonsook@mindspring.com

2. NASA CEOP Project Background Purpose of NASA CEOP Prototype is to improve CEOP science community access to satellite data and to enable • Efficient access to satellite, model output, and field data • Inter-comparing values of the different data types • emphasis on reference sites, specific monsoonal regions, specific observation periods Approach: Enable OPeNDAP client access to WCS Servers to facilitate access to satellite data • OPeNDAP Protocol for data access • Includes subsetting • Endorsed as a NASA Earth Science Standard • Server Implementation: Hyrax • OPeNDAP Clients • Especially popular among modelers • Significant time investment in tools of choice

3. CEOP Data Access Prototype Approach: Enable existing tools to access data from Web Coverage Servers Analysis / Display Tools GrADS User Live Access Server Ferret OPeNDAP request Web Coverage request WCS Server WCS Handler georectified spatial subset georectified NetCDF OPeNDAP Server IDV NCO Grid Data IDL Swath Data Matlab

4. Project Accomplishments • Developed design for the CEOP Satellite Data Server • Implement as “handler” module for type “wcs” within the OPeNDAP Hyrax architecture • Enables additional OPeNDAP clients accessing WCS servers • Developed converter for HDF-EOS to NetCDF (CF-1) in WCS Server • Applied to NASA for public release of the WCS handler in the Hyrax server software – NASA approval any day now • Completed initial testing with CEOP scientists • Prototyped data provenance (lineage) - imbedded lineage info with data payload Any WCS Server that can return a NetCDF CF or HDF-EOS CF response can be linked to the Hyrax OPeNDAP Server and accessed by OPeNDAP enabled clients Any OPeNDAP enabled client may access any of the WCS Servers whose holdings are registered in the Hyrax Server via the THREDDS catalog configuration.

5. Future Uses of NASA Prototype • OPeNDAP Hyrax Server (v 1.4.2) is available for pubic download. The WCS handler component will be released soon. • OPeNDAP Group will continue to maintain and provide access to the Hyrax Server. • Any organization can download the Hyrax server and install a WCS Server that returns CF-1 compliant NetCDF or HDF files. Use the “Lessons Learned From WCS Server Design and Implementation” document to provide info about installing a WCS Server with satellite data.

6. WCS Server Lessons Learned “Lessons Learned From WCS Server Design and Implementation” document • Need to understand the limits of the client analysis software (e.g. Grads) to communicate with the Hyrax Server (e.g. url length limit, special chars excluded, etc.) • Coordinate Reference System Transformation capability is necessary (e.g. swath to earth coordinate system). Many methods to do CRST. CRST is product & parameter specific. Parameters in WCS interface not sufficient to define the difference among products. Limits to spatial and temporal extents often need to be applied due to performance consideration. • Need to offer mosaicking of multiple file granules to offer virtual coverage not tied to any one single physical file. Often a user’s area of interest exceeds one file. This leads to multiple requests from the client and the need for client to mosaick the granules together. • Time stamps – observation time for each point on grid not the same. Additional data array needed to record the observation time for each grid point in the returned coverage. But need to discuss with user how to convey the timestamp and how to group data according to time (e.g. day/night).

7. WCS Lessons Learned (cont) • Quality screening is product specific and difficult to define QA parameters via WCS interface • Performance and scaling – with virtual coverage offering, scalability strongly driven by client request. • Database/directory structure and informative url – users expect to see info such as variable type, spatial & temporal extents, etc. in the url path name Document will be submitted to the NASA Standards Process Group as a Technical Note. Plan to enquire at OGC – if this document is of interest to the OGC community Potential submission to the GEOSS Best Practices Registry

8. Provenance Info Explored “Provenance within Data Interoperability”- document • Retrospective vs prospective – choose retrospective- document operations on data as they occur • Embedded vs out of band – embed provenance within the data payload • Standards – lack of single definitive standard; presents a challenge for client developers to use provenance info; we are formatting the lineage info using ISO 19115 schema but include it within the History element in CF-1. • WCS Server – record any of 3 processing steps: (1) swath scan/frame CRS to WGS84 lat/long CRS transformation; (2) interpolation; (3) per-pixel quality screening • Producing an example script for Grads client – script will (1) read in the History element and output it to a file; (2) parse the external history file for the original data urls and fetch the data Document will be submitted to the NASA Standards Process Group as a Technical Note.

9. OGC-Geoscience Gateway Project Background • THREDDS • Provides catalogs of datasets served via a variety of protocols • Geoscience community protocols: http/netCDF, OPeNDAP, ADDE (Abstract Data Distribution Environment - I.e. McIDAS) • OGC protocols: WCS (Web Coverage Service) • Depends on other organizations and protocols for discovery and search of the data • OGC Geoscience gateway complements THREDDS by: • ingesting the information in THREDDS catalogs • providing a standards-based (i.e., OGC) discovery interface that enables users to search for data available on distributed THREDDS servers

10. OGC-Geoscience Gateway

11. Project Accomplishments (Geoscience) • Designed THREDDS to ISO 19115/19115-2 mapping • Implemented THREDDS to CS/W metadata ingestor • Ingested the Unidata THREDDS catalog (weather forecast model) and the NCDC THREDDS catalog • Implemented OpenSearch gateway to the CS/W Server • Worked with the GI-GO client team (Italy) and the ESRI client team to enable their clients to successfully search the GMU CS/W Server. • Working with the Gi-GO client team to agree on how the user should access data held by WCS server after a catalog search

12. CS/W & WCS Interoperability “Interoperability Between OGC CS/W and WCS Protocols” document • CS/W Server implementation & ingesting THREDDS metadata into CS/W Server; Ingest some Thredds metadata into collection level metadata and some into product/file level metadata. • CS/W Profile proliferation – many metadata profiles of CS/W; A catalog system implementing CS/W interface must also implement a metadata profile. We chose ISO 19115 Profile. • Deep dataset hierarchies; Thredds catalogs are available at various levels – different datasets, different models; output options for specific model; file access options. Which level catalogs should be harvested for the discovery in the CS/W search? • Freetext search – CS/W search/response protocol is highly structured with multiple fields having searchable text. Which fields should be searched by the client when presenting a simple freetext field to the client? ESRI client only searches the title for freetext search; GI-GO client searches multiple fields. The same keywords in two separate clients may generate different results even when submitted to the same server.

13. CS/W & WCS Interoperability (cont) • Last Mile Problem – Ultimate goal is to return a search result that a client can use to issue a WCS request for data. Should be transparent to user without cut & paste of urls. We inventory at a coverage level so that a client can use a GetCoverage request. Enrico Boldrini & Lorenzo Bigagli (Gi-GO client team) suggested that sufficient info be provided in the fields of the ISO 19115 CI_OnlineResource element to allow the client to issue a specific direct GetCoverage request without having to use additional information. This is a less explored area in the use of OGC standards. • Plan to submit document to the NASA Standards Process Group as a Technical Note • Plan to enquire through Galeon project about OGC interest in this document • Potential submission to GEOSS Best Practices Registry

14. Software/Info Available • Hyrax OPeNDAP Server software from the OPeNDAP Group • Documents: • “Lessons Learned From WCS Server Design and Implementation” • “Provenance Within Data Interoperabilty Standards” • “Interoperability Between OGC CS/W and WCS Protocols” Please contact yonsook@mindspring.com for more information