OPeNDAP Present and Future

1. OPeNDAP Present and Future An Overview Encompassing Current Projects & Potential New DirectionsDave Fulker and James Gallagher

2. Rough Outline • Background • OPULS (an OPeNDAP-Unidata collaboration) • DAP4 (to supersede DAP2) • Experimental extensions (Async access, UGRID subsets) • Hyrax over Amazon/S3 • Elaboration on server functions • Perhaps binning, masking, a functional language? • Relationship to WPS & other Web services • Hyrax (& WCS) in OWS-9 OPeNDAP, Inc.

3. Origins • Scientists (ocean fluxes & temps) envisaged use of http for remote data access (1993) • Collaboration with the designer of the JGOFS data system… • Led to Distributed Ocean Data System (DODS) • DODS later was renamed OPeNDAP (to be explained momentarily…) OPeNDAP, Inc.

4. OPeNDAP Now Is: • An acronym • “Open-source Project for a Network Data Access Protocol” • Often a synonym for “DAP” • A not-for-profit corp. developing/supporting • “DAPx” - a web-services protocol for data access • Deployed by hundreds of data providers internationally • Employed in many analysis packages (MATLAB, e.g.) • Designated a “Community Standard” by NASA • Server & client implementations* of DAP *Note: there are other implementations

5. Available Software • Free end-user applications that include DAP support: panoply, idv, nco, … • Commercial: IDL, Matlab, ArcGIS • SDKs: The netCDF C and Java libraries; OC; libdap; Java OPeNDAP, PyDAP • Each of these provides its own API and they span C, C++, Java and Python • Data serves: PyDAP, Hyrax, TDS, … OPeNDAP, Inc.

6. Concept: Clients Get Just the Data They Need, as They Need them • Accessing data via URLs (i.e., URL = dataset) • Appending query strings to subset or run server functions • Getting responses of two (general) types: • Metadata - dataset descriptions & catalogs (textual) • Content - values and metadata (binary or textual) • Using responses in diverse ways, e.g. • MATLAB maps responses to its internal math types • netCDF library allows apps to work as though reading a local file

7. NOAA grant forOPeNDAP-Unidata Linked Servers (OPULS) • Goal 1: conformance & linkage between OPeNDAP & Unidata DAP-servers, with short-term outcomes: • New data-model & protocol specs: DAP4 • Consistent behaviors of OPeNDAP & Unidata servers • Data-type richness (NetCDF4, HDF5, RDBs) • Extensions (i.e., new server behaviors): • Irregular-mesh subsetting • Asynchronous access • Goal 2: common framework for OPeNDAP & Unidata servers, aiming for an architecture that • Underpins the unique strengths of both • Reduces likelihood of redundant effort

8. OPULS Progress So Far • Draft of DAP4 data model & protocol specs • Sufficient for the full richness of NetCDF-4 and HDF-5 files (including “Groups,” e.g.) • Progress on rigorous conformance-testing • Successful extensibility experiments • Irregular-mesh (i.e., UGRID) subsetting • Asynchronous access (as may be useful for near-line data storage) • Amazon cloud deployment (more later…)

9. Other technologies OPULS considered • JSON responses as an alternative to XML • Decided they added too much bulk to the specification and two many requirements for implementers • Could be added in a future version • Can be built using XSLT from DAP4 XML • OpenSearch • Not incorporated into DAP4 for many of the same reasons • The DAP4 metadata response specifically includes support for these OPeNDAP, Inc.

10. OPULS and Feedback • OPULS is ready for community feedback • Design documents are online • Web site: http://docs.opendap.org/ • The current draft specification is there as well • Many features are already available in C++ and C implementations OPeNDAP, Inc.

11. Hyrax over Amazon/S3 • Exploits a natural fit between DAP-based services and cloud services • Initial progress already achieved under the OPULS grant • Bears interesting similarities to the challenge of asynchronous data access • May yield a new community of OPeNDAP users OPeNDAP, Inc.

12. More about clouds… • Hyrax is trivial to run on the Amazon cloud • We are looking at ways to work with data held in S3 • S3 characteristics: • Flat; • Modest response times; • Simple GET/PUT type API OPeNDAP, Inc.

13. Using S3 • Tried S3 file systems – found them wanting • Not interoperable (hardly surprising, but limiting) • Extra layer to software stack • Now working with XML ‘catalogs’ • XML documents create a faux hierarchy • XML + XSLT  HTML (i.e., a ‘free’ web interface) • XML + Hyrax + caching  DAP access • The XML is very similar to THREDDS catalogs OPeNDAP, Inc.

14. Elaboration on Server Functions • Proposition: the future of OPeNDAP may lie in provision of data-proximate (i.e., server-side) functions that: • Deliver precisely defined subsets • Reduce the number of off-target retrievals • I.e., enable querying of complex dataset properties • Remap/transform data to simplify data use, especially multi-source data integration • Effective caching will be required OPeNDAP, Inc.

15. Server Functions, DAP4 • DAP2 supports functions and functional composition • Currently, DAP4 treats ‘functions’ and a ‘functional language’ as an extension • DAP4 provides more complete support for functions, including metadata responses (DAP2 does not provide this; a gap in the DAP2 specification) • Support for POST OPeNDAP, Inc.

16. Server Functions, experimentation • UGrid: Unstructured Grid (irregular mesh) subsetting • We have implemented a clone of the GDS server’s syntax for functions • Enables current netCDF-based DAP clients (e.g., ECMF) to use the Ugrid function • Other projects: Multi-instrument inter-calibration OPeNDAP, Inc.

17. Some Server-Function Ideas • Binning: returns a distribution (as a raster of boolean values on a user-specified grid) of data values satisfying some criteria • Masking: accepts a raster of zero/nonzero values as a query argument, perhaps as a geospatial selection criterion, e.g. • Perhaps some (limited?) form of functional language for very rich capabilities • WPS, et al. OPeNDAP, Inc.

18. Summary • DAP is based on a domain neutral data model and an expression-based constraint language • While not ‘RESTful’ in the strictest sense, it is a REST design in spirit (DAP predates the term by several years) • OPULS is a collaborative project between OPeNDAP and Unidata that intends to update DAP • We are also running several experimental mini-projects within its context: • Asynchronous access, Unstructured Grid access, Cloud computing and an expanded, function-based, server-side processing system • DAP servers provide a good platform on which to build OGC web services, as described in the following presentation. OPeNDAP, Inc.