Grid-enabling OGC Web Services

1. Grid-enabling OGC Web Services Andrew Woolf, Arif Shaon STFC e-Science Centre Rutherford Appleton Lab

2. Outline • OGC background • Profiling approach • Refactoring approach • Grid-enabling WPS

3. OGC background • Key architectural frameworks • Reference Model for Open Distributed Processing (RM-ODP, ISO 10746-{1,2,3,4}) • ISO TC211 Domain Reference Model (ISO 19101)

4. RM-ODP • Enterprise viewpoint • roles, activities, policies (incl. VO) • Information viewpoint • semantics of information and information processing (static, invariant, dynamic schema) • Computational viewpoint • interfaces and computational objects (cf. CORBA IDL, WSDL portTypes) • Engineering viewpoint • distribution infrastructure (e.g. web services, WSRF vs OGSI) • Technology viewpoint • choices of technology (e.g. app servers, DBMS)

5. …and described by metadata. …in a defined logical structure… …delivered through services… A geospatial dataset… …consists of features and related objects… ISO 19101 ‘Domain Reference Model’

6. registries resource discovery metadata integration catalogue data XML information services WSRF access SOA WSDL OGC ‘Spatial Data Infrastructure’

7. Summary

8. Grid-enabling OGC: profiling • Use existing specifications • Incorporate ‘Grid’ functionality through convention, e.g. • specific parameters for security tokens • Implementation uses Grid middleware at backend

9. Grid-enabling OGC: refactoring • Problems: • tight coupling: service ↔ data • data providers have to be service providers • latency • general approach for asynchronous • service coherence • data → (view, download, process) • workflow abstraction • abstraction: data, operations • service consistency • data resource: WPS vs. W{M,F,C}S

10. Grid-enabling OGC: refactoring • WSRF: • separates ‘service’ and ‘stateful resource’ upon which service acts • WS-Resource (‘implied resource pattern’): • identity (WS-Addressing) • lifetime (WS-ResourceLifetime) • state/properties (WS-ResourceProperties) • publish-subscribe mechanism for state changes (WS-Notification)

11. Grid-enabling OGC: refactoring Existing OGC WSRF-refactored

12. Grid-enabling WPS • WPS provides standardised means of publishing, discovery and use of geospatial processes on the web • Fundamentally, both WPS and Grid provide a remote interface for invoking computational processes

13. Grid-enabling WPS • JSDL provides a standardised description of a computational job and its required resources • Similarities with WPS: • process description information (Identifier, JobIdentification/JobName) • process outputs and inputs (DataInputs, POSIXApplication/Input,Argument,Output, DataStaging/Source) • sufficient to produce a valid JSDL document • Differences with WPS: • resource requirements (filesystem parameters, disk space, operating system, CPU, etc.) • data staging instructions for input and output

14. Grid-enabling WPS • WPS+JSDL as interface to encapsulated Grid resource • JSDL consumed by compliant implementation, e.g. GridSAM • Two broad options: • conceptually distinct data input parameters (through a specific ‘JSDL’ parameter) • “ordinary” data input parameters

15. Grid-enabling WPS • Three approaches to distinct JSDL data input parameters: • URL to a complete pre-created JSDL document, or valid snippet http://foo.bar.1/wps?version=1.0.0&request=Execute&service=WPS&Identifier=WeatherGenerator&DataInput=other_inputs=xxx;JSDL=http://www.foo.com/myfoo.jsdl@Format=text/xml&storeExecuteResponse=true&status=true • URL-encoded JSDL document or snippet http://foo.bar.1/wps?version=1.0.0&request=Execute&service=WPS&Identifier=WeatherGenerator&DataInput=other_inputs=xxx;JSDL=%3CJobDefinition%3E%3CJobDescription%3E%3CResources%3E%3CTotalDiskSpace%3E%3CExact%3E5%3C%2FExact%3E%3C%2FTotalDiskSpace%3E%3CTotalCPUCount%3E%3CExact%3E1%3C%2FExact%3E%3C%2FTotalCPUCount%3E%3C%2FResources%3E%3C%2FJobDescription%3E%3C%2FJobDefinition%3E@Format=text/xml@Schema=http://server/jsdl.xsd&storeExecuteResponse=true&status=true • Microformat http://foo.bar.1/wps?version=1.0.0&request=Execute&service=WPS&Identifier=WeatherGenerator&DataInput=other_inputs=xxx;JSDL=[TotalDiskSpace=5;TotalCPUCount=1]@Format=text/kvp&storeExecuteResponse=true&status=true

16. Grid-enabling WPS • JSDL parameters encoded as individual WPS DataInput parameters http://foo.bar.1/wps?version=1.0.0&request=Execute&service=WPS&Identifier=WeatherGenerator&DataInput=other_inputs=xxx;TotalDiskSpace=5;TotalCPUCount=1&storeExecuteResponse=true&status=true • BUT.... • JSDL parameters are not inputs to the WPS process, but rather specify required resources

17. Grid-enabling WPS • Security • OGC security is subject of considerable ongoing work • for now, used lightweight (and dodgey) solution: • MyProxy parameters embedded in request • secure transport layer

18. Grid-enabling WPS • Implementation: • UK National Grid Service • BADC Trajectory Service • part of OWS-6 testbed • WPS 1.0.0-compliant

19. Grid-enabling WPS Pylons Framework UK NGS Execute Process (SOAP) MyProxy Server Grid-enabled WPS Grid Verify Cert. WPS SOAP/ Proxy Layer GridSAM Service Response Query/ Response Job Id WPS Response GridSAM Client

20. Grid-enabling WPS

21. Conclusions • OGC adopts a sophisticated architectural approach to distributed geospatial data/processing • NB: OGC-OGF MoU, OGF-28 (Munich Mar. 2010) • Grid-enabling OGC: • ‘Profiling’ approach • ‘Refactoring’ approach • Example: grid-enabled WPS profile

22. More info... • OWS-6 WPS Grid Processing Profile Engineering Report • http://portal.opengeospatial.org/files/?artifact_id=34977 • OWS-6 interactive demonstration • http://www.opengeospatial.org/pub/www/ows6/index.html • WPS Change Request for controlling & checking asynch processes • https://portal.opengeospatial.org/files/?artifact_id=36732