Grid interoperability developments at CC-IN2P3

1. Grid interoperability developments at CC-IN2P3 Lyon, 7 September 2010 Sylvain Reynaud

2. Plan • JSAGA • the SAGA specification • the JSAGA implementation • example applications • other grid interoperability developments at CC-IN2P3 • JJS • JUX • JPySAGA • perspectives JSAGA

3. SAGA is a specification standard recommendation of object oriented simple 80/20 rule: 20% of the features cover 80% of the needs integrated ("consistent") common "look & feel" for all the functional packages (see slide) high-level hide low-level details about middlewares middleware-independent use the same API to access all middlewares language-independent Java C++ python (wrappers) stable initial recommendation SAGA 1.0 (15 January 2008) a 1st update soon (backward compatible) extensible Service Discovery, Message… Simple API for Grid Applications JSAGA

4. Simple API for Grid Applications JSAGA Source: S Jha, T Kielmann, A Merzky

5. The JSAGA implementation • JSAGA is an implementation of SAGA • written in/for Java • focuses on • uniform usage of middleware…. (beyond what they natively support) • ease of extension………………. • efficiency and scalability………. • control of the API behavior........ (configuration rather than attempts) • operating-system independency (tested on ) • Under LGPL license (thanks to the design of adaptor interfaces) JSAGA

6. Uniform usage of existing grids JSAGA

7. out.txt out.txt JSAGA GridFTP Example – uniform usage SAGA job SAGA job user applications WMS gsiftp Gatekeep. Globus VOMS JSAGA

8. DGrid proxy adapt + translate submit job receive status transfer to any protocol EGEE proxy + attributes translate submit job poll status delegate transfer out.txt out.txt EGEE D-Grid JSAGA Example – what happens behind SAGA job user applications modified job wrapper script WMS gsiftp Gatekeep. Globus VOMS JDL RSL VOMS WMS GridFTP Globus GK Unicore LCG-CE CREAM-CE JSAGA

9. JSAGA interface Layered software architecture implementation JSAGA

10. Used by end users web portals, GUI, CLI… Used by application developers a single uniform interface object oriented, high-level Implemented by adaptors each "way" to implement a given feature has 1 interface service oriented, low-level Implemented by middleware each technology has its own interfaces SAGA Java binding core engine JSAGA adaptors interfaces JSAGA adaptors JSAGA interface Layered software architecture implementation user applications middleware APIs JSAGA

11. JSAGA adaptor interfaces adaptor adaptor GridFTP Globus GK Layered software architecture • Used by end users • web portals, GUI, CLI… • Used by application developers • a single uniform interface • object oriented, high-level • Implemented by adaptors • each "way" to implement a given feature has 1 interface • service oriented, low-level • Implemented by middleware • each technology has its own interfaces user applications transfer output listen status changes JSAGA core engine upload data get output get status JSAGA

12. adaptor adaptor Layered software architecture • Used by end users • web portals, GUI, CLI… • Used by application developers • a single uniform interface • object oriented, high-level • Implemented by adaptors • each "way" to implement a given feature has 1 interface • service oriented, low-level • Implemented by middleware • each technology has its own interfaces user applications transfer output listen status changes JSAGA core engine JSAGA adaptor interfaces GridFTP Globus GK upload data get output get status JSAGA

13. adaptor adaptor Layered software architecture • Middleware API can be used more efficiently • engine selects best interface for each user request • Developing new adaptors is easier • just wrap functionalities supported by middleware API user applications transfer output listen status changes JSAGA core engine JSAGA adaptor interfaces GridFTP Globus GK upload data get output get status JSAGA

14. Minimal dependencies on external libraries external tools e.g. no gLite-UI operating system tested on done construction planned gatekeeper gLite-WMS wsgram unicore6 cream naregi ssh local InMemCred Globus G. Legacy G. RFC820 MyProxy VOMS JKS SSH Login / pwd X509 Supported technologies Grid techno. Others Execution Grid techno. Others Data Logical Physical rns lfn srb irods catalog rbyteio gsiftp srm srb irods http https sftp ftp file zip cache Grid techno. Others Security JSAGA

15. Selection of security context • Automatic selection is attractive to beginners, but it is… • inefficient • waste of time while trying with wrong contexts • jobs may fail at the end because they didn't run with expected VO/role • unreliable • may create files that will not be accessible to partners • may lock an account (e.g. by trying with 3 wrong passwords) • insecure • sending all our passwords to unexpected services is not safe • JSAGA's solution : configure selection of security context • bind job/data services to context types (+ URL patterns) • dynamic, per session, robust (all conflicts are detected at config-time) JSAGA

16. Pandora Gateway the grid platform of the company a leading Grid/Cloud Computing Business Services Provider used in several grid projects, including SimExplorer software for simulation experiment management includes a workflow engine that uses JSAGA to run simulations on grids used by experiences in biology computer science physics food industry Example applications JSAGA

17. jCAE(Java CAE) Computer Aided Engineering with support for distributed computing facilities Elis@ web portal for industrial and academic production grids Elis@ is used in several grid projects, including / Example applications Calcul Intensif pour les LOgiciels de CAO Electronique et les applications embarquées BioInformatique (protein sequence alignement...) Plasturgy (simulation of injection…) JSAGA

18. Java Job Submission tool to submit jobs on EGEE/EGI grid infrastructure efficient and scalable resource selection based on the QoS observed while submitting jobs http://cc.in2p3.fr/docenligne/269 uses JSAGA in order to hide middleware evolutions for security globus proxy  VOMS proxy data management gridFTP  SRM execution LCG-CE  CREAM-CE WMS used by D0 experiment to submit 15 000 jobs/day on EGEE/EGI (mainly Europe) Open Science Grid (USA) Other interoperability tools – JJS JSAGA

19. Java Universal eXplorer multi-protocols file explorer extensible for protocols just put the JSAGA plug-ins into the directory "lib/". for viewers http://cc.in2p3.fr/docenligne/821 Other interoperability tools – JUX JSAGA

20. C Python Jython a user application a user application SAGA-C++Py legacy python bind. Boost wrapper Other interoperability tools – JPySAGA Python Bindings for SAGA JPySAGA ? JySAGA C++ Bindings for SAGA Java Bindings for SAGA SAGA-C++ JSAGA Java SAGA Java GAT JSAGA

21. Develop new plug-ins NorduGrid middleware in progress… batch systems (DRMAA-based) Grid Engine batch systems (CLI-based/ssh) torque Grid Engine Develop API extensions Grid-RPC (from SAGA core) Service Discovery API (a SAGA extension) with plug-ins for technologies BDII with plug-ins for schemas GLUE CIM Upcoming… • Interoperability demo at next OGF (Brussels, 25-29 October) • objective is not to show interoperability between middlewares, but to show interoperability between SAGA implementations JSAGA

22. Job submission principle describe user's job submit it to the grid middleware select the execution host OGF standards BES, DRMAA, SAGA examples Remote Procedure Call (RPC) principle deploy user's service on grid call a procedure middleware select the service instance OGF standards Grid-RPC, SAGA examples Upcoming… job Grid-RPC JSAGA

23. Questions ? JSAGA