ILDG Middleware Status

1. Presented By: Bálint Joó, Jlab, USA Working Group Members: G. Beckett (EPCC, UK) T. Boku (CCS Tsukuba, Japan) D. Byrne (EPCC, UK) P. Coddington (CSSM, Australia) M. Ernst (DESY, Germany) M.Sato (CCS Tsukuba, Japan, Convener) J. Simone (Fermilab, USA), W. Watson (Jlab, USA, Convener) ILDG Middleware Status

2. Contents • Reminder of Our Task • Reminder of Our Architecture • Summary of Current Discussions • Future plans

3. Our Mission • We develop standard interfaces that facilitate the interoperation of a grid of grids • Local grids • Control Local Policies (Eg Security) • Control Local Grid Implementations • Choice of software components

4. Our Architecture • Simple veneer over local implementations • Minimally intrusive (wrapper?) • Not too burdensome to implement • Language and platform neutral if possible • Our choice: SOAP Web Services

5. Our Architecture: Components XML ILDG service description file Security Infrastructure (Identity management access control encryption) Metadata Catalogue Web Service (MDC) Logical File Name (LFN, formerly GFN) Replica Catalogue Web Service (RC) SRM service (data management)

6. ILDG Service File • Publishes Service Endpoints • Each collaboration currently has entries for MDC and RC • Started filling this out at MW Meeting in Tsukuba • http://www.lqcd.org/ildg/Services.xml • http://www.lqcd.org/ildg/Services.xsd • Entries for LatFor & CSSM

7. Logical File Names (LFN) • Changed to LFN from GFN to fit better with MDWG usage • Unique and Persistent name for files • MWWG mandated URI format lfn://<ILDG Collaboration>/<Collaboration Specific Path> • Collaboration indexes service file • Collaborations control their own path

8. MDC Web Service • Maps metadata to LFNs • Provides “Virtual XML Database” • Prototype WSDL definition by M. Sato • Production MDC Web Services deployed by LatFor & CSSM • Interface definition in flux • Raw queries – variable return types • Targeted Queries – return LFNs

9. Replica Catalog Service • Maps (data) LFNs to URL • Intention was to map to SRM URL • SRM component deals with transfer • SRM requirement temporarily relaxed – RC can return any URL • No standard download protocol • Client must have appropriate tool • Prototype implementation at Jlab

10. SRM Service • SRM definition v3 almost ready • Fewer backward incompatible changes than v1->v2 transition. • Migration should be simple(r) • Popular dCache storage management software still uses v1.x (Fermilab, DESY) but Jlab uses v2. v1 and v2 incompatible • ILDG uses v2 SRM, but SRM use is not mandated currently

11. Security Infrastructure • Currently quite nebulous • Previously defined 'ILDG Group Files' • XML Documents containing user certificates • Not very grid-like, not best match with LatFor preference/implementation • Currently considering VOMS service • Security not explicit in current interfaces

12. WG Organization • Had very nice and productive meeting in Tsukuba in Oct 2005. Minutes on ILDG web site. • Instituted more frequent teleconference calls – first of these was on Dec 5. Minutes on ILDG web site. Next call Jan 23, 2006. • Middleware BLOG set up on ILDG web site to aid reporting progress.

13. Plans • Finish up discussions and make a decision on using VOMS by end of February • US basic service up with some data by end of January. • EPCC to convert MDC GUI browser to work with Web Services • Interoperability testing Feb-May 2006. • Interoperable system by June 2006.

14. Current Discussions • MDC • Need to fix interface • Small inconsistencies between WSDL from LatFor and Japan – to be fixed • Clarify semantic meaning of query functions • Raw queries – return results appropriate to query • LFN queries – return markovChainUR or dataLFN • Need document retreival queries ?

15. <?xml version=”1.0” ?> <gaugeConfiguration ...> <management/> <implementation/> <algorithm/> <precision> <markovStep> <markovChainURI>lfn://myCollab/myEnsemble</markovChainURI> <series>1</series> <update>010170</update> <avePlaquette>0.53380336E+00</avePlaquette> <dataLFN>lfn://myCollab/myFilename</dataLFN> </markovStep> </gaugeConfiguration> XPath funcion XPath Number Type XML Tags Raw Queries: Raw Query: number(/gaugeConfiguration/markovStep [dataLFN="lfn://myCollab/myFilename"]/avePlaquette) Returns: 0.53380336 Raw Query: /gaugeConfiguration/markovStep [markovChainURI="lfn://myCollab/myEnsemble"]/dataLFN Returns: <dataLFN > lfn://myCollab/myFilename </ dataLFN >

16. <?xml version=”1.0” ?> <gaugeConfiguration ...> <management/> <implementation/> <algorithm/> <precision> <markovStep> <markovChainURI>lfn://myCollab/myEnsemble</markovChainURI> <series>1</series> <update>010170</update> <avePlaquette>0.53380336E+00</avePlaquette> <dataLFN>lfn://myCollab/myFilename</dataLFN> </markovStep> </gaugeConfiguration> Single Return Type: Array of LFNs for all queries LFN Queries: LFN Query: /gaugeConfiguration/markovStep[update="010170"] Returns: { lfn://myCollab/myFilename } LFN Query: /gaugeConfiguration/markovStep [markovChainURI=”lfn://myCollab/myEnsemble”] Returns: { lfn://myCollab/myFilename, lfn://myCollab/myFilename2,... }

17. Comparison • Raw Queries • Powerful: Need Only 1 Query function • Can use XPath functions eg: to count matches • Query returns different types • Eg: numbers, strings, XML Snippets • Can use to implement LFN Queries • Also good for Web interfaces, testing

18. Comparison • Domain Specific (eg LFN) Queries • Less Flexible • Can't execute arbitrary XPath functions • More logic in clients • More complex • Need more functions: • Query/Get Config XML , Query/Get Ensemble XML • Tailored to Particular Use Cases • Eg: Command Line Tools (find & download) • Don't need XML database necessarily

19. Security Infrastructure • Can use SSL to encrypt communications with services • Can have certificate based authorization in Web servers, Application servers, and even some databases • Use firewalls to secure back ends • Credential forwarding for service aggregator portals may need thought

20. Certificate Exchange Service (MDC/RC) Clear Encrypted Connection Client Certificate Exchange Thought Needed Aggregate Service (MDC/RC) Encrypted Connection Client Certificate forwarding? Proxying? How to maintain secure connection?? Australian Service (MDC/RC) Japanese Service (MDC/RC) LatFor Service (MDC/RC) US Service (MDC/RC) UK Service (MDC/RC) Firewall Actual Database Actual Database Actual Database Actual Database Actual Database

21. VOMS: Virtual Organisation Management Service • Technology for identity and role management • Distributed and decentralized • Part of LCG, OSG and gLite • LatFor moving to production use • To be trialled in UK and Japan • In US, Fermilab will use infrastructure on collaborative project with DESY

22. VOMS • Interest in VOMS expressed by Australia • WG to come to a decision on standardising use of VOMS by end of February 2006 • Personal worry: This is choosing software rather than defining an interface. • Pragmatism: LatFor requires users' certificate in their VOMS server. Will eventually hold everyone's certificate

23. LCG Software Components • Michael recommended use of LCG software: • Rich in features, presented as RPMs through software repositories, with installation tools (apt-get, yaim) • Can install just components we need • Can use any Red Hat Enterpise Linux like system (Scientific Linux (SL), RHEL, Fedora Core) CERN supports SL.

24. LCG Components • Large User Community (LHC, OSG) • Used by LatFor (File Catalog – LFC, and VOMS) • In US Fermilab will have infrastructure • BUT • Don't want to force particular technology/software. That's not our mandate !

25. Conclusions • Components coming along – individual collaborations will present own status reports • MDC interface solidifying currently • Most groups have received funding/effort • Most groups have local infrastructures and are now working on ILDG issues • I have a LOT of work ahead of me.

26. US Middleware status • Effort recently revitalised with arrival of Bálint at the Jlab. • General Web Service Infrastructure: • Apache Web Server • Tomcat Application Server • Axis Web Services Framework • X509 Certificates, SSL encryption • Prototype Data: MILC Datasets from NERSC Gauge Connection

27. US Middleware Status • RC Prototype ready for some time • Need secure connection and certificate based authentication for production use. • RC needs data in it to be useful. • In discussion with MILC to choose suitable LFNs for data in NERSC archive. Recent helpful changes in metadata schema facilitate things.

28. US Middleware Status • In analysis stage for MDC • Plan: use eXist native XML database • Pure Java • Integrates well with Tomcat and Axis • Comes with Web Service API • Sophisticated XML capabilities: • Validation against Schemata, XQuery, XUpdate • Expect very thin ILDG layer on top

29. US Middleware Status • Progress with ILDG File Format • Conversion routine from NERSC format to ILDG format exists within MILC code and QDP++ using QIO library. • MILC Code can read QDP++ files and those provided by Dirk Pleiter (LatFor) • Dirk could read files produced by QDP++ and MILC

30. US Middleware Status • File format (cont'd) • Only minor discrepancy • QIO uses LIME type: ildg-data-lfn • Dirk's code & spec give: ildg-data-LFN • Dirk suggests case insensitivity in codes • Backward compatible solution for all sides. • Carleton willing to change QIO to use case insensitive checking of LIME types.