Global Accounting in the Grid and Cloud

1. Global Accounting in the Grid and Cloud John Gordon, STFC HEPiX, Beijing 18th October 2012

2. Outline • History • Yesterday • Today • Tomorrow • New types of Accounting Record • Future

3. Overview • The APEL accounting system has been gathering cpu accounting records for the LHC experiments from around the world since 2004 • It now contains data from 2x10**9 jobs from 350 sites in 50 countries. • Work is under way to extend accounting to storage and cloud

4. History

5. History • LCG->EGEE->WLCG->EGI+EMI • Including along the way: • Gratia (OpenScience Grid) • DGAS (Italy) • SGAS (NorduGrid)

6. LCG • In the beginning, there was Les Robertson. • Trying to form a Grid for LHC Computing from the plethora of national and international grid projects which had sprung up around the world • The LHC Tier model* predated Grids but Grids offered technology on which to build the LHC distributed computing model(s). • Getting working middleware was a problem but not the only one. There was also a need for a number of other operational services required for a working Grid. • Les persuaded Tier1s to take responsibility for defining and developing various missing operational components • Karlsruhe – helpdesk • Lyon – operations portal • CERN – monitoring • RAL – accounting * MONARCH Report

7. The Start of APEL • Dave Kant designed and wrote parsers for a few batch systems to gather usage data from the batch logs and the user grid identity information from Globus GRAM and then the LCG-CE. • The APEL publishers combined this into a usage record for each job and sent it via R-GMA to RAL where they were processed, summarised, and visualised • So the first sites started to publish

8. By December 2004, 15 CEs at 13 sites Dave Kant -

9. EGEE • APEL accounting became a core service in EGEE (2004-2010) and the client was rolled out to more sites • CESGA took over running and developing the portal

10. Others • Driven by WLCG, data from other accounting systems were incorporated, • DGAS – Italy • Gratia – Open Science Grid • SGAS – Nordugrid • And EGEE extended beyond Europe to Asia Pacific, India,

11. WLCG • EGEE pushed all sites to publish but functionality was driven by WLCG. • The central APEL repository stores data by site • The Portal pulls in topology to drive reports • Tier1 Reports • Tier2 federations formally defined • Comparison with Pledges. (Tier2 Report)

12. APEL 2011-12 • APEL collects cpu usage data from 274 sites by the APEL client software and from a further 90 where data is collected by other software and published to the central APEL repository. • It is thus a single worldwide point of reference for all accounting data for a range of VOs(LHC and other international VOs as well as regional and national VOs). • These non-APEL sites include alternative middleware stacks(ARC) within EGI and e-infrastructures within EGI (Italy, NorduGrid) and outside (Open Science Grid in US). • The data from systems/services not using the APEL Client were doing direct database inserts. • >2x10**9 jobs. 61% ATLAS • Reached 73Mjobs/month in 2012 • 4M HS06Years almost 50% in the last 12 months.

16. Visualisation • The Accounting Portal at CESGA • Pulls data from APEL, applies various topologies (T1, T2, countries, NGIs) • Allows dynamic queries at any point in a tree showing one variable as a function of two dimensions • Variables: njobs, cpu, wallclock, normcpu, normwall, cpueff • Dimensions: month, site/region, VO,

19. 5

22. Internals

23. APEL – 2011 EGI Accounting Portal APEL client CONSUMER BROKER APEL client ActiveMQ APEL client MySQL http://goc-accounting.grid-support.ac.uk MySQL MySQL Summaries created External clients

24. APEL 2012- Deployment Stage 1 MySQL http://goc-accounting.grid-support.ac.uk APEL client External clients MySQL MySQL CONSUMER BROKER Summaries created ActiveMQ MySQL • CPU JobRecords • CPU Summaries External clients EGI Message Brokers Record loader Receiving SSM EGI Accounting Portal Sending SSM

25. APEL 2012- Deployment Stage 2 APEL client http://goc-accounting.grid-support.ac.uk BROKER CONSUMER ActiveMQ DBunloader MySQL EGI Message Brokers MySQL • CPU JobRecords • CPU Summaries Converted JobRecords New APEL client External clients EGI Message Brokers Record loader Receiving SSM EGI Accounting Portal Sending SSM

26. APEL – 2013 nagios pub/sync http://goc-accounting.grid-support.ac.uk DBunloader New APEL client MySQL • CPU JobRecords • CPU Summaries New APEL client EGI Message Brokers EGI Message Brokers External clients Record loader Receiving SSM EGI Accounting Portal Sending SSM

27. What is SSM? SSM Status Deployed in production for ~ 10 months No operational difficulties (except for an LDAP query bug) Robust, fast Handles load easily Allows separation of messaging and server: filesystem buffer but no message checking • Secure Stomp Messenger • Simple program to send and receive messages • Independent of message content • Uses STOMP (and EGI brokers) • ~1k lines of python code

28. Accounting and SSM • SSM can transport any data • We use it for accounting • Production: • APEL message format • Testing: • EMI Compute Accounting Record (CAR) • EMI Storage Accounting Record (StAR)

29. SSM 2.0 • SSM 1 works well • But we are changing it • Why? • It’s a bit over-complicated • Difficult to develop against • Could be simpler • Interoperability issues: • Crypto not well defined • Synchronous messaging • Unnecessary (?) message sequence • EMI has different accounting systems which need to interoperate • One python program is not good enough.

30. EMI Messaging Protocol for Accounting (EMPA) • More logical, simpler • Use persistent queues • Not replies • Use SSL • Instead of encryption • (encryption still an option) • Use the infrastructure

31. New Publishers • A number of other accounting clients and systems are in the process of publishing to APEL • All they need to do is extract the relevant jobs from their database, (combine into a summary), transform to the UR format and write to a local directory. • SSM does the rest, shipping the files to APEL.

32. Publishers • Emi-apel (bulk of EGI sites) • DGAS (Italy) • Open Science Grid (USA, others) • SGAS (NorduGrid, Switzerland, Finland) • Individual Sites (CERN, NIKHEF, CC-IN2P3) • Unicore (Poland, Germany) • Globus/IGE (Germany, ??) • ARC/JURA (Hungary and other ARC) • PRACE (many but only selected VOs) • EDGI – Desktop Grids • MAPPER - Multiscale Simulation (PRACE+EGI)

33. Extend EGI Infrastructure APEL ARC/SGAS Sites DGAS OSG/Gratia SSM/ActiveMQ Messaging MAPPER, EDGI

34. New Regionalised Structure NGI APEL Emi-apel ARCCE Emi-apel Emi-apel NGI SGAS Central APEL Emi-apel OtherRepo Portal • Regionalised APEL Server, collects job records from NGI sites • Send summaries on to central APEL • Visualisation Portal

35. New Accounting Records

36. Readiness to Receive New Records • The APEL repository is in the middle of a migration but the capability to receive usage records over SSM is already in production for both Job Records and Summary Job Records. It is being used by Open Science Grid and CERN, with a number of other publishers in migration. • Consumers are in place on the test service for CAR and StAR. • Adding others is a lightweight process. Much simpler than defining the record schema.

37. APEL – 2013 http://goc-accounting.grid-support.ac.uk DBunloader New APEL client MySQL • CPU JobRecords • CPU Summaries • StorageRecords • StorageSummaries New APEL client EGI Message Brokers EGI Message Brokers External clients Record loader Storage clients Receiving SSM Storage loader EGI Accounting Portal Sending SSM

38. CAR • CAR is an EMI revision of the OGF UR v1.0 rationalising some issues and including some common extensions deployed in a number of implementations. • Not UR 2.0 which is a bigger revision started by OGF UR-WG • XML document • CAR v1.0 was agreed but now some issues have arisen and CAR v1.0.1 is being finalised • APEL new schema incorporates CAR v1.0

39. Storage • StAR (Storage Accounting Record) is a version of the OGF UR 1.0 adapted for accounting of storage utilisation • Developed by EMI, submitted as a public document to OGF, revised in the light of public comments. • Added Sitename, allocated space, and revised time definitions. • Status – being implemented by EMI storage providers (dCache, StoRM, DPM) in EMI-3

40. Storage Solutions • dCache (EMI) • StoRM (EMI) • DPM (EMI) • Castor • EOS • BestMan • Gratia (already collecting) • xrootd • hadoop • Gstat • Cloud Storage • (Unicore) • (ARC) • iRods • ????? The middleware in scope for EGI and WLCG Anything that collects storage information could also publish usage records eg Julia Andreeva talk at CHEP

41. Cloud • The EGI Federated Cloud Task Force has adopted a VM UR based on OGF UR • Many values like memory and network were defined in the UR but seldom if ever implemented. • Sensors to collect the information and construct usage records were written. • OpenNebula and OpenStack so far. • SSM is used to send to RAL.

42. Future

43. APEL continues to collect more cpu accounting data for a widening ranges of infrastructures, projects, and VOs • A more distributed architecture should reduce bottlenecks and give more control to the NGIs • New types of accounting record will be added (Storage, Cloud, ...)