Project Status Report

1. Ian Bird WLCG Overview Board CERN, 29th October 2013 Project Status Report Ian.Bird@cern.ch

2. Outline Ian.Bird@cern.ch WLCG Collaboration & MoU update WLCG status report Summary from RRB

3. WLCG MoU Topics Ian.Bird@cern.ch • Additional signatures since last RRB • Greece, Kavala Inst. signed in June, • Tier 2 for ATLAS, CMS • Thailand: National e-Science Infrastructure Inst. • Tier 2 for ALICE & CMS • Latin America: Federation (CLAF), • Tier 2 for all 4 experiments – • initially CBPF (Brazil) for LHCb and CMS • New Tier 1s: update at this meeting

4. Short overview of activity Ian.Bird@cern.ch

5. Data 2008-2013 Tape Usage Breakdown CERN Tape Writes 27 PB 23 PB 15 PB CERN Tape Archive CERN Tape Verification Data Loss: ~65 GB over 69 tapes Duration: ~2.5 years Ian.Bird@cern.ch

6. Data transfers Global transfers CERN Tier 1s Ian.Bird@cern.ch CERN export rates driven (mostly) by LHC data export Global transfers at high (but somewhat reduced rate)

7. Resource occupancy Tier 0+1 Tier 2 Ian.Bird@cern.ch

8. Use of HLT farms during LS1 Ian.Bird@cern.ch • LHCb, ATLAS, CMS all commissioned HLT farms for use during LS1 & beyond • Simulation, re-processing, analysis • ATLAS and CMS use Openstack (open source cloud software) to manage their farms • Allows simplified configuration and allocation of resources • Identical to what is being done in the CERN CC to provide IaaS • LHCb use DIRAC to manage their HLT farm

9. HLT Farm use ATLAS: up to 15k jobs (≈a large Tier 2) CMS: use of 6k cores HLT upgrade will need ~100Gb/s link LHCb: HLT is largest simulation site Ian.Bird@cern.ch

10. ALICE Ian.Bird@cern.ch • Processed all of Run 1 data (~7.5 PB) • Moved a lot of analysis load into analysis trains • Now see periods where analysis >50% of jobs • Making bigger trains  improves efficiency • CPU efficiency problems resolved: ~80% even for analysis now • Introducing a data popularity service • Helps to optimize no. replicas and allows to reclaim disk space

11. ATLAS Ian.Bird@cern.ch • Good use of all resources made available (> pledges) • Effort on-going in software to optimize resource utilization for Run 2 • Reducing CPU use, event sizes, memory use etc. • More aggressive replication and data placement policies for Run 2  starting now • Reduce AOD replicas to 1 at T1 and T2 • Popularity-based dynamic data placement and automated clean up of secondary copies • Expect more use of tape retrieval as a consequence

12. CMS Ian.Bird@cern.ch • Some reduction in resource usage in recent months following completion of 2012 re-reconstruction and analysis • Starting 2011 reprocessing • Using HLT farm • Starting some preparations for 2015; eg. kinematic steps for simulations • Goal to commission new activities during LS1 • Disk/tape separation on-going • Data federation deployment in hand • All sites use fallback; testing remote access • Next production releases of sw will support multi-core • Also have dynamic data placement in development

13. LHCb Ian.Bird@cern.ch • Completed 1st incremental stripping of 2011+2012 • Starting 2nd campaign • Will keep full.DST on disk in 2014 to simplify this • 2012 simulation campaign ongoing • Factor 2 reduction in event size (compression + reduction of stored information) • Disk use increasing – but received more disk than pledges for 2013 • Starting to do analysis at Tier 2s with disk (T2-D) • Goal of 10 such sites; 4 commissioned in the summer • Work in hand on use of virtualisation techniques

14. Summary of RRB Ian.Bird@cern.ch

15. Assumption for resource needs LHC Live time Pile-up for ATLAS and CMS Resource usage efficiency Growth assumed achievable with flat budgets Ian.Bird@cern.ch

16. Higher trigger (data) rates driven by physics needs Based on understanding of likely LHC parameters; Foreseen technology evolution (CPU, disk, tape) Experiments work hard to fit within constant budget scenario Evolution of requirements Estimated evolution of requirements 2015-2017 (NB. Does not reflect outcome of current RSG scrutiny) 2008-2013: Actual deployed capacity Line: extrapolation of 2008-2012 actual resources Curves: expected potential growth of technology with a constant budget (see next) CPU: 20% yearly growth Disk: 15% yearly growth Ian.Bird@cern.ch

17. CRSG comments/ recommendations Ian.Bird@cern.ch Run 2 requests have become more definite since Spring – assumed flat budgets; ALICE and LHCb scrutinised requests have not always been met at T1. RRB requested to help find a way to resolve this; CRSG strongly supports on going efforts to improve software efficiency, notes that resulting gains are already assumed in the requests for Run 2; Effectiveness of disk use only partly reflected in occupancy. Welcome efforts (popularity, etc.) but would like a metric to take account of access frequency; Networks have been exploited to reduce disk use and move processing between tiers. Concern that poorly networked sites will be underused and cost implications of providing network capacity.

18. Updated scrutiny schedule Ian.Bird@cern.ch • Spring of year n • Final scrutiny of requests for year n+1 and look beyond • Review use of resources in previous calendar year, n-1 • Autumn of year n • Look forward to requests for year n+2 and beyond • If necessary, consider year n+1 requests • For individual experiments if they want significant changes • Or for all experiments if, say, LHC running parameters change significantly • CRSG asks experiments to submit documentation on 1 February and 1 August

19. Summary Ian.Bird@cern.ch • During LS1 activities have slowed somewhat after the conference season • Evolution of the computing models: • Document close to final – see later talk • Implementations under way in preparation for Run 2 • October RRB generally welcomed the strategy to work hard to fit within flat budgets, although clearly some concerns about this