WLCG Service Report

1. WLCG Service Report Maria.Girone@cern.ch,Harry.Renshall@cern.ch ~~~ WLCG Management Board, 7th December 2010 Covers 2 week period from 22 November

2. Introduction • Generally smooth operation on experiment and service side • 4 GGUS alarms • Service Incident Reports received: • 16/11 and 26/11: GGUS Database down • 29/11: CERN CASTOR name server • Service Incident Reports to be produced / updated: • HI run: IN2P3 ATLAS slow transfers • 27/11, 30/11, 02/12, 04/12: CERN ATLAS Offline DB node reboots • Coincidence with log rotate – now turned off – still to be understood • RAL disk server problem – still ongoing – but interim SIR • Some site availability issues – commented directly in Availability Plots – in particular several sites suffered power failures • FNAL 22/11, NDGF 24/11, RAL 01/12 2

3. GGUS summary (2 weeks) 3

4. https://gus.fzk.de/ws/ticket_info.php?ticket=64577 ATLAS ALARM->CERN-CNAF TRANSFER FAILURES WLCG MB Report WLCG Service Report

5. https://gus.fzk.de/ws/ticket_info.php?ticket=65015 ALICE ALARM-> FZK CREAM-CE WLCG MB Report WLCG Service Report

6. https://gus.fzk.de/ws/ticket_info.php?ticket=65019 ATLAS ALARM-> SARA MATRIX LFC REG.FAILURES WLCG MB Report WLCG Service Report

7. https://gus.fzk.de/ws/ticket_info.php?ticket=64726 SIR: CERN CASTOR name server CMS ALARM->CERN CASTOR WLCG MB Report WLCG Service Report

8. GGUS SIRs • Down 3.5h on 16/11 and 1.5h on 26/11 • No web access / no ticket update in both cases • 16/11: outage isn't understood up to now – more details in SIR – maybe load related • 26/11: the reason for the outage was an unscheduled reboot of the Oracle DB cluster failing for all nodes. The DB reboot was caused by a combination of hardware failure and miss-configuration. The hardware failure made the DB cluster lose some essential control files. The miss-configuration prevented the cluster reboot.

9. CASTOR NAMESERVER • Problem with the Name Server database. The root cause seems to be related to a problem accessing one of the db datafiles. The database set the problematic datafile offline. • All CASTOR instances were affected, the service was unavailable between 15h and 16h Geneva time (UTC + 1)

10. 2.1 2.2 2.2 2.2 2.2 2.2 2.2 1.1 1.1 4.1 4.1 4.1 4.1 3.1 4.2 4.2 4.3 4.3

11. Analysis of the availability plots ATLAS 1.1 INFN: SRM tests were failing: lcg-cp (no files have been lcg-cp on this endpoint), lcg-cr (error reading token data header: connection closed), lcg-del (unary operator expected) possibly connected to INFN transfer and storage problems (original problem reported on Friday the 19th – Network Problems GGUS: 64459). Site reported that it was a temporary storage outage. ALICE 2.1 FZK: VOBox User Proxy Registration test was failing (the user is not allowed to register his proxy within the VOBOX ). 2.2 Green box for IN2P3 & FZK: CE tests were failing. SAM CE tests should be ignored as ALICE is using CREAM CE. There are no CREAM CE direct submission tests for ALICE yet (work in progress). CMS 3.1 ASGC: The CMS software installation SAM test was failing in 5 CEs of ASGC. CEs started publishing CMS SW releases that were not yet installed - Savannah: 117339. LHCb 4.1 IN2P3: Issues with IN2P3 shared area – GGUS:59880 (performance issues) and GGUS:62800 (SW installation). CE-tests were failing: install, job-Boole, job-DaVinci, job-Brunel,job-Gauss, sft-vo-swdir (‘took more than 150 seconds while 60 seconds are expected’ OR ‘software check failed’). SRM DiracUnitTestUser test failing occasionally (failed to put file to storage.). 4.2 PIC: CE tests were failing: sft-vo-swdir (took more than 150 seconds while 60 seconds are expected) and lhcb-job-DaVinci (software not Installed). 4.3 SARA: CE tests were failing: sft-vo-swdir (took more than 150 seconds while 60 seconds are expected). SRM lcg-cr test failed once on the 28th: CGSI-gSOAP running on volhcb29.cern.ch reports Could not load client credentials.

12. 2.1 1.1 2.1 2.1 4.1 3.1 3.1

13. Analysis of the availability plots ATLAS 1.1 NDGF: SRM tests were timing out on the 30th (a day before the planned downtime of the SRM) while trying to write a file into the TAPE buffer disk. GGUS:64843. ALICE 2.1 Green box for IN2P3, NIKHEF and RAL: CE tests were failing. SAM CE tests should be ignored as ALICE is using CREAM CE. There are no CREAM CE direct submission tests for ALICE yet (work in progress). CMS 3.1 RAL: The CE sft-job test was failing with the following proxy-related errors: ‘Got a job held event, reason: Globus error 131: the user proxy expired (job is still running)’ & ‘Failed to get expiration time of proxy’. The failing test was running as user ‘cmssgm’, which is only allowed to have one job at a time. One job had been stuck for a few days, preventing other cmssgm jobs from running. Savannah:118201. LHCb 4.1 IN2P3: Minor issues with IN2P3 shared area – GGUS:59880 (performance issues): The sft-vo-swdir CE test was timing out (took more than 150 seconds while 60 seconds are expected).

14. Other Issues • Shared s/w area: very common – and repetitive – cause of problems to experiments • Some instabilities seen by ATLAS at IN2P3 during HI run – no longer seen • Good communication established between site and experiment (daily reports) which will be useful also at daily operations call • SIR in preparation

15. Summary • HI data processing and proton data reprocessing proceeding smoothly • Data taking for 2010 now over 15