1 / 23

CERN Computing Fabric Status LHCC Review , 19 th November 2007

CERN Computing Fabric Status LHCC Review , 19 th November 2007. Ingredients. Coarse grain functional differences in the CERN computing fabric T0  Central Data Recording, first pass processing, tape migration, data export to the Tier 1 sites CAF

palmer
Télécharger la présentation

CERN Computing Fabric Status LHCC Review , 19 th November 2007

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. CERN Computing Fabric Status LHCC Review , 19th November 2007 Bernd Panzer-Steindel, CERN/IT

  2. Ingredients • Coarse grain functional differences in the CERN computing fabric • T0 •  Central Data Recording, first pass processing, tape migration, • data export to the Tier 1 sites • CAF •  selected data copies from the T0 (near real-time) , calibration and • alignment, analysis, T1/T2/T3 functions • means something different for each experiment • Both contain the same hardware, distinction is done via logical configurations • in the Batch system and the Storage system • CPU nodes for processing ( ~65% of the total capacity for the T0) • Disk server for storage ( ~40% of the total capacity for the T0) • Tape libraries, tape drives and tape server • Service nodes Bernd Panzer-Steindel, CERN/IT

  3. Growth rates based on the latest experiment requirements at CERN ~linear growth rates ! underestimates ? experience/past shows exponential growth rates Bernd Panzer-Steindel, CERN/IT

  4. Preparations for 2008 I • Tendering preparations for the 2008 purchases started in May 2007 • Deliveries of equipment have started, first ~100 nodes have arrived and are being installed • More deliveries spread over the next 4 month • heavy logistic operations ongoing : • preparation for the installation of ~2300 new nodes • racks, rack preparations, power+console+network cabling, • shipment and unpacking, installation (physical and logical), • quality control and burn-in tests • preparations to ‘retire’ ~1000 old nodes during the next few month Bernd Panzer-Steindel, CERN/IT

  5. Preparations for 2008 II • Resource increase in 2008 • more than doubling the amount of CPU resources •  ~ 1200 CPU nodes (~4 MCHF) • 2. increasing the disk space by a factor 4 •  ~ 700 Disk servers ( ~6 MCHF) • The experiment requirements for disk space this year were underestimated. • We had to increase disk space by up to 50% • during the various data challenges and productions. • 3. increase and consolidation of redundant and stable services •  ~ 350 service nodes (~3 MCHF) • Grid services (CE, RB, UI, etc.) , Castor, Castor data bases, condition • data bases, VO-boxes, experiment specific services (bookkeeping, production • steering, monitoring, etc.), build server don’t underestimate the service investments ! Bernd Panzer-Steindel, CERN/IT

  6. Power and Cooling • Current computer center has a capacity of 2.5 MW for powering the nodes • and 2.5 MW cooling capacity. A battery based UPS system allows for 10 min • autonomy for the 2.5 MW. • Power for critical nodes is limited to about 340 KW (capacity backed-up by • the CERN diesel generators). no free capacity left (DB systems, network, AFS, • Web, Mail, etc.) • We will reach ~2 MW already in March 2008 and will not be able to host • the full required capacity in 2010 • Started activities already more than a year ago, slow progress • now active discussion between IT PH and TS • Identify building in Prevessin (Meyrin does not have enough power available) • and start preparations for infrastructure upgrade • Budget is already foreseen Bernd Panzer-Steindel, CERN/IT

  7. Material Budget • based on the latest round of requirement gathering from the experiments during the late summer period. • includes provisioning money for a new computer center • presented to the CCRB on the 23rd of October small deficit but within the error of the cost predictions CPU server, Disk storage, Tape storage and infrastructure, service nodes, Ortacle Data Base infrastructure, LAN and WAN network, testbeds, new CC costs spread over 10 years Bernd Panzer-Steindel, CERN/IT

  8. Processors I cost of a full server node cost of a separate single processor Bernd Panzer-Steindel, CERN/IT

  9. Processors II • less than 50% of a node costs are the processors plus memory • 2007 was special year , heavy price war between INTEL and AMD, INTEL pushing their quad-cores (even competing with their own products) • new trend  dual motherboard per 1u unit, very good power supply power efficiencies, as good as for blades • our purchases will consist out of these nodes with a possibility of getting also blades Bernd Panzer-Steindel, CERN/IT

  10. Processors III • Technology trends : • aim to have a two year cycle now • architecture improvements and structure reduction • (45nm products already announced by INTEL) • multi-core 2 3 4 6 8………… • BUT • what to do with the expected billion transistors and multi-cores ? • the market is not clear  wide spread activities of INTEL e.g.\ • -- initiatives to get multithreading into the software, quite some time away, complicated • especially debugging (we have a hard time to get our ‘simple’ programs to work) • -- co-processors (audio, video, etc.) • -- merge of CPU and GPU (graphics) , AMD + ATI combined processors, • NVIDIA  use GPU as processor, INTEL  move graphics to the cores • -- on the fly re-programmable cores (FPGA like) • not clear where we are going • specialized hardware in the consumer area  change of price structure for us Bernd Panzer-Steindel, CERN/IT

  11. Memory I Bernd Panzer-Steindel, CERN/IT

  12. Memory II • still monthly fluctuations in costs, up and down • large variety of memory modules frequency and latency 533 and 667 MHz about 10% cost difference, factor 2 for 1 Ghz higher frequency goes along with higher latency CAS • how does HEP code depend on memory speed ? • DDR3 upcoming , more expensive in the beginning • is 2 GB per core really enough ? Bernd Panzer-Steindel, CERN/IT

  13. Disk storage I cost of a full disk server node cost of a separate single disk Bernd Panzer-Steindel, CERN/IT

  14. Disk storage II • Trends • cost evolution of single disks is still good ( ~ factor 2 per year, model dependent) • lots of infrastructure needed • upgrade of CPU and memory  footprint of applications : RFIO, Gridftp, buffers, new functions, checksums, RAID5 consistency checks, data integrity probes • need disk space AND spindles  use smaller disks or buy more  increase overall costs • solid-state-disks, much more expensive (factor ~50)  data base area • hybrid disks • good for VISTA (at least in the future, does not work yet…) • but higher price e.g. new Seagate disks + 256 MB flash == + 25% costs • general trend for notebooks • can’t profit in our environment  seldom cache reuse Bernd Panzer-Steindel, CERN/IT

  15. Internal Network I The physical network topology (connections of nodes to switches and routers) is defined by space, electricity, cooling and cabling constraints Network router Service Nodes Disk Server CPU Server Bernd Panzer-Steindel, CERN/IT

  16. CPU server Internal Network II Logical network topology Changing access patterns, high aggregate IO on the disk servers 3000 nodes running 16000 concurrent physics applications are trying to access 1000 disk servers with 22000 disks Disk server Bernd Panzer-Steindel, CERN/IT

  17. Internal Network III • Need to upgrade the internal network infrastructure : • decrease the blocking factor on the switches = spread the existing servers • over more switches • Changes since the 2005 LCG computing TDR : • disk space increased by 30 % • concurrent running applications increased by a factor 4 (multi-core technology evolution) • computing model evolution, more high IO applications • (calibration and alignment, analysis) • doubling the number of connections (switches) to the network Core routers which as a consequences requires also to double the number of routers  additional investment of 3 MCHF in 2008 (already approved by finance committee) Bernd Panzer-Steindel, CERN/IT

  18. Batch System some scalability and stability problems in spring and early summer solved with the upgrade to LSF 7 and hardware upgrade of LSF control nodes Much improved response time, removed throttling bottlenecks average about 75000 jobs/day peak value 115000 jobs/day up to 50000 in the queue at any time tested with 500000 jobs Bernd Panzer-Steindel, CERN/IT

  19. Tape Storage • Today we have : • 10 PB of data on tape, 75 million files • 5 Silos with ~30000 tapes, ~5 PB free space • 120 tape drives (STK and IBM) • during the last month we have 3PB written to tape and 2.4 PB read from tape small files and spread of data sets over too many tapes caused very high mount load in the silos • space increase to 8 free PB in the next 3-4 month • more drives to cope with high recall rates and small files •  need Castor improvements Bernd Panzer-Steindel, CERN/IT

  20. CASTOR much improved stability and performance during the summer period (Castor Task Force) regular running at ‘nominal’ speed (with 100% beam efficiency assumed) very high load on the disk servers small scale problems observed  identified and fixed (Castor+Experiment) CMS CSA07 complex patterns and large number of IO streams require more disk space for the T0 (probably factor 2) successful coupling of DAQ and T0 for LHCb, ALICE and ATLAS (not yet 100% nominal speed) CMS is planned for the beginning of next year ATLAS export tests and M5 run Bernd Panzer-Steindel, CERN/IT

  21. Data Export ATLAS successfully demonstrated for several days their nominal data export speed (~1030 MB/s) all in parallel to the CMS CSA07 exercise  no Castor issues, no internal network issues Bernd Panzer-Steindel, CERN/IT

  22. Experiment DM system Tape storage Batch system Disk storage Data Management • Enhance Castor disk pool definitions • activity in close collaboration with the experiments, new Castor functionalities • are now available (access control)  avoid disk server overload, better tape recall efficiencies • Small files creating problems for the experiment bookkeeping systems and the HSM tape system  need Castor improvements in the tape area (some amount of small files will be unavoidable) Experiments are investing into file merging procedures creates more IO streams and activity, needs more disk space • Data integrity the deployment and organization of data checksums needs more work will create more IO and bookkeeping • CPU and data flow efficiency To increase the efficiencies one has to integrate the 4 large functional units much closer (information exchange). Bernd Panzer-Steindel, CERN/IT

  23. Summary • Large scale logistic operation ongoing for the 2008 resource upgrades • Very good Castor performance and stability improvements • Large scale network (LAN) upgrade has started • Successful stress tests and productions from the experiments (T0 and partly CAF) • Power and cooling growth rate requires a new computer center, planning started Bernd Panzer-Steindel, CERN/IT

More Related