1 / 21

Towards ATLAS Data Challenges 2

Towards ATLAS Data Challenges 2. LCG-GDB 10 th June 2003 Gilbert Poulard ATLAS Data Challenges Co-ordinator CERN EP-ATC. Outline. DC1: a starting point for DC2 What has been achieved DC2 Main goals Planning Resources. ATLAS DC1 (July 2002-April 2003).

zody
Télécharger la présentation

Towards ATLAS Data Challenges 2

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Towards ATLAS Data Challenges 2 LCG-GDB 10th June 2003 Gilbert Poulard ATLAS Data Challenges Co-ordinator CERN EP-ATC

  2. Outline • DC1: a starting point for DC2 • What has been achieved • DC2 • Main goals • Planning • Resources G. Poulard - GDB

  3. ATLAS DC1 (July 2002-April 2003) • Primary concern was delivery of events to High Level Trigger (HLT) and to Physics communities • HLT-TDR due by June 2003 • Athens Physics workshop in May 2003 • Put in place the full software chain from event generation to reconstruction • Switch to AthenaRoot I/O (for Event generation) • Updated geometry • New Event Data Model and Detector Description • Reconstruction (mostly OO) moved to Athena • Put in place the distributed production • “ATLAS kit” (rpm) for software distribution • Scripts and tools (monitoring, bookkeeping) • AMI database; Magda replica catalogue; VDC • Job production (AtCom) • Quality Control and Validation of the full chain • Use as much as possible Grid tools G. Poulard - GDB

  4. ATLAS DC1 (July 2002-April 2003) • DC1 was divided in 3 phases • Phase 1 (July-August 2002) • Event generation and detector simulation • Phase 2 (December 2002 – April 2003) • Pile-up production • Classical batch production • With Grid tools on NorduGrid and US-ATLAS-Grid • Reconstruction (April-May 2003) • Offline code only • Worldwide exercise with many participating institutes G. Poulard - GDB

  5. DC1 Task Flow • As an example, for 1 sample of di-jet events: • Event generation: 1.5 x 107 events in 150 partitions • Detector simulation: 3000 jobs Zebra Zebra Athena-Root I/O Zebra Di-jet Hits/ Digits MCTruth Digits MCTruth Atlsim/G3 + Filter Ntuple Atlsim Pile-up Athena Reco. HepMC (~450 evts) (5000 evts) Pythia 6 Hits/ Digits MCTruth Digits MCTruth Ntuple Atlsim/G3 + Filter Athena Reco. Atlsim Pile-up HepMC 105 events Digits MCTruth Ntuple Athena Reco. Hits/ Digits MCtruth Atlsim Pile-up Atlsim/G3 + Filter HepMC Event generation Pile-up Detector Simulation Reconstruction G. Poulard - GDB

  6. Tools in DC1 physics metadata perm production log trans production log replica catalog physics metadata recipe catalog Perm. production log Trans. production log physics metadata replica catalog recipe catalog AMI Magda AMI VDC Magda AtCom GRAT automatic production framework interactive production framework G. Poulard - GDB

  7. DC1 in numbers G. Poulard - GDB

  8. ATLAS DC1 Phase 1 : July-August 2002 3200 CPU‘s 110 kSI95 71000 CPU days 39 Institutes in 18 Countries • Australia • Austria • Canada • CERN • Czech Republic • France • Germany • Israel • Italy • Japan • Nordic • Russia • Spain • Taiwan • UK • USA grid tools used at 11 sites 5*10*7 events generated 1*10*7 events simulated 3*10*7 single particles 30 Tbytes 35 000 files G. Poulard - GDB

  9. Primary data (in 8 sites) Pile-up: Low luminosity ~ 4 x 106 events (~ 4 x 103 NCU days) High luminosity ~ 3 x 106 events ( ~ 12 x 103 NCU days) Data (TB) Simulation: 23.7 (40%) Pile-up: 35.4 (60%) Lumi02: (14.5) Lumi10: (20.9) Data replication using Grid tools (Magda) G. Poulard - GDB

  10. ATLAS DC1: July 2002-April 2003Goals : Produce the data needed for the HLT TDR Get as many ATLAS institutes involved as possibleWorldwide collaborative activityParticipation : 56 Institutes (39 in phase 1) • Australia • Austria • Canada • CERN • China • Czech Republic • Denmark * • France • Germany • Greece • Israel • Italy • Japan • Norway * • Poland • Russia • Spain • Sweden * • Taiwan • UK • USA * • New countries or institutes • * using Grid G. Poulard - GDB

  11. Grid in ATLAS DC1 US-ATLAS EDG Testbed Prod NorduGrid part of simulation reproduce part of full phase 1 & 2 Pile-up phase 1 data production reconstruction several tests reconstruction GRAT & Chimera G. Poulard - GDB

  12. DC1 production on the Grid • Grid test-beds in Phase 1 • 11 out of 39 sites (~5% of the total production) • NorduGrid (Bergen, Grendel, Ingvar, OSV, NBI,Oslo,Lund,LSCF) • all production done on the Grid • US-ATLAS-Grid (LBL, UTA, OU) • ~10% of US DC1 production (~900 CPU.days) • Phase 2 • NorduGrid (full pile-up production & reconstruction) • US ATLAS-Grid (BNL, LBNL, Boston U., UTA, Indiana U., Oklahoma U, Michigan U., ANL, SMU) • Pile-up: • ~ 10TB of pile-up data, 5000 CPU.days, 6000 Jobs • Reconstruction • ~ 1500 CPU-days; 3450 Jobs • UK-Grid: Reconstruction • ATLAS-EDG: on-going test (reconstruction) G. Poulard - GDB

  13. ATLAS Data Challenges: DC2July 2003 – July 2004 • At this stage the goal includes: • Full detector simulation with Geant4 • Pile-up and digitization in Athena • Deployment of the complete Event Data Model and the Detector Description • Use as much as possible the LCG Applications software (e.g. POOL) • Test the calibration and alignment procedures • Perform large-scale physics analysis • Use widely the GRID middleware • Use more and more GRID tools • Run as much as possible the production on LCG-1 G. Poulard - GDB

  14. Task Flow for DC2 data (Athena-ROOT) Athena-POOL Athena-POOL Athena-POOL H  4 mu ESD AOD Hits MCTruth Digits Athena Pile-up +Digits Athena Geant4 Athena HepMC Digits ESD AOD Hits MCTruth Athena Pile-up +Digits Pythia 6 Athena Geant4 HepMC Athena Digits Hits MCTruth Athena Pile-up +Digits ESD AOD HepMC Athena Geant4 Athena Byte-stream Digitization (Pile-up) Event generation Detector Simulation Reconstruction G. Poulard - GDB

  15. DC2 and Grid tools • Much work done: • Magda (BNL); AMI (Grenoble) used already in DC1 • Other tools in different stages of development and test: • GANGA (ATLAS-LHCb UK main effort) • Chimera (US) [exploit Virtual Data ideas (DC1)]; DIAL (BNL) • AtCom used to generate the jobs (batch and grid flavors) • GRAT (US); Grappa (US) • A coherent view of tool use and integration is emerging but need more work and thinking • RTAG11 being discussed in SC2 • MySQL is currently used as database • LCG-1 • We intend to use and contribute to validate LCG-1 components when they become available (R-GMA; RLS; …) • ATLAS-EDG becoming ATLAS-LCG task force G. Poulard - GDB

  16. DC2: a possible scenario • “Long” preparation phase • To put in place and validate the various components • LCG Applications software (e.g. POOL) • Simulation: GEANT4 • Event Data Model • Digitization; Pile-up; “Raw data” like (“Bytestream!) • Reconstruction (eventually with Lvl1 & Lvl2 trigger code) • Get experience with LCG-1 • Production of the “simulated” data • Worldwide exercise • Concentration of the data at CERN • How: • Hybrid production system (LCG-1; “classical” batch; other GRID flavors) • “DC2” • Reconstruction of data; production of ESD and AOD at TIER0 replicated to TIER1s • New calibration and alignment • Reprocessing • Analysis (needs a physics content). G. Poulard - GDB

  17. DC2:Time scale • End-July: Release 7 • Mid-November: pre-production release • February 1st: ”production” release • April 1st • June 1st: “DC2” • July 15th • Put in place, understand & validate: • Geant4 • POOL persistency & LCG App. • Event Data Model • Digitization; pile-up; byte-stream • Conversion of DC1 data to POOL and run reconstruction • Testing and validation • Run test-production • Start final validation • Start simulation • Pile-up & digitization • Transfer data to CERN • Start Reconstruction on “Tier0” • Distribution of ESD & AOD • Calibration; alignment • Start Physics analysis • Reprocessing G. Poulard - GDB

  18. ATLAS Data Challenges: DC2 • “DC2” • Reconstruction run onTier0 • Several streams • Express lines • Calibration and alignment lines • Different output streams • ESD and AOD replicated to outside sites • Out of Tier0 • Re-calibration • new calibrations and alignment parameters • Re-processing • Analysis • How? • Should be LCG-1 • Other possibilities not excluded G. Poulard - GDB

  19. DC2 resources (based on Geant3 numbers) G. Poulard - GDB

  20. ATLAS Data Challenges: DC2 • We are building an ATLAS Grid production & Analysis system • We intend to put in place a “permanent” Monte Carlo production system • If we continue to produce simulated data during summer 2004 we want to keep open the possibility to run another “DC” later (November 2004?) with more statistics G. Poulard - GDB

  21. Other ATLAS requests • TDAQ functionality and large scale performance tests. • Test period: March 2004 • Test duration: 4 weeks • Number of machines: • 1st 2 weeks: 250 machines • Last 2 weeks: 500 machines • Combined test-beam? G. Poulard - GDB

More Related