1 / 55

Computing in High Energy Physics

Computing in High Energy Physics. INSTR02 March 6 2002 Ren é Brun CERN. Computing in High Energy Physics. A wide subject I want to focus on what I know best Offline Computing and its relations with Online Data Storage & Management Software Frameworks Distributed Data Access & Analysis

davidmsmith
Télécharger la présentation

Computing in High Energy Physics

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Computing in High Energy Physics INSTR02 March 6 2002 René Brun CERN INSTR02

  2. Computing in High Energy Physics • A wide subject • I want to focus on what I know best • Offline Computing • and its relations with Online • Data Storage & Management • Software Frameworks • Distributed Data Access & Analysis • the GRID projects • Show Evolution and discuss trends Computing in HEP / INSTR02

  3. The OLD Model Experiment specific software in Fortran Event Generators Pythia, Isajet Data Structures Banks Managers ZEBRA BOS Geant3 PAW CERNLIB Computing in HEP / INSTR02

  4. Understanding Trends • Looking backward (past 20 years) is necessary to understand the current trends. • I am not reading in the crystal ball. My perception is subjective and biaised by my current involvement in the ROOT project. View is LHC-centric. • I will not risk myself beyond the LHC startup (or more precisely 2006/2007). • Some examples from the ALICE collaboration Computing in HEP / INSTR02

  5. Move to OO Programming • The first attempts to introduce the OO concepts in HEP software were made in 1989. These attempts were unsuccessful for many reasons: • OO languages were not mature and not widely available. • The performance of compilers could not compete with the Fortran compilers. • The introduction of a new system requires that this new system provides all the facilities of the previous system in a stable and proven environment. This is a chicken and the egg problem. Convincing the silent majority to adopt a new tool or framework is a challenging work. • The problem of Object Persistency had been totally underestimated or a wrong solution assumed. Last, but not least, the introduction of a new system could only be done with the strong support of a major laboratory and a long term commitment. The situation at CERN had been very confusing. Computing in HEP / INSTR02

  6. Languages In 1991, my boss said: “I do not know what will be the future language but I know that it will be called Fortran” Fortran: rapidly vanishing C++: the main stream language Java: much less used than expected C# the Microsoft big ? Computing in HEP / INSTR02

  7. Java: The end ? Java is the current maintream language for Web-based applications. Still not really portable. Java applications seen as very slow. MS wants to kill Java. JAS: Interesting features, but not used because expmts went to C++ and Linux. Computing in HEP / INSTR02

  8. ROOT Downloads 123,000 binaries download 1,300,000 clicks per month 35,000 docs in 12 months 2200 reg users in roottalk Computing in HEP / INSTR02

  9. Productivity C++ is a very powerful language, but extremely difficult to learn. Transition from the Fortran world has been very painful. Slow process because basic libraries not available in the early days Computing in HEP / INSTR02

  10. Fortran90 ??? A bit of History C++, Commercial Software • 1994/95 Official line projects start • RD44 (Geant4), RD45 (Objectivity) • Jan 1995. ROOT project starts in NA49,ALICE • Sep 1995. LHC++ (CERNLIB replacement) starts • Oct 1995. JAS (Java Analysis Studio) starts (SLAC) • Nov 1998. CDF adopts ROOT, followed by STAR, RHIC,etc • Jan 2000. LHC++ renamed to Anaphe • Oct 2001. CMS changes its baseline from Objectivity to ROOT • Feb 2002. ROOT is used by most HEP experiments • LCG/SC2 committees. ROOT official support In 1994, fundamental divergence of opinions in Application Software group in IT. The PAW/Geant3 team is dismantled. C++, Open Source ROOT Computing in HEP / INSTR02

  11. Current Situation • Rapidly evolving from the Old Model to the New Model described later. • Painful transition to C++ in most major collaborations. • A few collaborations have experience with ODBMS systems (ie Objectivity) • But the days of Objectivity in HEP are counted. • Some limited experience with Geant4 • Move to ROOT Computing in HEP / INSTR02

  12. Situation in the US • BaBar: pioneers with Objy. Complex situation.Kanga/Root used in small labs. Analysis with PAW. Growing Root community. G3-->G4 • Glast: Use Gaudi + Root (G3 ->G4?) • STAR: Root + MySQL + G3 • Phenix: Root + Objy(catalog) +G3 • Phobos: Root + Oracle + G3 • JLAB: Complex situations with the small expmts. Root in the main exp. G3. Attempts to use G4. Computing in HEP / INSTR02

  13. Situation in the US • CDF: Root-based I/O and analysis (online + offline). Final transition from Fortran to C++. File catalog in Oracle. G3, attempts to use G4 • D0: home-grown I/O system (using CINT). Move from PAW to Root for data analysis. SAM (Oracle) catalog. G3 • Minos: Root-based framework. G3 • BTeV: ? Attempts to use G4 • Miniboon: Fortran ? Computing in HEP / INSTR02

  14. Situation in Japan • Belle:Home grown tools in C++: • - non-OO Data Management System Panther • ADAMO-like structure with a C++ interface • - Analysis Framework : "B.A.S.F." • module and path structure • dynamic link of modules and I/O drivers • event-by-event parallel processing capability on SMP • - Communication tool over network : "NSM" • shared memory/message passing capability over network • to be used for slow control • Linear Collider: C++/Root-based. G3 Computing in HEP / INSTR02

  15. Situation at DESY • H1: Moving from Fortran-BOS to C++/Root in a BOS-compatible way. Use Oracle, G3. • ZEUS: Moving from Fortran to C++. Have used Objectivity. PAW analysis. Root growing. G3 • Hera-B: Has moved from Fortran to C++/Root • Tesla: Moving from Fortran G3 to C++/G4 Computing in HEP / INSTR02

  16. Situation with CERN non-LHC • Aleph, Opal, Delphi, L3: Fortran/Zebra/PAW/G3. • Compass: C++ home-grown system + Objectivity. PAW-->Root for analysis. G3 • Harp: Objectivity, G4 • AMS: C++ home-grown system. First guinea pigs with Objectivity. Use PAW, Oracle and Root, G3 • NA60: Root (Alice clone) Computing in HEP / INSTR02

  17. Situation with LHC • ALICE: Root framework + MySQL. G3 and G4. Alien. Abandoning G4. Interface to Fluka • ATLAS: Develop Athena framework (based on Gaudi). Objectivity & Root. G3. Experience with G4. • CMS: Orca/Cobra frameworks. Was based on Objectivity. Switching to Root. G3. Experience with G4. • LHCB: Gaudi framework with Root I/O. G3, starting with G4. Computing in HEP / INSTR02

  18. How Much Data is Involved? High Level-1 Trigger(1 MHz) High No. ChannelsHigh Bandwidth(500 Gbit/s) Level 1 Rate (Hz) 106 1 billion people surfing the Web LHCB ATLAS CMS 105 HERA-B KLOE CDF II High Data Archive(5 PetaBytes/year) 10 Gbits/s in Data base 104 CDF 103 H1ZEUS ALICE NA49 UA1 STAR 102 104 105 106 107 LEP Event Size (bytes) Computing in HEP / INSTR02

  19. ALICE Event/100 Front View of a simulated event with only 1/100 of the expected multiplicity Computing in HEP / INSTR02

  20. ALICE Event/100 Side View of a simulated event with only 1/100 of the expected multiplicity Estimated size of one raw event = 40 Mbytes Simulated event with hits = 1.5 Gbytes Time to simulate one event = 24 hours After L3, the DAQ will generate 1.25 GigaBytes/second 2 PetaBytes/year Computing in HEP / INSTR02

  21. CERN – Tier 0 (CERN - Tier 1) 2.5 Gbps Z 622 Mbps Y X Tier 1 155 mbps 155 mbps 622 Mbps Uni n Lab a Tier2 Uni b Lab c   Department  Desktop LHC Computing - a Multi-Tier Model 'X''Y''Z': RAL, IN2P3, FNAL, BNL, FZK(?), . . .  Organising Software: "Grid-Middleware" "Transparent" user access to applications and all data  Computing in HEP / INSTR02

  22. People Number of people and world-wide collaborations have a big impact on the Computing strategy Computing in HEP / INSTR02

  23. Worldwide Software Development • Code Management tools • Is CVS sufficient ? • Concurrent development implications • side-effects of unstable source code • concepts of old/pro/new obsolete • Nightly builds • Code inspection/Quality checks procedures • Bug reporting system • HyperNews • Web-based documentation & information Computing in HEP / INSTR02

  24. Move to OO programming True Objects Collections in C++ with built-in Object persistency and automatic schema evolution Typical scenario Banks systems Zebra,Bos Banks-like systems in C/C++ Hand-written I/O converters Data structures in Fortran common blocks Computing in HEP / INSTR02

  25. Dynamic Linking A Shared Library can be linked dynamically to a running executable module Experiment libraries A Shared Library facilitates the development and maintenance phases User libraries General libraries Application Executable Module Computing in HEP / INSTR02

  26. Class 1 Class 2 Class 8 Class 3 Class 7 Class 4 Class 6 Class 5 Whiteboard Data Communication Computing in HEP / INSTR02

  27. Task Task Task Task Task Task Task Task Task Task Task Task Task Task Task Task Task Task Task Task Task Generic task handling Det Det Det Det Det Det Det Computing in HEP / INSTR02

  28. A B event In KBytes From a simple nested parenthesis structure ((..)(..)(((..)(..)(…..)))) To complex Objects graphs with internal and external references Computing in HEP / INSTR02

  29. Lines of code ALICE: 600,000 C++ lines +ROOT + G4 +..+.. G4: 600,000 C++ lines ROOT: 700,000 C++ lines 700 classes 30 sub-systems Computing in HEP / INSTR02

  30. A CORE system Experiment Software High Level apps GRID toolkits Graphics, Histograms Virtual file Catalog GUI: Interpreter I/O subsystem Objects Dictionary: Introspection Data bases Event Store Computing in HEP / INSTR02

  31. Data Bases: model-1 • Put everything in an Object Data base • like Objectivity • or Oracle 9i • Choice of RD45 project • Many experiments initially following this line • Abandonned by most experiments recently • Solution not suited for interactive analysis Computing in HEP / INSTR02

  32. Data Bases: model-2 • Put write-once data in an object store • like ROOT in Streamer mode • Use a RDBMS for : • Run/Event catalogs • Geometry, calibrations • eg with ROOT<->Oracle interface • http://www.phenix.bnl.gov/WWW/publish/onuchin/rooObjy/ • or with ROOT <-> Objectivity interface • http://www.phenix.bnl.gov/WWW/publish/onuchin/RDBC/ • Use ROOT split/no-split mode for data analysis Combining 2 technologies ROOT Oracle Computing in HEP / INSTR02

  33. The LHC Computing Grid Project Next week at CERN Computing in HEP / INSTR02

  34. A ROOT-based CORE system agreed Experiment Software High Level apps GRID toolkits Graphics, Histograms Virtual file Catalog GUI: Interpreter I/O subsystem Objects Dictionary: Introspection Data bases Event Store Computing in HEP / INSTR02

  35. Calibrations Event Store histograms Run/File Catalog Trees Geometries ROOT + RDBMS Model ROOT files Oracle MySQL Computing in HEP / INSTR02

  36. Memory <--> TreeThe Tree entry serial number Memory T.GetEntry(6) 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 T.Fill() 18 T Computing in HEP / INSTR02 tr

  37. 0 1 2 3 4 5 0 6 0 1 7 1 2 2 8 3 3 9 4 4 10 5 5 6 6 11 7 7 12 8 8 13 9 9 10 14 10 11 15 11 12 16 12 13 13 17 14 14 15 18 16 15 17 16 18 17 18 Tree Friends Entry # 8 Public read User Write Public read Computing in HEP / INSTR02 tr

  38. Chains of Trees f0 f1 f2 f3 f4 f5 f6 f7 0 0 0 0 0 0 0 0 2 4 4 5 5 4 7 11 0 12 17 23 26 32 37 45 49 0 1 2 3 4 5 6 7 Binary search in table above find slot 4, local entry 2 T.GetEntry(2) in f4.root TChain ch(“T”); ch.Add(“f0.root”); ch.Add(“f1.root”); ch.Add(“f2.root”); ch.Add(“f3.root”); ch.Add(“f4.root”); ch.Add(“f5.root”); ch.Add(“f6.root”); ch.Add(“f7.root”); ch.GetEntry(28); ch.GetEntryWithIndex(12,567); Computing in HEP / INSTR02 tr

  39. Automatic Schema Evolution Computing in HEP / INSTR02

  40. Auto Schema Evolution (2) Computing in HEP / INSTR02

  41. Self-describing files • Dictionary for persistent classes written to the file. • ROOT files can be read by foreign readers (JAS) • Support for Backward and Forward compatibility • Files created in 2002 must be readable in 2015 • Classes (data objects) for all objects in a file can be regenerated via TFile::MakeProject Root >TFile f(“demo.root”); Root > f.MakeProject(“dir”,”*”,”new++”); Computing in HEP / INSTR02

  42. Online/Offline • Groups traditionally separated. • Growing synergy dictated by common needs for data bases and event storage, GUI, graphics, analysis systems, L3 software, input from detector simulation programs, tools. • Inter-process objects and messages passing requires a common object dictionary. • Growing synergy imposed by the Data Challenges. • A good example: ALICE Data Challenges Computing in HEP / INSTR02

  43. AliRoot Raw Data Simulated Data GEANT3GEANT4FLUKA DAQ Performance Monitoring FileCatalogue ROOT I/O CERN TIER 0 TIER 1 ROOT Regional TIER 1 TIER 2 ALICE Data challenges CASTOR GRID Computing in HEP / INSTR02

  44. Writing to local disk Migration to tape MB/s MB/s ALICE Data Challenge III • Need to run yearly DC of increasing complexity and size to reach 1.25GB/s • ADC III gave excellent system stability during 3 months • DATE throughput: 550 MB/s (max) 350 MB/s (ALICE-like) • DATE+ROOT+CASTOR throughput: 120 MB/s, <85> MB/s • 2200 runs, 2* 107 events, 86 hours, 54 TB DATE run • 500 TB in DAQ, 200 TB in DAQ+ROOT I/O, 110 TB in CASTOR • 105 files > 1GB in CASTOR and in MetaData DB • HP SMP’s: cost-effective alternative to inexpensive disk servers • Online monitoring tools developed Computing in HEP / INSTR02

  45. CPU CERN units Fantastic evolution, but not the only issue Intel-compatible processors only ? IA-64 processors ? Event-level parallelism Computing in HEP / INSTR02

  46. Disk GBytes Disk bandwidth still a problem IDE: 10->20 MB/s. Latency: > 5ms Keep on disk DST/AOD as much as possible In 2007, cost of one Petabyte of disk or tape should be the same Computing in HEP / INSTR02

  47. GRID activities Several Projects Starting in the US (iVDGL) and Europe (EDG) No time to describe these projects. Concentrate on one application http://alien.cern.ch Computing in HEP / INSTR02

  48. OSU/OSC LBL/NERSC Dubna Birmingham NIKHEF Saclay GSI CERN Padova Merida IRB Bologna Lyon Torino Bari Cagliari Yerevan Catania Kolkata, India Capetown, ZA ALICE GRID resources http://www.to.infn.it/activities/experiments/alice-grid 1000 physicists 37 people GRID-aware 21 institutions Computing in HEP / INSTR02

  49. ALICE GRID • File Catalogue as a global file system on a RDB • TAG Catalogue, as extension • Secure Authentication • Interface to Globus available • Central Queue Manager ("pull" vs "push" model) • Interface to EDG Resource Broker available • Monitoring infrastructure The CORE GRID functionality • Automatic software installation with AliKit • http://alien.cern.ch Computing in HEP / INSTR02

  50. Architecture Bookkeeping Data access File catalogue --./ | |--r3418_01-01.ds | |--r3418_02-02.ds | |--r3418_03-03.ds | |--r3418_04-04.ds | |--r3418_05-05.ds | |--r3418_06-06.ds | |--r3418_07-07.ds | |--r3418_08-08.ds | |--r3418_09-09.ds | |--r3418_10-10.ds | |--r3418_11-11.ds | |--r3418_12-12.ds | |--r3418_13-13.ds | |--r3418_14-14.ds | |--r3418_15-15.ds Tier1 Authentication Perl5 |--./ | |--cern.ch/ | | |--user/ | | | |--a/ | | | | |--admin/ | | | | | | | | | |--aliprod/ | | | | | | | |--f/ | | | | |--fca/ | | | | | | | |--p/ | | | | |--psaiz/ | | | | | |--as/ | | | | | | | | | | | |--dos/ | | | | | | | | | | | |--local/ | | | | | | | |--b/ | | | | |--barbera/ API ALICE LOCAL | |--36/ | | |--stderr | | |--stdin | | |--stdout | | | |--37/ | | |--stderr | | |--stdin | | |--stdout | | | |--38/ | | |--stderr | | |--stdin | | |--stdout Services (file transport, sync) ALICE ALICE USERS SIM Secure authentication service independent of underlying database D0 SOAP path char(255) |--simulation/ | |--2001-01/ | | |--V3.05/ | | | |--Config.C | | | |--grun.C T2526 dir integer(11) hostIndex integer(11) <fk> type char(4) entryId integer(11) <pk> dir integer(8) T2527 name char(64) type char(4) owner char(8) dir integer(8) ctime char(16) name char(64) comment char(80) owner char(8) content char(255) ctime char(16) method char(20) comment char(80) methodArg char(255) content char(255) gowner char(8) File catalogue : global file system on top of relational database method char(20) size integer(11) Files, commands (job specification) as well as job input and output, tags and even binary package tar files are stored in the catalogue methodArg char(255) gowner char(8) Central task queue size integer(11) ALIEN Computing in HEP / INSTR02

More Related