Taller de Altas Energías Oviedo 2009

1. Taller de Altas Energías Oviedo 2009 Trigger/DAQ at LHC Jorge F. de Trocóniz Universidad Autónoma de Madrid

2. Proton -Proton 2804 bunch/beam Protons/bunch 1011 Beam energy 7 TeV (7x1012 eV) 1034cm-2s-1 Luminosity Bunch Crossing rate 40 MHz Proton Collision rate ≈ 107-109 Parton (quark, gluon) l e + l New physics rate ≈ .00001 Hz Higgs Event selection: 1in 10,000,000,000,000 e - Particle Z o e + Z o jet jet SUSY..... e - Collisions at the LHC

3. Beam crossings: LEP, Tevatron & LHC • LHC will have ~3600 bunches • And same length as LEP (27 km) • Distance between bunches: 27km/3600 = 7.5m • Distance between bunches in time: 7.5m/c = 25ns Tevatron Run I Tevatron Run II

4. pp Collisions at 14 TeV at 1034 cm-2s-1 • 20 min bias events overlap • H  ZZ, Z  mm H 4 muons: the cleanest (“golden”) signature And this (not the H though…) repeats every 25 ns

5. Time of Flight c=30cm/ns; in 25ns, s=7.5m

6. Physics selection at the LHC

7. Trigger/DAQ Requirements • N (channels) ~ O(107); ≈20 interactions every 25 ns • need huge number of connections • need information super-highway • Muon, calorimeter and tracker information should correspond to each other • need to synchronize detector elements to better than 25 ns • In some cases: detector signal > 25 ns • integrate more than one bunch crossing of information • need to identify bunch crossing • Can store data at ≈ 102 Hz • need to reject most interactions • On-Line selection (cannot go back and recover events) • need extraordinary monitoring and understanding

8. ( ) T REJECTED ACCEPTED Triggering • Task: inspect detector information and provide a first decision on whether to keep the event or throw it out The trigger is a function of : Event data & Apparatus Physics channels & Parameters • Detector data not (all) promptly available • Selection function highly complex  T(...) is evaluated by successive approximations, or TRIGGER LEVELS (possibly with zero dead time)

9. Physics selection at the LHC L1 Trigger

10. Particle Signatures in the Detector

11. L1: Only Calo and Muon • Compare to tracker info • Pattern recognition much faster/easier • Complex algorithms • Huge amounts of data • Simple algorithms • Small amounts of data • Local decisions • Need to link sub-detectors

12. Muon Trigger Calorimeter Trigger RPC CSC DT HF HCAL ECAL CSC local trigger DT local trigger RegionalCalorimeterTrigger Patterncomparator trigger CSC TrackFinder DT TrackFinder GlobalCalorimeterTrigger Pipelined 40 MHz, Latency < 3.2 s 4+4  4  4  MIP+Quiet bits Global Muon Trigger e/, j, ET, ETmiss, … 4 (with MIP/ISO bits) Global Trigger max. 100 kHz L1 Accept Level 1 Trigger Structure (CMS)

13. L1 Muon Local Trigger (CMS)

14. Strong magnetic field in iron return yoke (~2T) efficiently discriminates muons as a function of pT Muon Tracks with3.5, 4.0,4.5,6.0GeV/c Perform tracking analysis in real time for L1 Muon Trigger: Track-Finder L1 Muon Regional Trigger (CMS)

15. Technologies in Level-1 systems • ASICs (Application-Specific Integrated Circuits) used in some cases • Highest-performance option, better radiation tolerance and lower power consumption (a plus for on-detector electronics) • FPGAs (Field-Programmable Gate Arrays) used throughout all systems • Impressive evolution with time. Large gate counts and operating at 40 MHz (and beyond) • Biggest advantage: flexibility • Can modify algorithms (and their parameters) in situ • Communication technologies • High-speed serial links (copper or fiber) • LVDS up to 10 m and 400 Mb/s; G-link, Vitesse for longer distances and Gb/s transmission • Backplanes • Very large number of connections, operating at ~160 Mb/s

16. BTI Finds alignment between hits at the right BX TRACO Finds correlations between inner and outer BTI segments Kills ghosts and selects “best” 2 muon segments for any BX Trigger Server Trigger Bus Trigger Boards L1 Muon Trigger: Local Segment-Finder

17. LUTS LUTS PHYSICS L1 Muon Trigger: Track-Finder Algorithm

18. Stratix FPGA on mezzanine VME Interface, Control & Readout -neighbor, ETTF, WS connections -neighbor connections Input Receiver CMS Muon Track-Finder Board

19. CMS Muon Track-Finder Trigger

20. L1 Calo Trigger: e/g Algorithm ET( ) + max ET( ) > ETmin Isolated “e/g” ET( ) / ET( ) < HoEmax At least 1 ET( , , , ) < Eisomax Fine-grain: ≥1( ) > R ETmin

21. L1 Calo Triggers: jet and  Algorithms • Issues are jet energy resolution and tau identification • Single, double, triple and quad thresholds possible • Possible also to cut on jet multiplicities • Also ETmiss, ΣET and ΣET(jets) triggers “-like” shapes identified for  trigger Sliding window: • granularity is 4x4 towers = trigger region • jet ET summed in 3x3 regions ,  = 1.04

22. Global Trigger • Global Trigger: a very large OR-AND network that allows for the specification of complex conditions. • Trigger Menu for LHC luminosity L=2x1033 cm-2s-1 • Thresholds chosen to yield of 16 kHz • DAQ bandwidth at startup 50 kHz • Safety factor of 3 • Available bandwidth split in four groups • Inclusive muon, di-muon • Inclusive electron, di-electron • Inclusive tau, di-tau • jets, ET, ETmiss, combinations } 4.7 kHz } 4.3 kHz } 3.0 kHz } 3.6 kHz

23. L1 Trigger Decision Loop • Synchronous 40 MHz digital system • Typical: 160 MHz internal pipeline • Latencies: • Readout + processing: < 1ms • Signal collection & distribution: ≈ 2ms Local level-1 trigger Global Trigger 1 Primitive e, g, jets, µ ≈ 2-3 µs latency loop Trigger Primitive Generator Front-End Digitizer Pipeline delay ( ≈ 3 µs) Accept/Reject LV-1

24. Signaling and Pipelining

25. Physics selection at the LHC DAQ

26. Trigger/DAQ systems

27. Detectors Detectors Lvl-1 Front end pipelines Lvl-1 Front end pipelines Readout buffers Readout buffers Lvl-2 Switching network Switching network Lvl-3 HLT Processor farms Processor farms “Traditional”: 3 physical levels CMS: 2 physical levels Online Selection Flow ATLAS, LHCb, ALICE CMS

28. Trigger/DAQ parameters ATLAS No.Levels Level-1 Event Readout Filter Out Trigger Rate (Hz) Size (Byte) Bandw.(GB/s) MB/s (Event/s) 3 105 1.5x106 5 150 (102) LV-2 3.5x103 2 105 106 100 100(102) 3 LV-0 5x106 2x105 4 40 (2x102) LV-1 2x104 4 Pb-Pb103 5x106 51250 (2x102) p-p 500 106 100 (102) CMS LHCb ALICE

29. Technology Evolution

30. Lvl-1 trigger Detector Front-ends Readout Switch fabric Controls Event Manager Farms Computing services DAQ: basic blocks • Current Trigger/DAQ elements Detector Front-ends, feed L1 trigger processor Readout Units: buffer events accepted by L1 trigger Switching network: interconnectivity with HLT processors Processor Farm + control and monitoring

31. Event Building • Form full-event-data buffers from fragments in the readout. Must interconnect data sources/destinations. Event fragments : Event data fragments are stored in separated physical memory systems Full events : Full event data are stored into one physical memory system associated to a processing unit Hardware: Fabric of switches for builder networks PC motherboards for data Source/Destination nodes

32. CMS Detector Readout: 3D-EVB Fed Builder : Random traffic Readout Builder : Barrel shifter

33. ATLAS: Level-2 and EVB • Regions of Interest (RoI): • If the Level-2 delivers a factor 100 rejection, then input to Level-3 is 1 kHz. • At an event size of 1 MB, this needs 1 GB/s • Dividing this into ~100 receivers implies 10 MB/s sustained – easily doable. • Elements needed: ROIBuilder, L2PU (processing unit), a lot of control and synchronization.

34. Physics selection at the LHC HLT

35. Event Filter: A Processor Farm • Basic elements: PC, Linux, Network • Despite recent growth, it’s a mature process: basic elements are all mature technologies. • Very cost-effective: • Linux is free but also very stable, production-quality. • Interconnect: Ethernet, Myrinet (if more demanding I/O); both technologies inexpensive and performing. • O(1000) processors.

36. CMS HLT Filter Farm

37. HLT Requirements and Operation • Strategy/design guidelines: • Use offline software as much as possible • Ease maintenance, but also best understanding of the detector • Boundary conditions: • Code runs in a single processor, which analyzes one event at a time • HLT (or Level-3) has access to full event data (full granularity and resolution) • Limitations: • CPU time • Output selection rate (~102 Hz) • Precision of calibration constants • Main requirements: • Satisfy physics program: high efficiency • Selection must be inclusive (to discover the unpredicted as well) • Efficiency must be measurable from data alone

38. HLT Regional Reconstruction (I) Global • process (e.g. DIGI to RHITs) each detector fully • then link detectors • then make physics objects Regional • process (e.g. DIGI to RHITs) each detector on a "need" basis • link detectors as one goes along • physics objects: same

39. HLT Regional Reconstruction (II) • For this to work: • Need to know where to start reconstruction • Seeding • For this to be useful: • Slices must be narrow • Slices must be few • Seeds from Level-1: • e/g triggers: ECAL • m triggers: Muon systems • Jet triggers: E/HCAL

40. HLT Example: Electron Selection (I) • Bremsstrahlung recovery: • Seed cluster with ET>ETmin • Road in f around seed • Collect all clusters in road  “supercluster” and add all energy in road • “Level-2” electron: • 1-tower margin around 4x4 area found by Level-1 trigger • Apply “clustering” • Accept clusters if H/EM < 0.05 • Select highest ET cluster

41. HLT Example: Electron Selection (II) • “Level-2.5” selection: add pixel information • Very fast, high rejection (e.g. factor 14), high efficiency (e=95%) • Pre-bremsstrahlung • If # of potential hits is 3, then demanding  2 hits quite efficient

42. HLT Example: Electron Selection (III) • “Level-3” selection • Full tracking, loose track-finding (to maintain high efficiency): • Cut on E/p everywhere, plus • Matching in h (barrel) • H/E (endcap) • Optional handle (used for photons): isolation

43. Online Physics Selection Event rate Level-1 HLT output Online Selection

44. Conclusions • The Level-1 trigger takes the LHC experiments from the 25 ns timescale to the 10 ms timescale. • Custom hardware, huge fan-in/out problem, fast algorithms on coarse-grained, low-resolution data. • Depending on the experiment, the next filter is carried out in one or two (or three) steps. • Commercial hardware, large networks, Gb/s links. • If Level-2 present: low throughput needed (but need Level-2). • If no Level-2: three-dimensional composite system. • High-Level trigger: to run software algorithms that are as close to the offline analysis as possible • Large processor farm of PCs running Linux. • Control and Monitoring is highly non trivial.

45. Graphical Summary