Weaknesses of the LHC Machine Protection System

1. Weaknesses of the LHC Machine Protection System Bernhard Holzer, CERN BE-ABP ... what a MPS should do: 2 major tasks * protect the machine in case of hardware / software failure * protect the machine in case of... " the experts " Personal Definition: I consider a quench already as something that should be avoided

2. 1.) Practical Applications of Murphy's Law: ... just some examples from RHIC: stories that you normally don't hear in EPAC reports - wrong sextupole polarity in the yellow ring (systematically) due to cpu problem of the machine physicists - BPM signals between the two rings interchanged ... again systematically - wrong BPM polarity at single BPM's - dead BPM ... indicating still an oribt offset of some mm --> orbit correction algorithm tries to "compensate" and in the end there are indeed 10 mm real offset. - vacuum valves are indicated as open ... but in reality are closed ... a nice beam dump - aluminum foil inside the vacuum chamber ... they just forgot to take it out - horizontal orbit correctors distort significantly the beam momentum on the first turn - Luminosity with 110 Bunches is not higher than with 55 bunches ... injection kicker diluted the transverse beam emittance

3. 2.) What are we talking about ??? Experience from HERA: the Machine Protection System has to handle a large number of "events" ... larger than I myself expected ... reasons spread over all hardware components beam current beam loss alarms , very fast beam loss alamrs ( < 5ms) and quenches in the first HERA run years per week.

4. 2.) Where do the problems come from ... just a number of most prominent examples: BPM's BLM's Power Converters RF (...can lead not only to dc current but to fast losses) Vacuum Experiments (!) Operateurs (!) Weakness of the MPS: clear enough: we should not forget any component but the MPS is only watertight if the hardware is perfect the logic of the software is ok if the protection system is redundant !!

5. Simple Example: the BPM's ( ... sorry Rhodry) BPM's are the backbone of the machine diagnostics system but they can be dead, show the wrong polarity, develop an offset and this can change spontaneously MPS might recognise a dangerous orbit ---> trigger the dump Orbit correction loop might compensate via local steering ---> BLM alarm / quench Example RHIC / HERA local offset in a BPM during Lumi-Run: Δx ≈16 mm leading to several quenches at injection and flat top

6. Lapidar Example: the BPM's 90° 90° Special situation: 90° lattice cite of the logbook 180 deg-bump check: WR 579 CX -5A WR 626 CX +13A WR 673 CX -5A

7. 3.) What can go wrong ? a rough statistics of 20 years HERA Injection: too early (during magnet cycle) too late (during accleration) into a filled bucket (timing problem) with kicker/septum off with magnet at transferline off after wrongly applied injection correction ... why ??? with closed collimators with closed vacuum valve with wrong magnet polarity (after maintenance day) Acceleration: failure of persistent current compensation errors in ramp correction tables tune jump during polarity switch of a quadrupole collimators too close to the beam head tail problems (chromaticity correction) magnet failures Luminosity: aperture limitations due to RF fingers beam quality issues: beam beam spoils the emittance (up to beam losses at the aperture limit) orbit correction loop: coil at limit or off dedicated beam orbit steering coasting beam (rf problems) failure at dump kicker failure of dump timing system collimator control defect (radiation problem) error in BLM / BPM signal processing (server) vacuum valve closes during luminosity run Nota bene: each of these errors lead to a beam loss alarm or quench

8. 4.) Nice example, because it was unexpected: strong development of dc current (coasting beam) due to rf noise DC beam contribution broken connection between rf pre amplifier & main driver in the tunnel ..."excellent" noise amplification ... driving DC contribution ... spoiling several luminosity runs accumulating up to 20 % DC contribution ... scraping ... did not solve the problem ... problem for the dump gap sudden jump of the rf timing by 18 ° dc current develops after a while sudden jump of the rf timing by several bunch positions dump kicker gap filled

9. 5.) Detection of Beam Losses Example HERA-p loss pattern around the storage ring beam losses seen by a single BLM failure of standard magnet (dipole /quadrupole) beam losses seen by a single BLM failure of a critical power converter --> very fast losses --> quench cannot be avoided --> and eventually damage of components Problem: MPS was not redundant in special cases a single system (eg. the BLMs) is not sufficient for the machine protection. Solution ... in special cases: FMCM direct & fast link between power converter & dump system

10. 6.) Possible Weakness of the LHC Machine Protection System ... ? Analysis of fast beam losses (A. Gómez) Phase space deformation in case of failure of RQ4.LR7 Short Summary of the studies: quench in sc. arc dipoles: τloss =20 - 30 ms BLM system reacts in time, QPS is not fast enough quench in sc. arc quadrupoles: τloss =200 ms BLM & QPS react in time failure of nc. quadrupoles: τdet = 6 ms τdamage = 6.4 ms failure of nc. dipole: τdamage = 2 ms → FMCM installed

11. Possible Weakness of the LHC MPS: Analysis of fast beam losses (A. Gómez) worst case: nc. dipole magnets: RD1.LR1 / LR5 simulaion of beam losses due to failure of RD1 damage level reached after 25 turns τBLM react.≈τdamage FMCM intsalled ... but redundancy does not really exist ... does it make sense to contemplate about a fast AC beam current monitor in LHC ??? experience is excellent: combination of fast FMCM and AC-BM installed at HERA in 2003/2004

12. 7.) Possible Weakness of ANY Machine Protection System ... ? ... the human beings HERA run year 2007 number of beam dumps and the reasons for it water cooling rf (4 systems) ... and the operators cryo systems power converters

13. 7.) Possible Weakness of ANY Machine Protection System ... ? ... the human beings especially problematic: the Monday morning effect, in other words: the experts * are actions that are dangerous really inhibited by the MPS ??? * is it possible to trigger actions from outside the CCC ??? eg. wire scanner / collimator from the office eg. power converters actions on site at the local controller Examples: correct bump but wrong IP --> BLM alarm local change of magnet currents --> BLM alarm wrong files in the sequencer --> spoils the machine run for a day !!! (still today I could kill the person) firing the wires for demonstration from the office only the expert can retract the collimators without warning ... and he did * is it possible to stop actions of the control system ? Can the operator or the MPS stop / inhibit orbit corrections / bumps / sequences in case of trouble Does / Should the MPS communicate with the control system ??

14. 8.) An Evident Weakness of the Machine Protection System ... ? ... its complexity Beam Current Current Safe LHC Monitors Energy Parameters Energy DCCT Dipole Injection Current 1 SPS Extraction Energy Beam Energy Kickers Interlocks Tracking SafeBeam DCCT Dipole Flag Current 2 TL collimators RF turn clock BLMs aperture Beam Dumping System BLMs arc Collimators / Absorbers Beam Dump Trigger Access Safety BPMs for Beam Dump System NC Magnet Interlocks Discharge LHC BPMs for dx/dt + dy/dt Switches Beam dI/dt beam current Interlock System dI/dt magnet current Cryogenics essential RF + Damper circuits Quench LHC Experiments Protection Powering Interlock Vacuum System auxiliary System circuits Power Converters Screens Operators Software Interlocks AUG Timing Software Interlocks UPS

15. 8.) An Evident Weakness of the Machine Protection System ... ? ... its complexity 140 "user systems" can trigger the alarm, each containing sometimes 1000 single devices infinite number of possible alarms need systematic checks establish procedures for testing, masking, book keeping ... "issue tracking" needs a lot of self disciplin some bad HERA examples 1994 One Mega Quench (beam induced) the head of the machine disabled the BLM system because of too many false alarms 2005 ... too many false alarms ... will be ignored

16. 8.) An Evident Weakness of the HERA Machine Protection System ... ? ... its complexity stable luminosity run in 2005 nice background situation, good lifetime, everybody is happy ... and the alarm system would like to dump the beam. the real problem: alarms are masked and ignored

17. 9.) For the Fun of it: Weakness of the HERA Machine Protection System ... ? ... its experiments provocative statement: there are a number of secondary collimator jaws and as primary collimators there are the FPS stations LHC Experiments Beam Does the MPS control / inhibit the experimets ?? Interlock System Operators Software Interlocks

18. 10.) Résumé: What we should avoid ... but what happened in other machines

19. 10.) Résumé: The LHC MPS, just some keywords for the coffee break * complexity: establish procedures for testing, masking, book keeping needs a lot of self disciplin * redundancy: how many independent alarms do you get in case of failure ( AC Monitor ? ) * don't forget the human beings: they need information & training * avoid fake alarms What we should avoid ... but what happened in other machines 5 mm grove in the HERA proton collimator detected in 2003 after many years of operation. t.q.m.m.