Data Acquisition Systems for Big Science

1. Data Acquisition Systems for Big Science Presentation for the NMI Dr John Coughlan STFC Rutherford Appleton Laboratory

2. Talk Outline • STFC Technology, Who we are • Data Acquisition for Big Science, CERN LHC • LHC-CMS Data Acquisition boards. • Next generation systems

3. STFC • Science and Technology Facilities Council, STFC • Largest of 7 UK Scientific Research Councils reporting to DIUS • 2,000 scientists & engineers at main site Rutherford Appleton Laboratory STFC, Rutherford Appleton Laboratory, Oxfordshire UK

4. STFC • Space Science, Lasers, X-Ray and Neutron sources, Particle Physics +… • Operate several world-class large-scale research facilities on site, e.g. VULCAN Laser, ISIS Neutron source, Diamond X-Ray Light Source. • UK Hub supporting activities of University researchers on other international facilities, e.g. ESA, CERN Geneva STFC, Rutherford Appleton Laboratory, Oxfordshire UK

5. Major Site Developments • Harwell Science and Innovation Campus… • New Detector Systems centre 2010/11 • Agreement for new ESA research centre STFC, Rutherford Appleton Laboratory, Oxfordshire UK

6. STFC Technology • STFC Technology Department • 300 Electronic & Mechanical Engineers with strong Scientific Expertise • Developing Instrumentation for Large Scale Science Experiments • Telescopes to Medical Imaging, Micro MEMS to Major engineering structures, Novel materials to Cryogenic Superconducting Magnets, Sensors for Space to Particle Physics. Electronics and Microelectronics… • Europe wide Microelectronics design support. EUROPRACTICE.

7. STFC Technology Electronics • PCB Board and System Design • FPGA Design. (Xilinx and Altera) • Embedded systems. • Board and System Test and System Integration • Microelectronics, ASICs, MAPS sensors • Current generation of Large Scale Projects for CERN completed, LHC • New projects starting R&D phase, European Free Electron Laser XFEL

8. CERN European Laboratory for Particle Physics. Geneva Big Science Large Hadron Collider LHC Understanding the Origin of the Universe Started Operation (briefly!) in 2008. No Black Holes yet…

9. CMS Experiment at LHC 2,500 Scientists & Engineers in collaboration 12,000 tons apparatus ~ 50 million electronic readout channels DAQ rate ~ 100 kHz

10. CMS Installation 200 metres Underground STFC Microelectronics 80,000 x Analogue Pipeline ASICs Extremely Radiation Hard VME Racks Water Cooled

11. One Collision LHC Data Acquisition • 40 Million Collisions every second. • Keep images from 100,000collisions every second. • Each image built up from 10 million analogue sensors. • Rarest new particles HIGGs ~ few dozen per year? • Need for Massively Parallel (FPGA) based Processing • ~ 1 TERA-Bytes / sec

12. LHC-CMS Data Acquisition Custom COTS

13. LHC DAQ Requirements • Performance driven by science & experiments. • Large channel counts. Large Form Factor PCBs • High bandwidths. Custom data protocols. • Specific Functions with Flexibility unknown science. FPGAs • Electronics developed in collaboration with scientists. Large communities. • In parallel with detector development. • Loose specifications. • Fully custom systems where performance and cost justify design investment • Long development cycles. Long operating times. 10+10 Years • Larger systems. Designs frozen early. Minimise risk.

14. LHC-CMS DAQ Board • Board Features • 9U x 440 mm (VME mechanics) • Optical Inputs, Analogue • 96 ADC Channels 10 bit @ 40 MHz • 10 x XC2V2000 • 24 x XC2V40 • 14 Layers • Double sided PCB • System has • 500boards (large nr for PP) • 24 Crates • 8 Racks • 50 KW • Delivered to CERN 2006 Data In ~ 3 GByte/s. Data Out ~ 200 MByte/s

15. LHC-CMS DAQ Board Development • Large boards. 9U VME Mechanics. High I/O count • Analogue & Digital • Rel large nr FPGA BGAs. 676 pins 1 mm pitch • Concerns for BGA assembly • Design for Test. Boundary Scan • Design started in 2001. • Prototypes (25 off) in 2003/4 before production (500 off) in 2005/6. • FPGA Virtex II. Fix technology early. • DCMs for channel synchronisation • DCI useful but power hungry • Double Data Rate I/O

16. LHC-CMS DAQ Board Specs Analogue components repeated on 2nd side • Board parameters: • - 9U x 440 mm VME64x form factor • Optical/Analogue/Digital logic ; 96 ADC @ 40 MHz channels • 14 layers (incl. 6 power). FR4. • Double-sided (secondary side with half of analogue components) • 6,500 components (most passives are 0402). Surface mount • ENIG metal finish, for BGAs. Pb/Sn • 20,000 connections ; 14,000 vias • 100 micron tracking, some 75 micron diff imp, min gap 100 micron, • 37 BGAs (larger FPGA 676 pins on 1mm pitch). All BGAs located on primary side. • Controlled impedance • - Boundary Scan JTAG all Digital devices 440 FED boards required for full CMS Tracker readout

17. FPGA Assembly on PCBs  3rd Batch Prototypes 2003 ALL 6 failed Boundary Scan on several BGAs. Shorts under BGAs. Rework failed too. 17,000 BGAs in system EU Tender process for PCB & Assembly. Exception Ltd 1 out of 500 production boards failed Tests 2006   

18. NewElectronics Cover Story LHC CMS CERN Industry Awards Gold Award : Exception Ltd

19. Other LHC-DAQ Boards • FPGA 130-90 nm generation • 1 Gbps serial data. • DDR2 • VME Form Factors • Conventional PCB manufacture. • Step change design required for Next Generation of Projects

20. Next Generation Systems European X-Ray Free Electron Laser XFEL DESY Laboratory, Hamburg Operating in 2013

21. Single Protein Molecules X-ray Imaging X-ray Diffraction Pattern Just before XRay pulse During the pulse After pulse 30,000 images per second each up to 16 MPixels

22. X-RAY Pixel Detectors DAQ For XFEL FPGA Detector Pixel Sensors SFP+ Image Builders 1 10G Fibre 30 m • 16M + tiled pixel detector at 30K frames/s -> 1 – 6 TByte/s • 128 x FPGA 40nm + 10G links • Off detector DAQ next gen Advanced Telecoms ATCA crates • 2008 - 2013… Scientific requirements ? 10 GBytes/sec x N cards 8

23. KEEP OUT K E E P O U T FMC RAM C O N N E C T O R FPGA MGT MGT C O N N E C T O R FMC FPGA C O N N E C T O R MGT MGT RAM KEEP OUT KEEP OUT 180 mm Image Builder for XFEL Demonstrator Advanced Mezzanine Card • AMC Form Factor. • Migrating to 8 FPGAs on 8U ATCA? • FPGA ~ 16 x 3-6 Gbps serial links • Analogue Cross Point for Image Building. 72x72 @ 3-6 Gbps • DDR2/3 ~ 2-4 GBytes • B/W 1-2 GBytes/sec In & Out • VITA57 FMC Mezzanine I/O • 2 x SFP+ opto TRx • 10 Gbps (XAUI or RXAUI PHY) • mTCAserial backplane X-point TX RX RX TX

24. Next Generation Board Issues • High speed diff pairs 3-6 Gbps. BGA pitch < 1 mm? • Reduce Fabrication Risk. Advanced PCB design and construction techniques. • Vias in pad, micro vias, Laser drill. Incremental build up layer PCB. • FPGA 40nm generation • FPGA to Memory interface. SO-DIMMs WASSO • Memory controllers Hard/Soft IP from FPGA vendors • 10 Gbps optical interfaces • 3-6 Gbps Serial Backplanes. • Power. Multiple POL. Analogue. • Decoupling caps. next gen FPGA packages. • Pb Free manufacture? • Tools Signal Integrity analysis, how to measure eye diagrams on board?

25. Design Tools • PCB Design Tool Flow based on CADENCE. • Fast serial design Signal Integrity analysis, HyperLynx • FPGA Design Flow Mentor Graphics • How to integrate PCB and FPGA design flow? Pin outs. • Need realistic FPGA designs to guide PCB layout, e.g. memory interfaces • Our expert (Paul Hardy) is here today

26. Summary • STFC Technology, Who we are • Data Acquisition for Big Science, CERN LHC • LHC Data Acquisition boards. • Next generation systems, XFEL

27. http://www.scitech.ac.uk/ STFC, Rutherford Appleton Laboratory, Oxfordshire UK

28. Spare Slides