1 / 19

Performance and Lessons from the CMS Global Calorimeter Trigger (GCT)

Focus: High Speed Serdes. Performance and Lessons from the CMS Global Calorimeter Trigger (GCT). Introduction. GCT hardware challenges over the last 2½ years. High Speed Serdes Data rate = 1.28 Gb/s Link speed = 1.6 to 2.0Gb/s Two problems Clock routing inside FPGA

jarah
Télécharger la présentation

Performance and Lessons from the CMS Global Calorimeter Trigger (GCT)

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Focus: High Speed Serdes Performance and Lessons from the CMS Global Calorimeter Trigger (GCT) GCT Performance & Lessons: Greg Iles

  2. Introduction • GCT hardware challenges over the last 2½ years. • High Speed Serdes • Data rate = 1.28 Gb/s • Link speed = 1.6 to 2.0Gb/s • Two problems • Clock routing inside FPGA • Reflections on copper links • Latter forced redesign • New design for GCT-GT links RCT to GCT Links: Optical links based on 8B/10B GCT to GT Links: Legacy links from old GCT project DC coupled electrical links GCT Performance & Lessons: Greg Iles

  3. RCT to GCT Source card receives 2 RCT cables with 32 differential ECL pairs each Leaf card used to receive 32 fibres @ 1.6Gb/s 8B/10B encoding with CRC check Link operating synchronously to TTC clock (80MHz) Rx-Elastic buffer bypassed for low latency operation. Patch Panel Approx 15m of fibre GCT Performance & Lessons: Greg Iles

  4. Discovery • Detected CRC errors in USC55 • Rare, but reproducible • Linked to particular 1 out of 16 MGTs • Hardware verification tests OK • Not identical test because it used local oscillator. • Moved system back to Lab • Tried different TTC clocking methods • Suspected clocking scheme or QPLL • req. 40ps pp, qpll < 50ps pp • Opted to run links with local 100MHz oscillator. GCT Performance & Lessons: Greg Iles

  5. 100MHz Links Low Latency Clk Domain Bridge Inserts extra commmas Lose ~1 bx in latency here Data @ 80MHz Data & Extra Comma @ 100MHz 2.0 Gb/s Serial Link TLK2501 MGT Data & Extra Comma @ 100MHz Gain ~1 bx in latency here because link running faster Fifo / DualPort RAM If ‘extra’ comma remove enable Data @ 80MHz Run at 100MHz – But still problem persists – NOT TTC clock issue ! GCT Performance & Lessons: Greg Iles

  6. Changed to asynchrounous mode Problem persists Arggghhhh !!! Reverted to very old firmware Links work OK… The implication is a timing error within the FPGA Look at fifo bridge from 100Mhz to LHC-80MHz More info from Xilinx App notes: XAPP763, XAPP670 Local Clock Resources Xilinx Virtex II Pro XC2VP70 Limited number of global clock nets. Use MGT local clock resources. GCT Performance & Lessons: Greg Iles

  7. MGT Rx Recovered Clk (Local) For low latency operation use MGT Rx Recovered clock and external fifo 16 MGTs in total: Local clocks = 10 Global clocks = 6 GCT Performance & Lessons: Greg Iles

  8. Local clock fix: Removed async signal used in 100Mhz domain, but sourced from 80MHz domain. Added CRC checker to 100MHz domain Later problems with Global clock: Force only local clk resources Low latency operation can be non trivial Doesn’t follow “standard” design Bypasss of the rx elastic buffer can be challenging Use oscillator to provide low jitter reference clock Part 1: Conclusios END – Part 1 RCT to GCT links NEXT-Part 2: GCT to GT links GCT Performance & Lessons: Greg Iles

  9. GCT to GT • Legacy link from original GCT. Required by GT • National Semiconductor DS92LV16 • Running just above spec (i.e. 80.1 v 80 MHz ref clock) • DC coupled • Uses Infiniband x1 cable, 100 Ohm Differential • Original testing with 3.0m Leoni cables • Final system required extra cables, but unable to obtain more. • Alternative supplier found: 1.5m Amphenol cables delivered. • Detect errors with random patterns Problem detected when we changed cables, but NOT a cable issue Thanks to Jan Troska and Francois Vasey for allowing us to borrow a high bandwidth scope GCT Performance & Lessons: Greg Iles

  10. Cable + Connector Trace measured with 1.5m Amphenol cable, connector and 100 ohm termination resistor Threshold at +/- 100mV for eye diagram data Measured across termination res. GCT Performance & Lessons: Greg Iles

  11. Powered & Populated Board Trace measured on current GTI card in loopback mode GCT Performance & Lessons: Greg Iles

  12. Reflections Leoni 0.5m, reflection ~226mV Leoni 2.0m, reflection ~168mV Amphenol 1.5m, reflection ~ 252mV Leoni 3.0m GCT Performance & Lessons: Greg Iles 12

  13. Try not to operate parts at specification limit DC balance with 8B/10B (or equivalent) Optically isolate Try to have just 1 transmission line i.e. Do not have 1 inch of FR4 + 1m cable + 1 inch of FR4 Links to GT will be replaced. Part 2: Conclusion END – Part 2 GCT to GT links NEXT-Part 3: New GCT-GT Interface GCT Performance & Lessons: Greg Iles

  14. Optical Global Trigger Interface Why necessary In original GCT project the GCT-GT interface was at the extreme limit of design specs Switching to to a shorter cable was sufficient to break the system. Capable of both: Transmitting (Tx) Receiving (Rx) OGTI: Replacement for GTI card GCT Performance & Lessons: Greg Iles

  15. Technology choice • Xilinx Virtex 5, XC5VLX110T-3FF1136C • 3rd generation Xilinx serial link – 3.75 Gb/s • Can run at higher speed to reduce latency. • Baseline to run links at 2.4Gb/s • 64% max spec • POP4 fibre optic transceiver • Transceiver: 4 in, 4 out • 850 nm multimode, 1 to 3.125 Gb/s • Baseline to run links at 2.4Gb/s • 77% max spec • Two suppliers • AvagoTech: HFBR-7934Z • Zarlink: ZL60304 GCT Performance & Lessons: Greg Iles

  16. OGTI POP4 (optics) HFBR-7934Z or ZL60304 x4 Tx or Rx Xilinx V5 XC5VLX110T- 3FF1136C Cross-point switch allows choice of upto 4 clocks Dual CMC header (~340 I/O) GCT Performance & Lessons: Greg Iles

  17. Theoretical Mininmum Latency Notes: (a) Latency calculated from clk edge sampling data into SerDes/FPGA until FPGA fabric (b) The numbers are obtained from datasheets and theoretical performace. (c) Assumed 1.0 bx here, but could be 2.0 depending on sync method. (d) No elastic buffer. Tx = 9.5 x RXUSRCLK, Rx = 10.5 RXUSRCLK (e) Assumed ¼ bx for IOB + 2 link speed clks (4 x ½). Based on current 80MHz to 100MHz bridge. (f) All numbers worst case, but no contingency. (g) block ram performance dependent. GCT Performance & Lessons: Greg Iles

  18. Future • Designing & testing at > 5Gb/s is unpleasant • e.g. the inductance of a few mm of scope probe is a problem • Xilinx recommends stubs on vias are removed. • rememeber that average PCB board is only 1.6mm thick ! • Intel, HP, IBM, Sun Microsystems, Luxtera, Kotura all working on silicon photonics • i.e. replace high speed electrical links on the PCB with optical interconnects. • Others better placed to comment on this. GCT Performance & Lessons: Greg Iles

  19. End GCT Performance & Lessons: Greg Iles

More Related