Outer Tracker Off-Detector Readout and Control Discussion

1. Outer Tracker Off-Detector Readout and ControlDiscussion John Coughlan SLHC Tracker Readout Meeting March 7th 2007

2. Starting Assumptions • Outer Tracker R > 60 cm • TOB &TEC (MSGCs) • Replace existing layers • Mini Strips • Readout of APV13 • Digital Readout • Just Readout (not Triggering) • Sparsification remains Off Detector ? • 50 nsec BX , 100 kHz L1 • Building blocks • Opto-Links 3-5 Gbps • Versatile Link GBT chipset • Based on material from Mark, Jan and Paulo from Feb Tracker Upgrade Workshop

3. Readout Channels • R > 60 cm 65% ~ 50 K APV25s ? • 10 Luminosity with 50 nsec BX -> 20 x track density • x 5 strips -> 4 x occ ~ 4% • ~ 50K APV25 -> 250 K APV13

4. Data Tracker Links • 40 K Uni Dir Pt2Pt links • Data ~ 80K APV25 • Synchronous System • Latency not critical APV FED • Readout FED & Control FEC decoupled

5. APV O/P Frame digital header 128 analogue samples Mark : Further up the readout chain 160 Mbits/sec (per chip – no sparsification) GBT off-detector rate = (no.of FE chips) x 160 Mbits/s digital link interface functionality DAQ Function multiplexing encoding and fast serialization sparsification here maybe? serial links 16 e.g. 20 FE chips on one link = 3.2 Gbits/s raw data (no sparsification / no encoding) if front end sparsification* (or faster links) then FE chips / link can increase Loss-less Compression ? simple data rate / FE chip calculation current APV data frame duration 7 msec for 140 samples digitize at 8 bits -> 1120 bits to shift out in 7 msec => 160 Mbits/sec 20 Ms/s readout -> 7 ms

6. Readout Rates : Sparsification Off Detector • 3.2 Gbps link @ 80% payload ~ 2.6 Gbps data • APV13 128 ch -> 160 Mbps • 16 x APV13 per GBT / link = 15 K GBT/ links • sFED : ORx 12 way x 8 ORx on 8U (feeding 8 x 12 MGBT FPGAs) • Input : 96 links/MGBT x 2.6 Gbps ~ 250 Gbps / sFED (cf FED ~ 25 Gbps) • Output (~4 x occ?) : / 10 -> 25 Gbps / sFED (cf FED 1.5 Gbps) Total System • ~ 250 K APV13 -> ~ 15 K links • ~ 160 sFEDs • ~ 10 sFED / ATCA crate -> 16 crates • ~ 300 Gbps / crate • ~ 2.5 Tbps to Filter Farm (cf FED 0.3 Tbps) Still a BIG system.

7. sFED Possible Implementation Keep FEC functions separate FED and FEC modules Communicate across backplane? ATCA Crate 8U Off Detector sFED Rear Transition Module ORx STTC GBT ORx ORx Cntrl/Mon ORx 12 x 3 Gbps ORx APV13 ORx ORx Power ORx SNAP 12 SDRAM Buffer FPGA Switch 10G Serial Backplane DAQ Crate Event Builder FPGAs MGBTs GBT PHY MAC Sparsification 1 per ORx ORx and FPGA on Mezzanine? Mezzanine Prototype

8. TTC Controls Tracker Links • 15 K Pt2Pt GBT links • Data ~ 250K APV135 • Non-Synchronous System • Latency not critical Pt 2 Pt • 320 control rings • 2,500 Bi Dir • TTC + Control/Monitor EC • TTC Latency critical • Upgrade • 320 x 0.65 x 5 ~ 1,000 rings • 8 K links • Or bigger rings? Pt 2 Pt

9. protocol eTRx oRx oRx TTC FPGA oTx splitter oTx protocol eTRx oRx oTx DAQ FPGA oRx protocol eTRx oRx oRx oRx oTx oRx protocol eTRx oRx In detector oTx Jan : Best of Both Worlds? • Broadcast downstream, Broadband upstream FEC FE Module PON Coupling Pt 2 Pt FED •  Far Fewer Links. Downstream PON. Don’t need upstream PON •  Every Data Link has dedicated TTC and Control.No more Rings. •  DAQ links no longer Synchronous (packets shared with EC mon) •  Couple Readout FED & Control FEC (EC cmds/mons)

10. TTC Controls Tracker Links • 15 K Uni Dir • Data ~ 250K APV135 • No longer Synchronous • Data and Monitor EC Pt 2 Pt • PON broadcast to 15K GBT • ‘FEC’ has OTx only • Each FE Hybrid has TTC/EC • No more rings • ‘CCU’ to chips via I2C PON

11. FEC I2C FED 16 x APVs Questions Simplex Pt2Pt Broadband for Data and Monitoring • How is this done? Protocols • Can we maintain data rates with protocols? FPGAs

12. Questions • Simplex PON for TTC and Control • Splitting ratios? How many FEC links. • How to partition FED and FEC functions and inter communications? • What will FE modules look like? Assume Less Variations. • Do we need full FEC logic on Data Links (PD Off Detector?) • Consequences for Partitioning schemes? • What will new TTC system look like? • Assume New Emulators (keep in Global Trigger crate)

13. sFED/FEC Integrate FEC functions on FED? ATCA Crate 8U Off Detector sFED Rear Transition Module sTTC OTx STTC Trigger Throttle ORx ORx 12 x 3 Gbps ORx ORx APV13 Monitor Cntrl/Mon Ethernet ORx ORx Power SDRAM Buffer FPGA Switch 10G Serial MPT 12 Backplane DAQ Crate Event Builder FPGAs MGBTs GBT Recv MAC ZS 1 per ORx ORx and FPGA on Mezzanine?

14. Final System Ideas • New sDAQ (sFEDs connected direct to Filter via Super Event Builder Network) • New sTTC (Broadcasting Filter Addresses to FEDs) • Crates • Just Mechanics, Power, Cooling. -> Control/Monitoring via Ethernet. • Serial Backplane based crates (Telecom ATCA , VME46?). • Less Slots (but wider) • Better Power & Cooling ? • Better control & monitoring ? • FED Event Builder Crate

15. First steps to sFED • Year 1 : with FPGA Development boards • Implement GBT receiver functionalities in FPGA fabric (Verilog models?) • Channel density vs FPGA resources • Fast FPGA Memory interfaces • Implement protocols for extracting Data and EC Monitoring packets in FPGA • Year 2 : Prototype sFED on small form factor Mezzanine? • PCB Serial Integrity Issues ; Clock for FPGA MGBT Issues • Year 3 : Aim to instrument for full Tracker Readout chain test? • Learn from existing systems ; ECAL DCC, GCT, Tracker Trigger , other expts • Who else is looking at sFEDs? Common CMS solution? a la FEC • Get some experience with ATCA (other projects)

16. Other tasks • Front End Modularity? • sFEC module Trigger , Controls • Consider Sparsification On Detector ?

17. Spare Slides

18. Jan: Versatile Link: Definitions & GBT oTx eTRx protocol oRx

19. Readout Rates : Sparsification On Detector • / 5? On Detector Total System • ~ 250 K APV13 • ~ 15 K -> ~ 3 K links • ~ 160 - > 80 sFEDs (limit of ATCA backplane switch) • ~ 8 crates • ~ 2.5 Tbps to Filter Farm (cf FED 0.3 Tbps)

20. Spare Slides • (Coexisting) Options: • User interface: Simple parallel bus • User interface: Industrial standard buses • Possible industrial standards: • Ethernet • PCIe (memory mapped bus) • Hyper Transport (memory mapped) • … Dedicated EC ASICor “Client” ASIC

21. Control Links : Present System • 320 CCU rings x 2 (redundancy) DOH x 4 links • 2,560 links • Half Tx and Half Rx @ 40 Mbps? • ~ 40 FECs • 80K / 320 ~ 250 APV25 / CCU ring • 50 K APV25 -> 250 K APV13 • Keep sFEC implementation separate from sFED • Is there a need to broadcast additional TTC info to FEs?

22. CCU Ring • APV25 • DCU • LD • PLL • MUX

23. Spare Slides

24. Spare Slides

25. Tracker Partition LTC TTCci TTCex APVE x 9 TTC crate (xN) FMM FED FED FED FED TTCoc (xN) Compact PCI crate FE modules FED crate FRLs (xN) DAQ PC FEC FEC FEC TTCoc FEC crate Kostas: Tracker Partition

26. Tracker Partition LTC TTCci TTCex APVE x 9 TTC crate (xN) FMM FED FED FED FED TTCoc (xN) Compact PCI crate FE modules FED crate FRLs (xN) DAQ PC FEC FEC FEC TTCoc FEC crate Kostas: Tracker Partition ?

27. CMS Optical Links web

28. Spare Slides FPGA