FCTS protocol and related DAQ issues

1. FCTS protocol and related DAQ issues Gregory Dubois-Felsmann & Steffen Luitz24 April 2008

2. Where things stand • Last detector workshop: • We agreed that a reasonable way to move forward was for us to propose a sketch of the protocol for event data movement between the front-end electronics and the DAQ system • This is primarily intended to provoke concrete reactions and discussion from those who have to design and build the electronics. They will very likely know better than we will what the constraints are, and we expect that will lead to changes in the protocol.Translation: if you don’t like our proposal, don’t be upset, just say what’s wrong. • Original goal: • Circulate a written draft in time for feedback and rethinking by Elba. SuperB FCTS protocol considerations

3. Reality • We’re starting this now and are working through the most critical decision branch points in defining the protocol. Today we’ll describe what’s at issue. SuperB FCTS protocol considerations

4. Start from BaBar • BaBar-Note-281 (v1.1) described the protocol for communication between the ROMs and the FEEs • The Conceptual Design document for the FCTS describes the extension of this protocol through the FCTS system. • More detail is available in the “FCTS Architecture” note • For now we are considering only the event data protocol, corresponding to the “run-time commands” in the BaBar protocol (viz. Section 3.1 of BaBar-Note-281) SuperB FCTS protocol considerations

5. BaBar event data protocol • In normal triggered data acquisition running: • Signal from Level 1 trigger (GLT lines) goes to FCGM • If system is not “busy” (within 2.x us of previous command) or “full” (no FE buffers available) or “inhibited” (external signal), send L1Accept command through FCTS to DAQ crates & ROMs • ROM forwards L1Accept command to FEEs • FEEs capture a previously specified window of data into a buffer (in triggered, i.e., non-EMC, systems) • Sometime later, when available, ROM sends ReadEvent command to FEEs, which send back the earliest available buffer • This relies on a trigger delivery latency that is confined to a fixed jitter interval, with readout windows large enough to cover the uncertainty. SuperB FCTS protocol considerations

6. Two choices • Basic requirement is no intrinsic per-L1Accept deadtime, and deadtime due to “full” designed to be at most ~1% at the nominal 100 kTps trigger rate. This can be achieved in two ways… SuperB FCTS protocol considerations

7. Alternatives • BaBar-like fixed-latency model • Works (roughly) if it is possible to deliver L1Accepts at a minimum spacing equal to the shortest time interval by which the (assumed fully pipelined) trigger can distinguish consecutive events. • Essentially this means that there can’t be a meaningful limit on the minimum command spacing (100 ns ~ 1% deadtime) • You don’t have to be able to do this for an unlimited number of events, of course, just for bursts long enough that statistically you don’t get significant deadtime due to “full”. We will do some modeling of this. • In almost any scenario this does require being able to handle overlapping readout windows. • Variable latency with addressing by time, and queueing of triggers • In general requires additional pre-L1Accept “ring buffer” space, as closely-spaced triggers will effectively be delayed in transit. SuperB FCTS protocol considerations

8. Buffering • We assume two levels of buffering in the FEEs, in both models: • A continuously-running ring buffer upstream of the L1Accepts, long enough for the maximum trigger latency, in either model. • In Model 1, its length can be essential equal to the trigger latency (plus some constant offset) • In Model 2, it needs to be longer than this by enough to handle 99% of anticipated trigger bursts. We need to do modeling to be quantitative about this, but we anticipate that the answer will be O(10us) of additional capacity (i.e., roughly a doubling). • A post-L1Accept buffer. This would likely be constructed as a number of fixed-size slots as in BaBar’s design. The number of L1Accept slots required needs to be determined by modeling, but is very likely to be substantially more than the four in the BaBar design. • The driving parameters are that events must be able to be acquired about ten times faster than in BaBar, but the actual readout probably cannot be comparably faster (link speeds are only 2x faster, and we probably cannot afford to have significantly more ROMs). • The amount of such buffering needed would be somewhat larger in Model 1, by an amount we’ll need to determine by modeling. SuperB FCTS protocol considerations

9. Buffering II • Overlapping readout windows need to be supported • This has implications for the copy of event data from the ring buffer to the post-L1Accept buffer. • We propose that the protocol support copy by reference when windows overlap. This reduces the internal bandwidth required in the FEEs. • This requires the system to model an event as composed potentially of one or more by-reference segments followed by a by-value segment. At a minimum, the by-value segment of an event must be retained somewhere in the system until enough time has passed that a future event cannot need any part of it. This could be done either in the FEE or in the ROM, trading off complexity in the FEE against complexity in the FCTS protocol and the ROM. • We expect to propose one possible implementation as a reference point but describe alternatives as well. SuperB FCTS protocol considerations

10. Command protocol • BaBar • ROM-to-FEE commands are 12 bits: a 0, a 1, a 5-bit command code, and a 5-bit trigger tag (sequence number). At 60 MHz, this takes 192 ns to transmit. • FCTS-to-ROM commands are 104 bits: the full 56-bit 60MHz clock counter (the unique event key), the post-FCTS state of the 32 trigger bits, the 12-bit ROM-to-FEE command, and four flag bits. These take ~1.75 us to transmit, a significant fraction of the minimum command spacing. SuperB FCTS protocol considerations

11. Command protocols for SuperB Model 1… • The BaBar ROM-to-FEE command content is probably OK. • The BaBar ROM-to-FEE command timing is barely compatible with Model 1 at the same clock speeds. • Ideally, for 1% deadtime a 100ns command interval would be needed. Somewhat longer intervals could be acceptable in several scenarios: • If the trigger cannot generate separate trigger decisions that close together, then they don’t need to be processed, but the longer this interval becomes, the more necessary it becomes for the trigger itself to be able to handle overlapping events and make appropriate decisions (e.g., not vetoing an time interval that contains both a Bhabha and a physics event). • If triggers must be delayed in transit because of a somewhat longer command interval, this can be OK as long as the accumulated delay in the maximum burst that needs to be supported (from modeling) is compatible with the trigger jitter specification. SuperB FCTS protocol considerations

12. Command protocols for SuperB Model 1… • The BaBar FCTS-to-ROM command timing is completely unacceptable. • It would lead to intolerable trigger delivery delays. • The command word length could be somewhat shortened. Two possibilities: • The post-FCTS-decision trigger bits could be treated as event data, with the FCTS read out as an additional detector system. (This would preclude the ROMs’ making FEX or other decisions based on trigger content.) • The timestamp could probably be shortened from 56 bits to 40-45 bits by treating it as relative within each data run. This would require a run identifier to become part of the unique event key. • These measures are probably insufficient by themselves. The command delivery link would still need to be made several times faster. • This in turn might preclude combining commands with clock distribution in the FCTS. • If these paths are separated, the ROMs would likely have to be able to resync commands with the clock in order for a BaBar-like event build and out-of-sync detection scheme to work. • We are still thinking this through. SuperB FCTS protocol considerations

13. Command protocols for SuperB Model 2… • The ROM-to-FEE command word needs to be extended to include a ring buffer address field. • A length field may also be needed if the resolution of overlapping events is made a responsibility of the FCTS and the ROMs. • The time resolution of the addressing will need to be somewhere between the system clock period (e.g., 16 ns) and the minimum useful time resolution of the trigger (perhaps 125 ns). • The ring buffer in Model 2 needs to be somewhat longer than the intrinsic trigger latency, to accommodate queued triggers. Pending detailed modeling, we are guessing that a buffer depth of several tens of us should be adequate. • This means that the address field should be in the range of 8-12 bits. (A length field, if required, could be 4-8 bits.) • In Model 2, the resulting increase in command transmission time is almost irrelevant: it just adds somewhat to the required ring buffer depth. SuperB FCTS protocol considerations

14. Command protocols for SuperB Model 2… • The FCTS-to-ROM command word needs to be extended by at least the address field. • If the resolution of overlapping events is made a responsibility of the FCTS and the ROMs, the command word would need to be further extended to include a set of descriptors for the multiple segments of an event with overlap. • Each descriptor would probably be both an address and a length. • Here again slower link speeds just translate into additional ring buffer space required. • Even 2-3 us command transmission time (e.g., if the command word is much more complex than for BaBar) could be accomodated. SuperB FCTS protocol considerations

15. Status • We think both approaches are implementable. • At the moment we are leaning toward Model 2 - the time-addressable model - because it appears to significantly loosen the requirements on the performance of the FCTS and ROM-to-FEE command links. • The corresponding cost is the increased complexity of the FEE needed to support an addressable - and significantly larger - ring buffer. • As stated in February, we will propose a fully-triggered readout for all systems. SuperB FCTS protocol considerations