1 / 20

SuperBelle Event Building System

SuperBelle Event Building System. Takeo Higuchi (IPNS, KEK) S.Y.Suzuki (CC, KEK). Belle DAQ Diagram. Belle Event Builder. COPPER/FASTBUS readout. Construction of event record Data transmission to RFARM. Configuration of SB Event Builder. CPR crate. CPR crate. CPR crate. CPR crate.

tana-marshall
Télécharger la présentation

SuperBelle Event Building System

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. SuperBelle Event Building System Takeo Higuchi (IPNS, KEK) S.Y.Suzuki (CC, KEK) 1st Open Meeting of the SuperKEKB Collaboration

  2. Belle DAQ Diagram Belle Event Builder • COPPER/FASTBUS readout • Construction of event record • Data transmission to RFARM

  3. Configuration of SB Event Builder CPR crate CPR crate CPR crate CPR crate CPR crate ROPC #04 ROPC #m ROPC #02 ROPC #03 ROPC #01 EFARM #01 RFARM #01 SW SW EFARM #02 RFARM #02 SW SW From our experience, the performance limit here is at 1-2Gbps EFARM #03 RFARM #03 SW SW … … EFARM #n RFARM #n … SW SW The m and n depend on the detector design.

  4. Regulation on FINESSE by EB • No event# skip / No event# shuffle • Any kind of data record must be sent out from the FINESSE upon the L1 trigger. • At least a capsule with event# tag (and empty body) must be generated even for a totally zero-suppressed event or an L2-aborted event. • Event # by the FINESSE must be increased monotonically by 1. … OK 8 7 6 5 4 3 2 1 … 11 10 9 8 5 4 3 1 NG  FATAL … 6 7 8 5 3 4 2 1

  5. Inside EFARM #xx • Multi-layer event building • Similar to the present EFARM configuration. EFARM #xx PC PC PC PC PC PC PC … … PC PC 1st layer 2nd layer i-th layer Exit layer

  6. Performance Study Inside EB (1) Master thesis of K.Ito (2005) 100BaseT 100BaseT 1000BaseT • P2P connection • Connection over SW N=8 GbE SW SW works fine P2P connection Connection over SW 100BaseT 1000BaseT @800Bytes/ev N=1..11 … Scalability up to N~10 GbE SW

  7. Performance Study Inside EB (2) Master thesis of K.Ito (2005) 100BaseT 1000BaseT N=1..20 N < 5 ~ 1stLayer Pseudo2ndLayer ROPC GbE SW Event rate Total data transfer rate

  8. Performance Study Inside EB (3) Master thesis of K.Ito (2005) N=1..10 N=5 N=5 2nd Layer ROPC Event rate Total data transfer rate 1st Layer 800 bytes/ev 1600 bytes/ev 3200 bytes/ev 6400 bytes/ev vs. # of 1st later PCs

  9. Concepts of Implementation • Switch from house-made technologies to generally existing technologies • E.g. (1): event buffering • Now: House made (shared memory + semaphore). • SB: Socket buffer within Linux kernel. • E.g.(2): state machine • Now: All EB nodes are managed by NSM. • SB: rwho or finger+.plan Only one NSM node to respond to master. This collect all EB nodes’ status via those commands. Less effort for documentation

  10. Concepts of Implementation Less effort for trouble shooting • Quit from complexity • NSM: Only one NSM node to respond to master. • D2: Own error reporting / logging  syslog. • No (or minimum) configuration file:the present EB requires large effort to add new ROPCsfor its multiple and complicated configuration files... • Common software for all EB PCs in each layer. We have just started the conceptual design.Following slides includes many unimplemented ideas.

  11. Network Link Propagation RO01 E101 E2 01 En 01 RF … COPPERs … RO0n E102 E2 02 COPPERs Network connections propagate at RUN STARTfrom DOWNstream to UPstream.  # or ports awaiting for connection can be unity.

  12. Determination of PCs to Connect Recall “no (or minimum) configuration file” • Idea #1: Determine EB PCs to connect from its own hostname + rule. • EB PC connects to all available (= accepting connection) PCs at run start, which match to the rule. • Two new schemes must be created: • Detection of PCs which are not booted but included to CDAQ. • Detection of PCs which are booted but excluded from CDAQ. … E.g.: E02_01 E01_11 E01_12 E01_13 E01_19 … E02_02 E01_21 E01_22 E01_23 E01_29 … E03_01 E02_11 E02_12 E02_13 E02_19

  13. Determination of PCs to Connect • Idea #2: “EB_CONFIG server”. • EB_CONFIG server knows which EB PCs are booted up and which are not. • EB_CONFIG server knows which subsystems are included and which are excluded (told from MASTER). • EB_CONFIG server knows which EB PCs belong to which subsystem (from a configuration file). • But this is what we like to avoid.

  14. Inside EB PC (Process Diagram)  upstream downstream  port #nnn EB PC xinetd fork(2) port #nnn connect to upstream port #nnn connect to upstream eventbuilder output todownstream … port #nnn connect to upstream eb_process ¶ An EB PC in the exit layer may have to output the built-up record to one of multiple destinations (RFARM PCs), which consequently means the EB PC accepts multiple connections from each destination PC. We have found a way to pass socket descriptor to another process. This enables us to perform event building, destination selection, and data transmission in a single process.

  15. Input Buffer • Use Linux kernel bufferinstead of house-made buffering scheme • Much easier implementation/maintenance. • Unlike when the present EB developed, very large buffer (~80MB) can be allocated in the kernel now. Now SuperBelle Insidekernel Sharedmemory Insidekernel Data Build Data Build recv shmop recv

  16. Output Buffer • Recent Linux provides information about how much unsent data remain in the kernel output buffer. • Established by SK DAQ team (Hayato , Yamada).  Enables dynamic traffic balancing. ioctl( sd, SIOCOUTQ, &val); unsent data idle EB PCin exit layer PC #00in RFARM #01 EB PCin exit layer PC #00in RFARM #01 Select data destinationto balance the traffic load. too busyto recv data PC #01in RFARM #01 PC #01in RFARM #01 idle PC #02in RFARM #01 PC #02in RFARM #01

  17. State Machine / Status Collection • Possible statuses of EB PC • Collection of EB PC statuses and EB Configuration From downstreamto the EB PC From the EB PCto upstream Ready to run rwho is used. EFARM side CDAQ side PC: EBNSM Collection ofEB PC statuses NSM command: EB_CONFIG … NSM reply:READY, NOTREADY … EB_CONF server nsmd Query: which EB PC is ready? to connect? included?

  18. Error Reporting and Logging INFO syslog(3) WARN EB Process /var/log/messages ERROR FATAL tail –f grep • Watch with • Throw error messages tothe NSM master.

  19. BASF Substitution • Separation of BASF from EFARM • Present EFARM strongly depends on BASF; BASF modules build up event records. May cause BELLE_LEVEL problem. • SuperBelle EFARM is not to use BASF to build up event. • Substitution of BASF • BASF-like substitution for online data quality check and/or L3 trigger use will be provided. const int (*userfunc[])(unsigned int *data, size_t size);

  20. Summary • We have started the design of event building farm for SuperBelle. • Regulation on the FINESSE is given from the EFARM. • Several performance studies about network data link are already made to input to the new EFARM design. • The new EFARM fully utilizes pre-implemented Linux functions instead our own invention. • The new EFARM tries to be self-configured to be free from configuration-file nightmare.

More Related