Panda Collaboration Meeting: FPGA-based Compute Nodes for High-Speed Data Acquisition

1. Panda Collaboration Meeting FPGA-based compute nodes • Motivation • Compute Node Features • ATCA • Fabric Interface • Base Interface • IPMI system: IPMB i2c bus • Remote Upgrade • Ethernet Performance • Summary / Outlook

2. Panda Collaboration Meeting Motivation • Interaction rates of the order of 10MHz typical event sizes 4 to 8 kB.  raw data rates 40GB/s - 200 GB/s • high flexibility and selectivity • continuously sampling data acquisition • Event selection in programmable processing units • Connections via high speed networks • (1 Gbps)

3. Panda Collaboration Meeting Main Features Universal highly configurable and scalable hardware platform for multiple applications • Large Compute Power • 4 Virtex-4 FPGAs with local DRAM • Total Bandwidth ~ 26Gbps • 13x RocketIO to backplane • 4x Gbit Ethernet • 4x Optical Links • 2 Embedded PowerPC in each FPGA • Slow control • Firmware update • ATCA compliant

4. Panda Collaboration Meeting ATCA • Advance Telecomm Computer Architecture • PICMG standard • 19” Rack: 14 slots – 8U • Air flow cooling -- fans • Rough Mechanics, • Shelf management • Shelf Manager • IPMI Bus (i2c) • IPMI Controller on each board • Power ~ 200W/slot • Full mesh backplane • High Speed differential lines

5. Panda Collaboration Meeting ATCA (2) Full mesh fabric interface. Main data transport channel Dual Star – Base interface Management Communications ZONE 1: Power and IPMI Bus

6. Panda Collaboration Meeting Each node is connected to each other High Speed differential lines (up to 10gbps) 13 RocketIO links 1 link to any other node 1.25 Gbps Total bandwidth ~ 16Gps Data Transport Full Mesh Fabric Interface

7. Panda Collaboration Meeting Dual Star Base Interface • Basic channel for configuration • 100Mb Ethernet • Firmware updates • Normal CPUs as hubs

8. Panda Collaboration Meeting Shelf Manager • ATCA shelf have an intelligent manager • Collects status information of the crate • Power consumption • Cooling capacity • Power Available • Negotiates power usage of every board

9. Panda Collaboration Meeting Shelf Manager (2) • Communication IPMI protocol • Hardware layer: i2C • 2 serial wires: Serial Data / Serial Clock • Replicated bus to increase availability IPMB_A IPMB_B

10. Panda Collaboration Meeting Shelf Manager (3) Example of POWER NEGOTIATION • “COMPUTE POWER PROP.” • Check that the board is there • “GET POWER LEVELS” • Up to 20 level supported • For only 2 • IDLE (P<10W) • RUN (P~200W) • “SET POWER LEVEL” • “RENEGOTIATE POWER” • Issued by the boards • Shelf manager start the negotiation

11. Panda Collaboration Meeting Remote Upgrade M. Liu, II Physikalisches Intitut (Gießen) • Power on and boot the system normally • New firmware received via Ethernet by the Linux in the PowerPC • The Linux OS has write access to the FLASH memory - New firmware written • Reboot the system and the new HW and OS will start. When failing during the upgrading procedure, a backup mechanism is needed: • Backup firmware selected by a switch • May by the IPMI Controller…

12. Panda Collaboration Meeting Ethernet Performance M. Liu, II Physikalisches Intitut (Gießen) PERFORMANCE MEASUREMENTS: XILINX evaluation board ML403 with a Virtex-4 FX12 • Point-to-point connection to a PC • Results: • Protocol Direction Throughput (Mbps) • TCP Board -> PC 297.80 • TCP PC -> Board 316.58 • UDP Board -> PC 375.33 • UDP PC -> Board N/A Performance bottleneck: Embedded PowerPC

13. Panda Collaboration Meeting Compute Node

14. Panda Collaboration Meeting Compute Node

15. Panda Collaboration Meeting Compute Node

16. Panda Collaboration Meeting Compute Node

17. Panda Collaboration Meeting Summary & Outlook • Panda DAQ requirements cannot be fulfilled with normal computers • The FPGA-based compute node will provide necessary the bandwidth and computational power • Concept is settled • Layout phase already started (Beijing) • Subsystems already tested • Ethernet, Remote Upgrade • But we lack of experience with the IPMI protocol