FECC-III INTEGRATION ISSUES

1. Stanford Linear Accelerator Center FECC-IIIINTEGRATION ISSUES Eric J. Siskind June 12, 2003

2. Project Goals • Replace Multibus-I micro hardware with commercial-off-the-shelf personal computers. • Replace special purpose accelerator networks (SLCnet, KISnet, PNET) with a single commercial-off-the-shelf TCP/IP network such as switched gigabit Ethernet. • Develop personal computer CAMAC interface to replace MBCD/MBCD-II.

3. New CAMAC Features • CAMAC interface differs from MBCD/MBCD-II in four major ways: • Two board design to support remote I/O; • Real-time preemptive hardware with recovery; • Maximizes CAMAC utilization when user software environment in PC is multi-threaded; • Large (i. e. comparable to existing micro) user programming environment within interface.

4. Hardware Architecture • Hardware consists of two circuit boards connected by up to 10 km of single mode fiber optics: • PCIL (“PCI Link”) board plugs into PCI option slot in PC; • FECC (“Front-End CAMAC Controller”) is now a stand-alone chassis with a single-width CAMAC module to access PDU triggers. • PC moves from micro’s current location to MCC computer room, leaving behind only FECC.

5. PCIL Features • Basically an intelligent special-purpose Network Interface Card. • Processor with DMA PCI block transfer master and DMA fiber optic link transmitter/receiver. • Makes no assumptions about nature of PC’s real-time operating system. • Semi-static assembly language PROM code with downloaded updates. • First generation hardware doubles as Alpha SLCnet interface.

6. PCIL Code Functions • Supports PC FECC message passing. • Supports FECC remote read/write access to PC memory with optional PC interrupt. • Performs function-specific scatter/gather DMA to/from PC memory for all classes of operations, avoiding unnecessary buffer copying by PC CPU. • Forwards trigger pattern from PCI registers to FECC (only in 1st generation; pattern forwarding in hardware in 2nd generation).

7. FECC Features • Logical successor to MBCD/MBCD-II. • Accesses host memory via point-to-point fiber optic link instead of parallel I/O bus. • 1st generation supports MBCD-II’s four strings of SLAC serial CAMAC crate controllers (5 MHz, bit serial, half duplex). • 2nd generation adds support for four strings of IEEE standard crate controllers (5 MHz, byte serial, full duplex) and dual Bitbus strings.

8. CAMAC Interrupt Recovery & Control • If a CAMAC block transfer is interrupted, hardware can repeat the previous write operation (address pointer load?) with updated write data before resuming the transfer. • Hardware can prevent an interrupt between CAMAC cycles transferring two 16-bit halves of a single 32-bit word. • Hardware can prevent an interrupt of a particular block transfer entirely (needed because of a current PIOP feature).

9. System Performance • Many PC jobs now can have simultaneously outstanding long CAMAC packages. • All packages are sent to FECC firmware ASAP, and parsed into their individual (block transfer) operations in a single crate (“packets”). • Packets are immediately queued to hardware for the cable accessing the target crate. • Cable operation begins as soon as any outstanding package has a packet requiring access to a crate on that cable. • Parallelism now achieved over multiple packages.

10. PCIL2 Intelligence • Analog Devices ADSP-21060 “SHARC” 40 MHz 32-bit DSP on 2nd generation PCIL. • 40k x 48 bit on-chip program memory; 64k x 32 bit on-chip data memory. • 256k x 48 bit external program/data memory. • 256k x 64 bit external DMA I/O data memory.

11. FECC3 Intelligence • IBM PPC405 250 MHz 32-bit PowerPC embedded in Xilinx FPGA on 3rd generation FECC. • 16k x 64 bit on-chip program memory; 16k x 32 bit on-chip data memory. • 2048k x 64 bit external program/data memory with DMA link/CAMAC hardware.

12. User FECC Programming • Real-time executive plus MBCD/Bitbus emulation uses ~15% of on-chip program memory, ~25% of on-chip data memory, and ~5% of off-chip RAM. • Remaining minimum of 15+ megabytes of memory on 250 MHz 32-bit processor exceeds capacities of existing 386/486 CPU boards. • Move existing 360 Hz interrupt driven code from micro to CAMAC interface to minimize CAMAC access delay. • C run-time environment with dynamic heap allocation; full environment and stack preserved over context switch.

13. 2nd Generation Hardware • PCIL2 uses 64 bit/66 MHz PCI slot. • Single specialized fiber optic link for PCIL2 FECC3 communication. • 125 megabyte/second full duplex link, but 9/19 used for hardware overhead. • All peripherals FPGA-based DMA except for Bitbus (programmed I/O). • Hardware based on one large FPGA per board, plus SHARC on PCIL2.

14. 2nd Generation Status • PCIL2 now debugged and tested. • Two PCIL2 boards built in 2001; three more in 2002. • FECC3 design in completed in 2002; FPGA delivery currently 29 August (5 month delay). • Initial build of two FECC3 boards in 2003; three more after debugging.

15. Bitbus Master • Existing master based on MCS-51 2 MHz 8 bit microcontroller with SDLC/HDLC serial link. • 8044BEM is an MCS-51 with mask-programmed firmware (iDCX-51 RTE plus Bitbus application). • FECC3 has dual-channel programmed I/O hardware (one PowerPC FIFO read or write per 32 bits) to send a packet to one slave and receive that slave’s response. • Poll list, counters, etc. maintained by new PowerPC code in FECC3. • Enhanced real-time support via reserved high priority buffers and multiple prioritized queues.

16. Schedule • Migration of 360 Hz processing in progress (TEG). • All optimizations not necessary at early stage. • Trying to deploy MPG plus one ordinary PC-micro in development system in January, 2004. • Hoping to replace MPG plus one ordinary micro in summer of 2004.

17. Simplified Hardware Configuration • No PCPC fast feedback (KISnet equivalent) network traffic. • Use 2nd generation PCIL-FECC hardware: • Include IEEE CAMAC and Bitbus; • Need PC with 64 bit/66 MHz PCI slot. • MPG  PC (PNET equivalent) pattern broadcast on dedicated point-to-point Ethernet link . • PC-based MPG (MP10/MP11) broadcasts pattern to existing micros via FECC3 BITbus interface reconfigured as a PNET transmitter; talks to SP00/SP01 and BIC via PC  VME reflective memory links.

18. Integration Projects • Build, boot, and debug iRMX-III in off-the-shelf personal computers. Provide minimal FECC debugging support. • Make real-time micro  micro data communications streams coexist with slower back-end  front-end traffic on switched TCP/IP network (deferred). • Move 360 Hz interrupt processing from micro to FECC (MPG + pattern consumer).

19. Integration Projects II • Change control of 360 Hz interrupt processing from shared memory to network link model. • Integrate PCIL-FECC CAMAC/Bitbus emulation. • Add support for real-time CAMAC/Bitbus; Modify fast feedback actuator to specify real-time priority for CAMAC/Bitbus operations. • Modify slower micro jobs to send larger packages to CAMAC interface to maximize parallelism. • Modify klystron job to protect PIOP CAMAC operations from interrupts.

20. Integration Projects III • Support logical to physical micro mapping. • Support PC access to file server delivering current versions of PCIL and FECC executable images appropriate to different types of micros.