Progress on EXPReS at JBO

1. Progress on EXPReS at JBO EXPReS Board Meeting, Utrecht, Jan 2008 Ralph Spencer and the JBO team

2. Contents • Outline of JBO project • Manpower • Protocol work • Progress on iBOBs • Handover to Richard. Progress with iBOBs at Jodrell

3. Outline of Work at JBO • NA4 Outreach work – Alastair Gunn - completed • SA1 Connection of 4 e-MERLIN telescopes to JIVE (see under Fabric) • SA2: The 10 Gbps data link equipment available, awaiting cross connection in Manchester between academic and commercial links • Fabric: Protocols for high speed networks –Stephen Kershaw : PFLDNet 2008 Manchester 5-7 March http://www.hep.man.ac.uk/PFLDnet2008/aboutManchester.php • Fabric: Input to e-MERLIN correlator at 4 Gbps Progress with iBOBs at Jodrell

4. E-MERLIN---e-VLBI system Progress with iBOBs at Jodrell

5. January 2008 Network protocols Stephen Kershaw

6. TCP • Is TCP suitable for transferring real-time data? • If data is lost TCP will slow transmission and retransmit. • Demonstrate the feasibility of using TCP for real-time applications Progress with iBOBs at Jodrell

7. CUBIC TCP variants • TCP using different algorithms for the window/throughput reduction • Some variants allow more efficient use of bandwidth and appear more suitable for real-time applications HighSpeed Progress with iBOBs at Jodrell

8. TCP variants • Orders of magnitude improvements for speed of recovery with many variants • Orders of magnitude reduction in buffering • Evident bursty behaviour – concerns for highly loaded packet switched networks Note spikes due to bursts of data Progress with iBOBs at Jodrell

9. Multicast • Multicast reduces load on networks when sending to multiple hosts • Test of multicast topology • Multicast UDP stream over JANET academic network • 100 2.2Mbps multicast IPTV streams originating from London, UK • End-hosts on Manchester University campus network • All Cisco network hardware, 100Mbps interfaces • Measurement of inter-packet arrival times • Time from arrival of packet ‘n’ to arrival of packet ‘n+1’ • Expected ~6ms for one 2.2Mbps stream Progress with iBOBs at Jodrell

10. Multicast • Latency spikes of up to 0.5s at intervals of 60s • Unexpected behaviour may limit the application of this technology • Correlation between streams and between PCs shows a common source of delay Progress with iBOBs at Jodrell

11. Multicast • Test of multicast topology - results • Multicast traffic routing is not as efficient as unicast on this network • Potential concerns for other networks • Does not look suitable for eVLBI Progress with iBOBs at Jodrell

12. DCCP • Stable kernel • Require tuning of virtual memory settings to prevent kernel panics • Experimental kernels • Revisions of late November 2007 address some memory and performance issues – yet to be tested on our systems. • CCID4 in development • Performance • CCID2 can attain 1GE line-rate butit is not a turn-key thing • Future work • Test 2.6.24 kernel and recent DCCP git tree – not mature enough yet for a definitive statement about suitability Progress with iBOBs at Jodrell

13. Progress on Digital Interfaces iBOBs and all that Jonathan Hargreaves/Richard Hughes-Jones

14. The iBOB System • As a transmitter: • Now able to transfer data at line rate to a test PC over a 10 GE (CX-4) link • Needs A/D connection and interface (Onsala/Metsahovi) • As a receiver • Needs to interface to VSI chip on station board • As a network test system • See also Richard’s talk • Fujitsu switch ordered Progress with iBOBs at Jodrell

15. JBO Receiver: Status Recent Progress • Sixteen packet FIFO implemented in the iBOB’s FPGA • Even though packets are transmitted at 4Gbps they might ‘bunch up’ and arrive at line speed. The FIFO is intended to smooth them out again • The 10Gb transceiver can run at 200MHz – enough to handle line speed, but the SRAM bandwidth is 167MHz for both read and write. Read out at 4Gbps consumes 64MHz of that. More resource than needed • Data flows from the iBOB to the Station Board defined (see next slide) • Data readout from the SRAM synchronised to the correlator clock, when present • When correlator clock is not present, iBOB defaults to test mode using the internal clock • VSI chip will buffer and reclock data and route it to the Station Board input Next Tasks • Test the new code in hardware using another iBOB to generate test packets • Write firmware for the VSI chip and either simulate or test on remote hardware Progress with iBOBs at Jodrell

16. Clock Control Lock 128MHz clock to correlator . 200MHz System Clock 128MHz 10GB RX ‘Yellow box’ on the Simulink design 4GB/s CX4 FIFO 16k deep Holds up to 16 packets in case short bursts arrive at line rate . SRAM Control . Re-orders out of order packets Counts lost packets Reads data out to the correlator at a steady 4Gbps ‘VSI chip’ Buffer and reclock data. Reroute it to the left and right polarisation station board (SB) inputs. . DATA to SB Input Correlator CLK DATA 64 bit DATA 64 bit DATA 32 bit DATA 32 bit DATA to SB Input ADDR DATA 2 x MDR80 ‘VSI’ cables SRAM 512k x 64 bit (512 packets = 9ms at 4Gbps) . iBOB Station Board JBO Receiver: Block Diagram Progress with iBOBs at Jodrell

17. Network Testing Device: Status Recent Progress • System clock increased from 100MHz to 200MHz to allow line speed data flows at 10Gbps (JH) • Added jumper bank to select ethernet IP and MAC address (JH & RHJ) • All registers, test modes and results can be accessed via ethernet control/response protocol (RHJ) • PC based software provides user interface to the ethernet control link (RHJ) • Ethernet based control supersedes the TinySH command line and is seen as a prototype for control of iBOBs sending real data • Currently testing iBOB to iBOB transfers on bench Next Tasks • Count lost packets, out of order packets and duplicate packets • Update real time field correctly each second – currently it is fixed over a burst • Test over network Progress with iBOBs at Jodrell