1 / 24

Collaborations in Networking and Protocols HEP and Radio Astronomy

Collaborations in Networking and Protocols HEP and Radio Astronomy. Richard Hughes-Jones The University of Manchester www.hep.man.ac.uk/~rich/ then “Talks”. VLBI Proof of Concept at iGrid2002 European Topology: NRNs, Geant, Sites. SURFnet. Manchester Jodrell. iGrid 2002. JIVE.

africa
Télécharger la présentation

Collaborations in Networking and Protocols HEP and Radio Astronomy

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Collaborations in Networking and ProtocolsHEP and Radio Astronomy Richard Hughes-Jones The University of Manchesterwww.hep.man.ac.uk/~rich/ then “Talks” Collaboration Meeting , 4 Jul 2006, R. Hughes-Jones Manchester

  2. VLBI Proof of Concept at iGrid2002 European Topology: NRNs, Geant, Sites SURFnet Manchester Jodrell iGrid 2002 JIVE SuperJANET4 Collaboration Meeting , 4 Jul 2006, R. Hughes-Jones Manchester

  3. Collaboration HEP, Radio Astronomy, Dante the NRNs, and Campus folks • Some results of the e-VLBI Proof of Concept Collaboration Meeting , 4 Jul 2006, R. Hughes-Jones Manchester

  4. Dwingeloo DWDM link Jodrell BankUK MedicinaItaly TorunPoland e-VLBI at the GÉANT2 Launch Jun 2005 Collaboration Meeting , 4 Jul 2006, R. Hughes-Jones Manchester

  5. e-VLBI UDP Data Streams • Collaboration HEP, Radio Astronomy, Dante the NRNs, and Campus folks • Good opportunity to test UDP Throughput:5 Hour run Collaboration Meeting , 4 Jul 2006, R. Hughes-Jones Manchester

  6. ESLEA and UKLight • Exploiting Switched Lightpaths for e-Science Applications • EPSRC e-Science project £1.1M 11.5 FTE • Core Technologies: • Protocols • Control plane • HEP data transfers – ATLAS and D0 • e-VLBI • Medical Applications • High Performance Computing • Involved with Protocols, HEP and e-VLBI • Stephen Kershaw appointed as RA (joint with EXPReS) • Investigate how well the protocol implementations work • UDP flows, TCP advanced stacks, DCCP (developed by UCL partners) • Also examine how the Applications “use” the protocols • Also the effect of the transport protocol on what the Application intended! • Develop real-time UDP transport for e-VLBI – vlbi_udp Collaboration Meeting , 4 Jul 2006, R. Hughes-Jones Manchester

  7. Reverse TCP ESLEA and UKLight • 6 * 1 Gbit transatlantic Ethernet layer 2 paths UKLight + NLR • Disk-to-disk transfers with bbcp • Seattle to UK • Set TCP buffer and application to give ~850Mbit/s • One stream of data 840-620 Mbit/s • Stream UDP VLBI data • UK to Seattle • 620 Mbit/s Collaboration Meeting , 4 Jul 2006, R. Hughes-Jones Manchester

  8. TCP Congestion windowgets re-set on each Request • TCP stack RFC 2581 & RFC 2861 reduction of Cwnd after inactivity • Even after 10s, each response takes 13 rtt or ~260 ms • Transfer achievable throughput120 Mbit/s • Event rate very low • Application not happy! tcpmon: TCP Activity for remote Farms:Manc-CERN Req-Resp • Web100 hooks for TCP status • Round trip time 20 ms • 64 byte Request green1 Mbyte Response blue • TCP in slow start • 1st event takes 19 rtt or ~ 380 ms Collaboration Meeting , 4 Jul 2006, R. Hughes-Jones Manchester

  9. ESLEA: ATLAS on UKLight • 1 Gbit Lightpath Lancaster-Manchester • Disk 2 Disk Transfers • Storage Element with SRM using distributed disk pools dCache & xrootd Collaboration Meeting , 4 Jul 2006, R. Hughes-Jones Manchester

  10. Send times • Pause 695 μs every 1.7ms • So expect ~600 Mbit/s • Receive times (Manc end) • No corresponding gaps udpmon: Lanc-Manc Throughputnot quite what we expected !! • Lanc  Manc • Plateau ~640 Mbit/s wire rate • No packet Loss • Manc Lanc • ~800 Mbit/s but packet loss Collaboration Meeting , 4 Jul 2006, R. Hughes-Jones Manchester

  11. EXPReS & FABRIC • EU Project to realise the current potential of eVLBI and investigate the Next Generation capabilities. • SSA • Use of GRID Farms for distributed correlation. • Linking Merlin telescopes to JIVE (present correlator) 4 * 1 Gigabit from Jodrell Links to 10 Service Challenge work. • Interface to eMERLIN – data at 30 Gbit/s • JRA - FABRIC • Investigate use of different IP Protocols • 10 Gigabit Onsala to Jodrell Links to 10 Gbit HEP work. • Investigate 4 Gigabit over GEANT2 Switched LightpathsUDP and TCP Links to Remote Compute Farm HEP work. • Develop 1 and 10 Gbit Ethernet end systems using FPGAsLinks to CALICE HEP work. Collaboration Meeting , 4 Jul 2006, R. Hughes-Jones Manchester

  12. FABRIC 4 Gigabit Demo • Will use a 4 Gbit Lightpath between two GÉANT PoPs • Collaboration with Dante – Discussions in progress • Continuous (days) Data Flows – VLBI_UDP and multi-Gigabit TCP tests Collaboration Meeting , 4 Jul 2006, R. Hughes-Jones Manchester

  13. Data Transfer CSR Access 2.8us 10 Gigabit Ethernet: UDP Data transfer on PCI-X • Sun V20z 1.8GHz to2.6 GHz Dual Opterons • Connect via 6509 • XFrame II NIC • PCI-X mmrbc 2048 bytes66 MHz • One 8000 byte packets • 2.8us for CSRs • 24.2 us data transfereffective rate 2.6 Gbit/s • 2000 byte packet, wait 0us • ~200ms pauses • 8000 byte packet, wait 0us • ~15ms between data blocks Collaboration Meeting , 4 Jul 2006, R. Hughes-Jones Manchester

  14. Calice • Virtex 4 board from pld Applications • PCI-express development card • Using the FPGA to send and receive raw Ethernet frames at 1 Gigabit • Package data from internal memory or external source into Ethernet • Considering building a 10 Gigabit Ethernet add-on card • Take data in on the1Gig links, processing it, send results out on 10Gig link. • Using 2 boards (2nd is a data generator) we could produce a small scale Calice DAQ, take data in, buffer it to the DDR2 ram, and then read it out, Ethernet frame it and ship to PCs. • Ideas for an Ethernet packet monitor. From Slides of Marc Kelly Collaboration Meeting , 4 Jul 2006, R. Hughes-Jones Manchester

  15. Backup Slides Further network & end host investigations Collaboration Meeting , 4 Jul 2006, R. Hughes-Jones Manchester

  16. VLBI Work TCP Delay and VLBI Transfers Manchester 4th Year MPhys Project by Stephen Kershaw & James Keenan Collaboration Meeting , 4 Jul 2006, R. Hughes-Jones Manchester

  17. VLBI Network Topology Collaboration Meeting , 4 Jul 2006, R. Hughes-Jones Manchester

  18. Timestamp3 Timestamp4 Packet loss Timestamp5 Receiver Sender Data3 Data4 RTT ACK Segment time on wire = bits in segment/BW Time • Remember Bandwidth*Delay Product BDP = RTT*BW VLBI Application Protocol TCP & Network Sender Receiver Timestamp1 Timestamp2 Data1 Data2 ●●● • VLBI data is Constant Bit Rate • tcpdelay • instrumented TCP program emulates sending CBR Data. • Records relative 1-way delay Time Collaboration Meeting , 4 Jul 2006, R. Hughes-Jones Manchester

  19. 1 way delay 100 ms 100 ms Message number 1-Way Delay 1 way delay – 10000 packets • 10,000 Messages • Message size: 1448 Bytes • Wait time: 0 • TCP buffer 64k • Route:Man-ukl-JIVE-prod-Man • RTT ~26 ms Collaboration Meeting , 4 Jul 2006, R. Hughes-Jones Manchester

  20. = 1 x RTT 26 ms 1 way delay 10 ms = 1.5 x RTT 10 ms ≠ 0.5 x RTT Message number 1-Way Delay Detail • Why not just 1 RTT? • After SlowStart TCP Buffer Full • Messages at front of TCP Send Buffer have to wait for next burst of ACKs – 1 RTT later • Messages further back in the TCP Send Buffer wait for 2 RTT Collaboration Meeting , 4 Jul 2006, R. Hughes-Jones Manchester

  21. Recent RAID Tests Manchester HEP Server Collaboration Meeting , 4 Jul 2006, R. Hughes-Jones Manchester

  22. “Server Quality” Motherboards • Boston/Supermicro H8DCi • Two Dual Core Opterons • 1.8 GHz • 550 MHz DDR Memory • HyperTransport • Chipset: nVidia nForce Pro 2200/2050 • AMD 8132 PCI-X Bridge • PCI • 2 16 lane PCIe buses • 1 4 lane PCIe • 133 MHz PCI-X • 2 Gigabit Ethernet • SATA Collaboration Meeting , 4 Jul 2006, R. Hughes-Jones Manchester

  23. Disk_test: • areca PCI-Express 8 port • Maxtor 300 GB Sata disks • RAID0 5 disks • Read 2.5 Gbit/s • Write 1.8 Gbit/s • RAID5 5 data disks • Read 1.7 Gbit/s • Write 1.48 Gbit/s • RAID6 5 data disks • Read 2.1 Gbit/s • Write 1.0 Gbit/s Collaboration Meeting , 4 Jul 2006, R. Hughes-Jones Manchester

  24. UDP Performance: 3 Flows on GÉANT • Throughput:5 Hour run • Jodrell: JIVE2.0 GHz dual Xeon – 2.4 GHz dual Xeon670-840 Mbit/s • Medicina (Bologna):JIVE 800 MHz PIII – mark6231.2 GHz PIII330 Mbit/s limited by sending PC • Torun:JIVE 2.4 GHz dual Xeon – mark5751.2 GHz PIII245-325 Mbit/s limited by security policing (>400Mbit/s  20 Mbit/s) ? • Throughput:50 min period • Period is ~17 min Collaboration Meeting , 4 Jul 2006, R. Hughes-Jones Manchester

More Related