OptIPuter System Software

1. OptIPuter System Software Eric Weigle et al. speaking for Andrew A. ChienComputer Science and Engineering, UCSD January 2006 OptIPuter All-Hands Meeting

2. OptIPuter System Software Architecture Applications Visualization DVC API Distributed Virtual Computer Middleware DVC Runtime Library DVC Configuration DVC Services DVC Communication DVC Job Scheduling DVC Core Services Resource Identify/Acquire Namespace Management Security Management High Speed Communication Storage Services Globus XIO GSI RobuStore GRAM Optical Signaling, Management GTP XCP UDT Photonic Infrastructure High-Speed Transport Protocols CEP LambdaStream RBUDP Distributed Applications/ Web Services Visualization Telescience SAGE JuxtaView Data Services Vol-a-Tile LambdaRAM PIN/PDC

3. Performance Across Layers 3 &5Layer Demos Applications Visualization Distributed Virtual Computer Novel Transport Protocols Optical Network Configuration Year 4 & 5: Integration, Performance Tuning, Tech Transfer Valerie Taylor et al (TAMU)

4. Cross Team Integration and Demonstrations • 2-layer Demo, TeraBIT Juggling, [SC2004, Nov2004] • DVC middleware, high-speed transport (GTP) • Move data between OptIPuter network endpoints • 10 endpoints across UCSD, UvA, UIC, Pittsburgh • Achieved 17.8Gbps, a TeraBIT in less than one minute • 3-layer Demo [AHM2005, Jan2005] • Visualization (JuxtaView/LambdaRAM), DVC middleware, high-speed transport • Remote data visualization (visualization: NCMIR; storage: UIC and UvA) • Use DVC to establish visualization environments • Automated Grid resource selection and binding • Achieved 2.6 Gbps on ~7 Streams • 5-layer Demos [iGrid2005, Sep2005] • Applications, visualization, DVC middleware, high-speed transports (GTP), optical network configuration (PIN/PDC) • Demo #1: Collaborative Data Visualization with Earth-Sciences • Demo #2: Real-time Brain Data Acquisition, Assembly and Analysis

5. OptIPuter High-Performance Transport Protocols • Bridge the Gap between High Speed Link Technologies and Growing Demands of Advanced Applications • TCP has well-documented performance problems on long-haul networks • Pursue complementary avenues of investigation • Efficient congestion/flow management, fairness among flows • High-speed group communication (multipoint-to-point, multipoint-to-multipoint) Network Connection Private Lambda Shared, Routed Managed Group Standard Routers Enhanced Routers Unicast GTP CEP RBUDP/ l-stream SABUL/ UDT XCP

6. Composite Endpoint Protocol (CEP)- Accomplishments & Plans • OptIPuter “Gold Roll” 1 (CEP v. 1.1) • Initial release, basic functionality • 32 Gbps in the LAN • TCP, some automatic tuning • Software Summit Release (CEP v. 1.2) • New file transfer, sockets API • New internal networking stack • Preliminary GTP, XIO carrier support • Support for 64-bit systems, more OSes • OptIPuter “Gold Roll” 2 (CEP v. 2.0) • Target: Fall 2006 • Code stabilization • Documentation • Improved scalability • Improved GTP, XIO integration • Improved performance • Suitable for public release Eric Weigle (UCSD)

7. Group Transport Protocol (GTP)- Accomplishments & Plans sources sinks 1 2 3 4 Active Sessions 5 6 7 8 9 Lambda Networks • Extend GTP to Sender Capacity Management • End node based allocation schemes at both sources and sinks to achieve good global performance and fairness • Proof of stability and convergence properties of GTP • Comprehensive comparison studies between GTP and other transport protocols • Implementation and Demonstrations with OptIPuter System Software • iGrid2005 • OptIPuter “Gold Roll” 1 (Basic functionalities) • 3 Publications • Analytical studies • Finalize convergence proofs in asynchronous cases • More comparison studies • Study the interaction between GTP and other TCP traffic • Implementation: OptIPuter “Gold Roll” 2 • GTP v. 2.0 • Target: Summer 2006 • Goals • Support both source and sink allocation schemes • Improved CPU efficiency and scalability • Improved CEP, XIO integration • Suitable for public release Ryan Wu (UCSD)

8. UDP-based Data Transfer Protocol (UDT)– Accomplishments & Plans • Based upon experience to date, a new version of UDT is being developed called Composible-UDT • Composible-UDT supports multiple high-speed congestion control algorithms, including • UDT decreasing-increases AIMD congestion control • Reliable UDP blast • TCP, HighSpeed TCP, Scalable TCP, BiC, FAST, Vegas, Westwood • GTP • Recent Experimental Studies • iGrid 2005 - High performance mining of data streams over UDT • 8 Gb/s computing histograms on web traffic data streams • 14 Gb/s transferring data memory-memory around the world • iGrid 2005 - Remote exploration of Sloan Digital Sky Survey data • 1.2 Gb/s transfer disk-disk from San Diego to South Korea • Plan for UDT 3.0 release • Composable-UDT (first full release) • Congestion controlled unreliable messaging • Firewall punching • Further work on protocol toolkit underlying Composible-UDT • Experiments to understand how to provide secure transport using UDT Robert L. Grossman & Yunhong Gu (UI-Chicago)

9. UDP Offload Engines for LambdaGrids • Offload UDP Based Protocols to the Network Cards. • Initial Results • 7.4 Gbps Maximum throughput. • 35% improvement over Host based UDP • Reduced CPU utilization. • 17% improvement in Latency • Future Plans • Build LambdaStream on the NIC. • Evaluate Partial Offload Engines with full offload. A Case for UDP-offload engines in LambdaGrids – Venkatram Vishwanath, et al. (PFLDNet 2006) - In collaboration with OSU, LANL, VT) Venkatram Vishwanath et al. (EVL) Venkatram Vishwanath et al. (EVL)

10. LambdaStream • An application-level transport protocol for streaming and data transfer for dedicated high-bandwidth networks. • Current Status • A single-stream version with a configurable design. • API which supports buffered and unbuffered communications. • Working towards a Multi-stream LambdaStream for Multipoint to Multipoint communication • Recent Results • 18 Gbps between Chicago and San Diego over TeraWave. • 18 Gbps between Chicago and San Diego over CaveWave. • Future Plans • Integrate with SAGE and other applications. • Performance Evaluation with MAGNET to identify end-system bottlenecks. Venkatram Vishwanath et al (EVL)

11. MAGNET • A monitoring apparatus for generic kernel event tracing. • Identify end system performance bottlenecks in a generic linux kernel and improve next generation protocols, middleware, and software applications. • Current status: • Uses the “probes” mechanism in the kernel. • Designed as a kernel module. • Monitor the Network Stack. • Future Plans: • More instrumentation points. • Performance Analysis of LambdaStream. • Analysis and Synthesis for Adaptive Visualization applications Joint work with Wu Feng, Mark GardnerLANL and Virginia Tech. Venkatram Vishwanath et al. (EVL)

12. Storage- Accomplishments & Plans • RobuSTore Design • RobuSTore Architecture Consisting of Coding Algorithm, Metadata Service, Admission Controller, and Security Schemes • RobuSTore Evaluation Across a Wide Range of System Configurations • Evaluate RobuSTore against Conventional Parallel Storage Schemes (i.e. RAID): • Explore Five Dimensions • # Disks, Data Size, Block Size, Network Latency, Degree of Redundancy •  5x Improvement on Robustness; 15x on Access Bandwidth •  Moderate Overhead: 2~4x Storage Capacity, 1.5x Network, Disk I/O • Simulation Study with More Complex and Realistic Workloads • RobuSTore Implementation (based on Lustre) and Deployment • Experiments on the OptIPuter Testbed • Evaluation Using Benchmarks and Neuroscience and Geophysical Application Workloads Huaxia Xia, Justin Burke (UCSD)

13. Vision – Real-Time Tightly Coupled Wide-Area Distributed Computing Real-Time Object network Goals • High-precision Timings of Critical Actions • Tight Bounds on Response Times • Ease of Programming • High-Level Prog • Top-Down Design • Ease of Timing Analysis Dynamically formed DistributedVirtual Computer K. Kim (UCI)

14. Real-Time Progress • RCIM (RT comm infrastructure mgt) • Study of TT Ethernet under way with the help of Hermann Kopetz • Hope to acquire the 1st unit some time in 2006. • IRDRM (Intra-RT-DVC resource mgt) • TMO (Time-triggered Message-triggered Object) Support Middleware (TMOSM) • Redesigned TMOSM improves modularity, concurrency, portability, and timing precision. It runs on Linux, WinXP, & WinCE. • Extending the TMOSM to exploit unique capabilities of Jenks’ cluster SPDS2. • Programming model • API for RT middleware enables high-level RT programming (TMO) without a new compiler. • The notion of Distance-Aware (DA) TMO, an attractive building-block for RT wide-area DC applications, was created created and a study for its realization is under way. • Enhancement the Network Infrastructure of OptIPuter • GPS receivers acquired from German vendor & installed in UCI/UCSD (Calit2 bldgs) • One-way message delay between UCI and UCSD measured • Jitters were less than 60 microsecs. • Application development experiments • Preparation of demos; at stage where LAN-based feasibility demos are working. • e.g., Low-jitter video, Fair and efficient Distributed On-Line Game Systems • Publications in IDPT2003, AINA2004, WORDS2005, ISORC2005, … Source: Kim, Jenks, et al. at UCI

15. Year 4 Plan • RCIM (RT comm infrastructure mgt) • Development of middleware support for TT Ethernet • IRDRM (Intra-RT-DVC resource mgt) • Extending the TMOSM to further exploit unique capabilities of Jenks’ cluster SPDS2 • Full development of Support for Distance-Aware TMOs • Interfacing TMOSM to the Basic Infrastructure Services of OptIPuter • Demos • Remote access and control of electron microscopes at UCSD-NCMIR • Remote control of an electric car OptIPuter Paths Remote Controlnode Local Relaynode Source: Kim, Jenks, et al. at UCI

16. Areas of Security Effort • Trusted remote computation (UCI) • Protection from cheating • Reduce effort on clients, coordinator effort • Solution: inject false positives (“chaff”) as a check • Broadcast keying (UCI) • Protect messages to large groups • Minimize number and distribution of keys • Solution: chromatic leap-frog keys, predeployed keysets • Secure network protocols (USC/ISI) • Transport disconnection, spoofing • Alleviate DOS attack impact • Solution: layer different algorithms, SPI-spinning Joe Touch & Mike Goodrich, UCI/ISI

17. IPsec Baseline Performance Mbps Packet size Joe Touch & Mike Goodrich, UCI/ISI

18. Effect of DOS Traffic • IETF TCPM WG doc • Need for IP-layer solution • IETF BTNS WG • Infrastructure-free security • IETF Triage session • Reducing load of spoofed DOS traffic • Goal: Internet Standard Mbps Packet size Joe Touch & Mike Goodrich, UCI/ISI

19. Summary of Accomplishments • Integration and Demonstration of Capability • All five layers (application, visualization, DVC, transport protocols, Optical network control) • Across campus, national and international-scale test beds • Distributed Virtual Computer • Integrate with network configuration (PIN/PDC) • Simulation study of service models of configurable network • Simulation study of efficient resource selection algorithms • Advanced Transport Protocols • GTP: Analytic and simulation study, extend to sender capacity management • CEP: Implement and evaluate N-to-M communication; integrate with XIO • SABUL/UDT: Development of composable congestion control algorithms • LambdaStream: Configurable design, API for buffered/unbuffered communication • Real-Time Programming/Networking/DVC • Time-triggered Message-triggered Object support middleware & API • Enhancement of optiputer network infrastructure with GPS • Evaluation of delay/jitter between UCI/UCSD • File/Storage Systems • New RobuSTore architecture with Erasure coding, statistical guarantees • Coding algorithm, metadata service, admission controller, security schemes • Evaluation across a wide range of system configurations & conventional schemes • Performance Analysis/Modelling • Performance Analysis of VolaTile & other OptIPuter software • Network Security (Touch & Goodrich)

20. OptIPuter ROCKS Roll Releases • OptIPuter System SW Roll (Optigold) v.1 [July2005] • Stable, integrated for OptIPuter iGrid2005 Demos • OptIPuter software • DVC Middleware v.1.0 • Core resource, security, namespace and job management services • Network binding service (interface with PDC) • DVC configuration and communication APIs • Advanced Transport Protocols • Group Transport Protocol (GTP) v.0.95 • Composite Endpoint Protocol (CEP) v1.1 • Optical Network Configuration • Photonic Domain Controller (PDC) v.2.0 • External Software • Globus Toolkit 4.0 • OptIPuter Software Summit 2006 (Jan-Feb, 2006) • Build a Completed, Tested, End-to-end OptIPuter Software Stack • Create Site for Downloadable OptIPuter Software

21. Summary of Plans • Lots of progress in 2005- Met and exceeded goals! • Planned Features & Improvements • Distributed Virtual Computer (DVC) • Second-gen architecture, prototypes, improved cross-team integration and resource selection • Real-Time (TMO) • RT communication infrastructure, Intra-RT DVC resource mgmt, applications (NCMIR) • Performance Analysis (Prophesy) • Develop and utilize archive for performance data • High Speed Protocols (CEP, LambdaStream, XCP, GTP, UDT) • CEP: version 2.0: code stabilization, integration, performance, documentation • GTP: version 2.0: convergence, comparison studies, integration with other protocols • UDT version 3.0: composible, congestion controlled, firewall punching, security • LamdaStream: UDP Offload, integration with SAGE & apps, bottleneck identification • Storage (Robustore) • Simulation/evaluation with more complex/realistic workloads • implementation and deployment (lustre, optiputer testbed) • Security • Explore IPsec variants to reduce attacker advantage • Planned 2006 releases • ~September 2006 – System Software Gold Roll version 2 • ~November 2006 – cross-team 5-level demonstrations (SC2006)